TIL on Aldrin Jenson

Clone any website locally with HTTrack

Mon, 02 Sep 2024 12:00:00 -0400

brew install httrack

httrack <website_url>

Source: https://hackzism.blogspot.com/2020/06/clone-websites-using-httrack.html

YouTube transcript formatter for LLMs

Sun, 21 Jul 2024 12:00:00 -0400

Demo here:
https://x.com/aldrinjenson/status/1815007108934332486

#!/bin/bash

# YouTube Transcript Extractor
# This script downloads and extracts the transcript from a YouTube video,
# calculates some statistics, and copies the result to the clipboard.

# Instructions:
# 1. Ensure you have the following dependencies installed:
#    - yt-dlp: https://github.com/yt-dlp/yt-dlp#installation
#    - jq: https://stedolan.github.io/jq/download/
#    - xclip (for Linux) or pbcopy (for macOS, comes pre-installed)
# 2. Make this script executable: chmod +x youtube_transcript_extractor.sh
# 3. Run the script with a YouTube URL: ./youtube_transcript_extractor.sh "https://www.youtube.com/watch?v=VIDEO_ID"

# Requirements:
# - Linux or macOS
# - Bash shell
# - yt-dlp
# - jq
# - xclip (Linux) or pbcopy (macOS)

# Function to copy to clipboard
copy_to_clipboard() {
    if command -v pbcopy > /dev/null; then
        pbcopy
    elif command -v xclip > /dev/null; then
        xclip -selection clipboard
    else
        echo "Unable to copy to clipboard: neither pbcopy nor xclip is available."
        return 1
    fi
}

# Function to format duration
format_duration() {
    local total_seconds=$1
    local hours=$((total_seconds / 3600))
    local minutes=$(( (total_seconds % 3600) / 60 ))
    local seconds=$((total_seconds % 60))
    printf "%02d:%02d:%02d" $hours $minutes $seconds
}

# Check if a URL is provided
if [ $# -eq 0 ]; then
    echo "Please provide a YouTube video URL as an argument."
    echo "Usage: $0 <YouTube_URL>"
    exit 1
fi

# YouTube video URL
URL="$1"

# Download subtitles and metadata
yt-dlp --skip-download --write-subs --write-auto-subs --sub-lang en --sub-format ttml --convert-subs srt --output "transcript.%(ext)s" --write-info-json "$URL"

# Clean up the subtitle file
sed -i.bak -e '/^[0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9][0-9] --> [0-9][0-9]:[0-9][0-9]:[0-9][0-9].[0-9][0-9][0-9]$/d' \
    -e '/^[[:digit:]]\{1,3\}$/d' \
    -e 's/<[^>]*>//g' \
    ./transcript.en.srt && rm ./transcript.en.srt.bak

# Create the final output file
sed -e 's/<[^>]*>//g' -e '/^[[:space:]]*$/d' transcript.en.srt > output.txt

# Remove the intermediate .srt file
rm transcript.en.srt

echo "Transcript extraction complete. The result is saved in output.txt"

# Calculate word count and line count
WORD_COUNT=$(wc -w < output.txt)
LINE_COUNT=$(wc -l < output.txt)

# Extract metadata
TITLE=$(jq -r '.title' transcript.info.json)
DURATION=$(jq -r '.duration' transcript.info.json)
UPLOAD_DATE=$(jq -r '.upload_date' transcript.info.json)

# Format duration
DURATION_FORMATTED=$(format_duration $DURATION)

# Prepare metadata string
METADATA="Video Title: $TITLE
Duration: $DURATION_FORMATTED
Upload Date: ${UPLOAD_DATE:0:4}-${UPLOAD_DATE:4:2}-${UPLOAD_DATE:6:2}
Word Count: $WORD_COUNT
Line Count: $LINE_COUNT"

echo "Transcript Statistics and Metadata:"
echo "$METADATA"

# Copy the transcript to clipboard with the specified format and metadata
{
    echo "The following youtube video transcript:"
    echo ""
    cat output.txt
    echo ""
} | copy_to_clipboard

if [ $? -eq 0 ]; then
    echo "Transcript and metadata have been copied to clipboard."
else
    echo "Transcript and metadata were not copied to clipboard due to an error."
fi

# Clean up metadata file
rm transcript.info.json

A super fast voice translation system

Sat, 20 Jul 2024 12:00:00 -0400

https://x.com/aldrinjenson/status/1814683816679903551

Super simple voice based translation system with probably the fastest speeds available right now

— Aldrin Jenson (@aldrinjenson) Original tweet date

Changing the email on old git commits

Sat, 20 Jul 2024 12:00:00 -0400

To change the email address for all commits in your Git repository to a new correct one, you can use the git filter-branch command. Here’s a step-by-step process:

First, ensure you have the correct email set for future commits:
```
git config user.email "your-new-correct-email@example.com"
```

Then, run the following command to rewrite the history:

git filter-branch --env-filter '
OLD_EMAIL="your-old-incorrect-email@example.com"
CORRECT_EMAIL="your-new-correct-email@example.com"
if [ "$GIT_COMMITTER_EMAIL" = "$OLD_EMAIL" ]
then
    export GIT_COMMITTER_EMAIL="$CORRECT_EMAIL"
fi
if [ "$GIT_AUTHOR_EMAIL" = "$OLD_EMAIL" ]
then
    export GIT_AUTHOR_EMAIL="$CORRECT_EMAIL"
fi
' --tag-name-filter cat -- --branches --tags

Replace your-old-incorrect-email@example.com with the wrong email you used, and your-new-correct-email@example.com with your correct email.\

After running this command, Git will rewrite your repository’s history, changing the email address for all commits.\
If you’ve already pushed your commits to a remote repository, you’ll need to force push the changes:
```
git push --force --tags origin 'refs/heads/*'
```

Be cautious with this operation, especially if you’re working on a shared repository, as it rewrites history. This can cause issues for other contributors if they’ve based work on the old history.

source: Claude sonnet 3.5 :)

Different options for doing LLM inference

Thu, 11 Jul 2024 12:00:00 -0400

Intro

While many developers default to using OpenAI’s APIs for large language model (LLM) inference, there are numerous alternatives available that offer different advantages in terms of cost, performance, and flexibility. This guide explores various options for LLM inference, including both self-hosted and cloud-based solutions, as well as serverless and traditional hosting approaches.

Cloud-Based Solutions

1. Replicate

Pros:
- Offers thousands of models, including popular and fine-tuned versions
- Supports custom model deployment (both public and private)
- Easy to get started quickly
Cons:
- Initial pricing may be higher compared to other options
If you want to build a prototype quickly for a startup, this is recommended

2. Modal.com

Pros:
- Affordable pricing (e.g., 10 applications for $9 on GPUs)
- Serverless architecture
Cons:
- Cold start issues, which can lead to longer initial processing times
Based on experience with Indic Subtitler, this is cheap but the free tier will have sufficient lag if you have a large model which needs to be loaded to memory once the serverless system boots up

3. Hugging Face Spaces

Pros:
- Always running, no cold start issues
- Exposes API using Gradio
Cons:
- No pay-as-you-go pricing

4. Fal.ai

Offers serverless inference options
Compare with Replicate and make a decision

5. Groq

Pros:
- Significantly cheaper than OpenAI (e.g., 10x less cost for Whisper model)
- Faster processing due to specialized LPU chips
Cons:
- Limited model availability (currently offering Whisper v3, Llama 2, Llama 3, Gemma, Mixtral)
- Requires contacting for premium trial, with potential delays as of now

6. AWS Options

Bedrock: Umbrella service for various AI tools
SageMaker: For model training and deployment

7. NVIDIA NIM (New)

Pros:
- Containerizes LLM models (similar to Kubernetes approach)
- Allows local development and easy cloud deployment
- Simple inference, similar to OpenAI APIs
- Supports local testing and cloud deployment
Cons:
- Currently not open for everyone (requires filling out a form to try)
How it works:
- Containerize your LLM model (e.g., GPT-3.5, GPT-4)
- Develop locally
- Push to cloud, which spins up a separate container for the model to run

Self-Hosted Solutions

1. eXLama V2

Pros:
- One of the fastest open-source LLM inference options
- Supports quantized models
- Based on LLaMA.cpp
Cons:
- Cannot offload to CPU; requires good VRAM

2. VLLM

Pros:
- High-performance inference
Cons:
- Does not support quantized models

3. LLaMA.cpp

Pros:
- Supports quantized models
- Offers HTTP server functionality for API exposure

4. BentoML

Supports serverless inference
Kubernetes based
Open Source

5. Seldon.io

Kubernetes-based open-source solution for scaling models
Alternative to BentoML

6. Ollama

Pros:
- One of the easiest ways to run models locally
- Supports multiple GGUF format versions easily pullable from Ollama hub
- Offers both CLI and API interfaces
- Recently added support for concurrency and parallelization
How it works:
- Pull models from Ollama hub
- Run locally with simple commands
- Use CLI or API for inference

Considerations When Choosing an LLM Inference Solution

Cost: Compare pricing structures, especially for long-term usage.
Performance: Consider factors like cold start times and inference speed.
Scalability: Evaluate how well the solution can handle increasing loads.
Flexibility: Check support for custom models and fine-tuning capabilities.
Ease of Use: Consider the learning curve and integration complexity.
Hardware Requirements: For self-hosted solutions, ensure you have the necessary hardware (e.g., GPUs with sufficient VRAM).

Conclusion

While OpenAI’s APIs are popular, exploring alternative LLM inference options can lead to cost savings, improved performance, or better alignment with specific project requirements. For those just starting, cloud-based solutions like Replicate or Modal.com offer an easy entry point. As your needs grow or become more specific, you might consider transitioning to more comprehensive services like AWS or self-hosted solutions for greater control and customization.

For local development and testing, tools like Ollama provide an excellent balance of ease of use and flexibility. NVIDIA NIM presents an interesting containerized approach that could bridge the gap between local development and cloud deployment.

Remember to regularly reassess your chosen solution as the field of AI and LLMs is rapidly evolving, with new options and improvements emerging frequently.

Generate subtitles locally using Whisper

Sat, 30 Mar 2024 12:00:00 -0400

#!/bin/bash
# Single script to convert a file to wav and have it be transcribed by whisper

# Check if a filename is provided as an argument
if [ -z "$1" ]; then
  echo "Usage: $0 <filename>"
  exit 1
fi

fileName="$1"

# If the file is not already a WAV file, convert it
if [[ $(file -b --mime-type "$fileName") != audio/wav ]]; then
  echo "Converting to WAV file"
  # Modify the conversion command as needed
  ffmpeg -i "$fileName" -acodec pcm_s16le -ar 16000 "$(basename "$fileName" .${fileName##*.}).wav"
  convertedFileName="$(basename "$fileName" .${fileName##*.}).wav"
else
  convertedFileName="$fileName"
fi

if [ ! -f "$convertedFileName" ]; then
  echo "Error: Conversion failed."
  exit 1
fi

# Transcribe the WAV file
echo -e "\n\n\nTranscribing..\n\n\n"
~/code/whisper.cpp/main -osrt -m /Users/aldrinjenson/code/whisper.cpp/models/ggml-medium.en.bin "$convertedFileName" && echo "Transcribed successfully" || echo "Error in transcribing!"

Next steps:

Add the above script to path

P.S. Do remember to change the path of whisper.cpp to the correct location in your computer.

Talking AI companion

Fri, 29 Mar 2024 12:00:00 -0400

A quick afternoon build — a voice-based AI companion meant to help people talk things out when they’re having a rough time.

Demo on X →

Drag and drop files from the macOS terminal

Wed, 20 Mar 2024 12:00:00 -0400

Use this cool tool:

https://github.com/Wevah/dragterm

Steps to install:

git clone the above repo
cd dragterm
brew install cocoapods
g++ DTDraggingSourceView.m main.m -framework Cocoa -o drag
Add the folder to PATH

export PATH="<your drag term folder path>:$PATH"

Usage:

drag <filename>

Bonus

If you are using ranger or some other CLI based file manager, you can easily add support to dragging from terminal using dragterm with a key-map as follows:

map <C-d> shell drag %p

Paste the above to .config/ranger/rc.conf to map ctrl+d to drag a file from ranger

Fetching Train availability in IRCTC

Mon, 11 Mar 2024 12:00:00 -0400

Problem

When you search for trains in irctc page, it never shows the seat availability directly. You’ll have to manually go and click the refresh button of every listed train

List of trains, but availability is not present

Steps to fix

Search for trains in https://www.irctc.co.in
Apply filters to narrow down the list of trains

Narrow down list of trains using filters

Paste the following script in console to get the availability status of all trains at once

document.querySelectorAll('.fa.fa-repeat').forEach(el=>el.click())

The end. Tada!

Note:

Ensure that you always try this out after narrowing down the list using filters. Else it’ll be too many requests going on at once.

Compressing images from the CLI

Fri, 01 Mar 2024 12:00:00 -0400

Steps

Move all your image files to a folder
calculate it’s folder size using du
create a copy of the folder.
install imagemagick

brew install imagemagick

in this folder, perform the following actions:

mkdir compressed
for i in *; do convert $i -resize 40% compressed/$i; done;

The above will compress all images by making the size 40% of the initial size and will move it to a folder called compressed

Check folder structure again using du
if it’s still too much, repeat again but this time reduce quality to even lower

Running parallel Ollama inference

Fri, 01 Mar 2024 12:00:00 -0400

Problem

Ollama runs based on a single backend concept.

Even if you have good specs in your system, every new request is processed in a queue manner to make maximum resource usage for a single query.

This is useful for personal laptop usage. But if you have a powerful machine with 10s of GPU RAM, and multiple users inferring at same time, it would be a problem

Solution

Run multiple ollama instances as isolated container using docker.

References:

Ollama blog announcing docker containers

Approach 1

Use docker swarm

docker swarm init

docker service create --replicas 10 \
--name ollama \
--constraint 'node.labels.gpu'==true \
--mount type=volume,source=ollama,target=/root/.ollama \
--publish published=11434,target=11434 \
ollama/ollama

This creates 10 replicas managed by docker swarm.

The base url is same as that of a single instance which you can connect to just like connecting to a regular standalone instance of ollama

Problem: This one doesn’t have GPU support

Approach 2

Manage everything ourselves with a tiny bit of help from litellm

Steps:

Write a script to create the specified number(n) of ollama container from port 11434 to 11434+n
Ensure that all of them uses the same volume to store the models so as to save up space
Create litellm proxy server using a custom config file(generated automatically from create script)
Connect to this litellm proxy just as connecting to a single ollama instance.

#!/bin/bash

# create.sh - to create a given set of docker containers running llama2

CONFIG_FILE="config.yml"
VOLUME_NAME="ollama_volume"
MODEL="llama2"

echo "model_list:" > $CONFIG_FILE

# Define function to create and manage ollama containers
create_ollama_container() {
    local container_name="ollama$1"
    local port=$2

    # Check if the container with the same name exists, if yes, remove it
    if docker ps -a --format '{{.Names}}' | grep -q "^$container_name$"; then
        echo "Removing existing container: $container_name"
        docker rm -f $container_name
    fi

    # Check if the container on the specified port exists, if yes, remove it
    if docker ps -a --format '{{.Ports}}' | grep -q "0.0.0.0:$port->"; then
        echo "Removing existing container on port $port"
        docker rm -f $(docker ps -a --format '{{if (index (split .Ports "/") 0) | (index (split . "->") 0) | (eq "'"0.0.0.0:$port"'")}}{{.Names}}{{end}}')
    fi

        # Create and run the new container
    docker run -d --gpus=all \
        -v $VOLUME_NAME:/root/.ollama \
        -p $port:11434 \
        --name $container_name \
        ollama/ollama

    echo "Created container: $container_name on port $port"

    # Add entry to the config file
    echo "  - model_name: $MODEL" >> $CONFIG_FILE
    echo "    litellm_params:" >> $CONFIG_FILE
    echo "        model: ollama/$MODEL" >> $CONFIG_FILE
    echo "        api_base: http://localhost:$port" >> $CONFIG_FILE

}

# Create n different ollama containers with unique names and ports
for i in {1..6}; do
    port=$((11434 + i))
    create_ollama_container $i $port
done

destroy.sh

#!/bin/bash
# Script to destroy running docker containers

docker stop $(docker ps -a -q) && docker rm $(docker ps -a -q)

test.sh

#!/bin/bash
# test.sh - to make request to a given  number of llama2 instances

rm file.txt && echo "file.txt cleare"

# Define the command to be executed
curl_command="curl --location 'http://0.0.0.0:8000/chat/completions' --header 'Content-Type: application/json' --data ' {
  \"model\": \"llama2\",
  \"messages\": [
    {
      \"role\": \"user\",
      \"content\": \"tell a long story about unicorns\"
    }
  ]
}' | jq .choices[0].message.content  >> file.txt"

# Run the command n times in parallel
for i in {1..3}; do
    eval "$curl_command" && echo "Command $i succeeded." &  # Run the command in the background
done

# Wait for all background processes to finish
wait

echo "All commands completed."

Sample config.yaml

model_list:
  - model_name: llama2
    litellm_params:
        model: ollama/llama2
        api_base: http://localhost:11435
  - model_name: llama2
    litellm_params:
        model: ollama/llama2
        api_base: http://localhost:11436
  - model_name: llama2
    litellm_params:
        model: ollama/llama2
        api_base: http://localhost:11437
  - model_name: llama2
    litellm_params:
        model: ollama/llama2
        api_base: http://localhost:11438