Blog posts
RSSDeep dives on LLMs, AI agents, and ML systems, and the engineering underneath the tools.
- Hierarchical RAG: Document Summaries to Chunk Retrieval Flat chunk retrieval in RAG applies the same similarity search to every chunk across every document simultaneously.
- Implement Speculative Decoding with a Draft Model in Python Generating text with a large language model is sequential by default: one token at a time, one forward pass per token.
- Implement Flash Attention from First Principles in NumPy Standard transformer attention materializes an N×N matrix of attention scores for every layer and head.
- Production RAG API with FastAPI, pgvector, and Claude Build a production RAG pipeline from chunking and vector search to Maximal Marginal Relevance (MMR) reranking, FastAPI serving, and pgvector indexing.
- LLM Under the Hood — Part 10: DPO Alignment Direct Preference Optimization from scratch in 30 lines of PyTorch. Real reward-gap measurements from a 100-step training run. The modern alignment default.
- The Geometry of Embeddings: Why Cosine Similarity Works Why cosine similarity works for text embeddings, when to prefer dot product or L2 distance, and how concentration of measure and anisotropy shape high-dimensional retrieval.
- LLM Under the Hood — Part 9: LoRA & QLoRA Implement LoRA from scratch in PyTorch — 0.91% trainable parameters with comparable quality to full fine-tuning. Plus QLoRA 4-bit quantization.
- Flash Attention Mechanics: How Tiled Attention Fits in SRAM Self-attention is the operation that lets every token in a sequence influence every other token.
- LLM Under the Hood — Part 8: Supervised Fine-Tuning (SFT) Run a real SFT loop on a tiny PyTorch Transformer. Watch loss drop, see why full fine-tuning is expensive, and set up the LoRA payoff in Part 9.
- Tensor Parallelism: Splitting a Model Across GPUs A 70B model in FP16 requires roughly 130 GB of memory. No single A100-80GB card can hold it.
- LLM Under the Hood — Part 7: Pre-trained Models & Tokenization Load real pre-trained LLMs with Hugging Face, decode BPE tokenization, and use chat templates. The modern industrial workflow in PyTorch, explained.
- Gradient Checkpointing: Trading Compute for Memory Gradient checkpointing re-runs parts of the forward pass during backprop instead of keeping all intermediate activation tensors in memory, trading FLOPs for VRAM.
- LLM Under the Hood — Part 6: The Decoder & Full Transformer Build the Decoder layer with masked self-attention and cross-attention. Assemble the full nn.Transformer in PyTorch with real print(model) output.
- Attention Is O(n²): FlashAttention vs Linear Attention Standard attention on a 32K-token sequence allocates 2,199 GB of score-matrix memory across a 32-layer model — 27× the capacity of an A100 80GB.
- LLM Under the Hood — Part 5: The Transformer Encoder Build a complete Transformer Encoder Layer in PyTorch with residuals, LayerNorm, and FFN. Real print(model) output of an 8,544-parameter encoder.
- Positional Encoding: From Sinusoidal to RoPE to ALiBi Transformers are permutation-invariant by design: without positional information, self-attention treats any permutation of tokens identically. Comparing sinusoidal positional encoding, RoPE, and ALiBi across expressiveness, context length, and extrapolation.
- LLM Under the Hood — Part 4: Multi-Head Attention Run attention in parallel across multiple heads, then inject word order with sine/cosine positional encoding. Real PE matrix visualized in PyTorch.
- Entropy, Cross-Entropy, and KL Divergence in LLM Training How Shannon entropy, cross-entropy, perplexity, and forward vs. reverse KL divergence govern LLM pretraining, temperature sampling, knowledge distillation, and RLHF.
- LLM Under the Hood — Part 3: The Attention Mechanism Build scaled dot-product attention from scratch in PyTorch. See real attention weights from a forward pass. The mechanism that powers every modern LLM.
- Backpropagation Without the Magic: A First-Principles Derivation Every ML engineer uses backpropagation daily. Most treat it as a framework primitive — loss.backward() runs and gradients appear.
- LLM Under the Hood — Part 2: From RNNs to LSTMs Why vanilla RNNs fail on long sequences and how LSTM gates fix it. Real PyTorch training and BPTT gradient measurements. Part 2 of LLM Under the Hood.
- Why Transformers Work: Attention as Learned Soft Retrieval Every practitioner working with transformers has encountered the attention mechanism described the same way: 'it lets tokens attend to each other.' That description is accurate and useless.
- LLM Under the Hood — Part 1: Introduction & Word Embeddings A 10-part series rebuilding modern LLMs in PyTorch from scratch. Part 1 starts with word embeddings and a real CBOW model trained on a tiny corpus.
- Feature Store Architecture: Fix Training-Serving Skew A recommendation model that performs well offline but degrades in production is one of the most common and most expensive problems in machine learning.
- Async DAGs vs Fibers: Stackless and Stackful Concurrency in C++ A future, a promise, a reactive stream, a sender — they're all the same async DAG: a graph of continuations that fires as results resolve. Fibers express the same graph as sequential code with a stack. Here's the architecture of each, down to the coroutine frame and the register swap, and how to choose.
- Multi-Agent Fan-Out: When Parallelism Bites Back Scatter-gather is one of the most seductive patterns in distributed systems: split a hard problem into N pieces, run them in parallel, collect the results.