InterviewPrepKit Blog https://interviewprepkit.dev/blog/ Deep dives on LLMs, AI agents, and ML systems by Armin Norouzi. en-us Attention Is O(n²): FlashAttention vs Linear Attention https://interviewprepkit.dev/blog/attention-complexity-flash/ https://interviewprepkit.dev/blog/attention-complexity-flash/ Tue, 22 Sep 2026 10:00:00 GMT Standard attention on a 32K-token sequence allocates 2,199 GB of score-matrix memory across a 32-layer model — 27× the capacity of an A100 80GB. LLM Under the Hood — Part 5: The Transformer Encoder https://interviewprepkit.dev/blog/llm-under-the-hood-part-05-encoder/ https://interviewprepkit.dev/blog/llm-under-the-hood-part-05-encoder/ Thu, 17 Sep 2026 10:00:00 GMT Build a complete Transformer Encoder Layer in PyTorch with residuals, LayerNorm, and FFN. Real print(model) output of an 8,544-parameter encoder. Positional Encoding: From Sinusoidal to RoPE to ALiBi https://interviewprepkit.dev/blog/positional-encoding-rope-alibi/ https://interviewprepkit.dev/blog/positional-encoding-rope-alibi/ Tue, 15 Sep 2026 10:00:00 GMT Transformers are permutation-invariant by design: without positional information, the model treats 'the cat sat on the mat' identically to 'the mat sat on the cat.' Positional encoding injects order… LLM Under the Hood — Part 4: Multi-Head Attention https://interviewprepkit.dev/blog/llm-under-the-hood-part-04-multihead-and-positional/ https://interviewprepkit.dev/blog/llm-under-the-hood-part-04-multihead-and-positional/ Thu, 10 Sep 2026 10:00:00 GMT Run attention in parallel across multiple heads, then inject word order with sine/cosine positional encoding. Real PE matrix visualized in PyTorch. Entropy, Cross-Entropy, and KL Divergence in LLM Training https://interviewprepkit.dev/blog/entropy-cross-entropy-kl-divergence/ https://interviewprepkit.dev/blog/entropy-cross-entropy-kl-divergence/ Tue, 08 Sep 2026 10:00:00 GMT Each code block in this article is self-contained and can be run independently — that is why a few helper functions (softmax, kl_divergence, etc.) are re-defined across snippets. LLM Under the Hood — Part 3: The Attention Mechanism https://interviewprepkit.dev/blog/llm-under-the-hood-part-03-attention/ https://interviewprepkit.dev/blog/llm-under-the-hood-part-03-attention/ Thu, 03 Sep 2026 10:00:00 GMT Build scaled dot-product attention from scratch in PyTorch. See real attention weights from a forward pass. The mechanism that powers every modern LLM. Backpropagation Without the Magic: A First-Principles Derivation https://interviewprepkit.dev/blog/backpropagation-first-principles/ https://interviewprepkit.dev/blog/backpropagation-first-principles/ Tue, 01 Sep 2026 10:00:00 GMT Every ML engineer uses backpropagation daily. Most treat it as a framework primitive — loss.backward() runs and gradients appear. LLM Under the Hood — Part 2: From RNNs to LSTMs https://interviewprepkit.dev/blog/llm-under-the-hood-part-02-rnns-and-lstms/ https://interviewprepkit.dev/blog/llm-under-the-hood-part-02-rnns-and-lstms/ Thu, 27 Aug 2026 10:00:00 GMT Why vanilla RNNs fail on long sequences and how LSTM gates fix it. Real PyTorch training and BPTT gradient measurements. Part 2 of LLM Under the Hood. Why Transformers Work: Attention as Learned Soft Retrieval https://interviewprepkit.dev/blog/why-transformers-work/ https://interviewprepkit.dev/blog/why-transformers-work/ Tue, 25 Aug 2026 10:00:00 GMT Every practitioner working with transformers has encountered the attention mechanism described the same way: 'it lets tokens attend to each other.' That description is accurate and useless. LLM Under the Hood — Part 1: Introduction & Word Embeddings https://interviewprepkit.dev/blog/llm-under-the-hood-part-01-introduction-and-embeddings/ https://interviewprepkit.dev/blog/llm-under-the-hood-part-01-introduction-and-embeddings/ Thu, 20 Aug 2026 10:00:00 GMT A 10-part series rebuilding modern LLMs in PyTorch from scratch. Part 1 starts with word embeddings and a real CBOW model trained on a tiny corpus. Feature Store Architecture: Fix Training-Serving Skew https://interviewprepkit.dev/blog/feature-store-architecture/ https://interviewprepkit.dev/blog/feature-store-architecture/ Tue, 18 Aug 2026 10:00:00 GMT A recommendation model that performs well offline but degrades in production is one of the most common and most expensive problems in machine learning. Async DAGs vs Fibers: Stackless and Stackful Concurrency in C++ https://interviewprepkit.dev/blog/async-dags-vs-fibers/ https://interviewprepkit.dev/blog/async-dags-vs-fibers/ Mon, 17 Aug 2026 00:00:00 GMT A future, a promise, a reactive stream, a sender — they're all the same async DAG: a graph of continuations that fires as results resolve. Fibers express the same graph as sequential code with a stack. Here's the architecture of each, down to the coroutine frame and the register swap, and how to choose. Multi-Agent Fan-Out: When Parallelism Bites Back https://interviewprepkit.dev/blog/multi-agent-fanout-patterns/ https://interviewprepkit.dev/blog/multi-agent-fanout-patterns/ Sun, 16 Aug 2026 00:00:00 GMT Scatter-gather is one of the most seductive patterns in distributed systems: split a hard problem into N pieces, run them in parallel, collect the results.