13 — Rapid-Fire Q&A

This chapter is a drill deck for the short questions that open almost every AI-engineering interview: what is a token, what happens in one forward pass, when should you not build an agent. Seventy-two of them follow, in the form you should say them out loud — one mechanism, one consequence, and one number wherever a number exists. Work through it and you will be able to answer the question and the follow-up that comes after it, which is nearly always some version of “why?”.

Every card stands on its own. Terms of art are defined where they appear rather than assumed from an earlier chapter, so you can open this file cold, at any card, and still follow it. Where a longer derivation exists elsewhere, there is a link — but you never need to follow it to understand the card in front of you.

How to use this deck

What goes in and what comes out. The input is the question in bold. The output is the paragraph underneath it: the answer you say out loud, three to six sentences, roughly twenty to forty seconds of speech.

The drill is to cover the answer, say yours, then uncover and compare. You are grading yourself on whether you named the mechanism, not on whether you used the same words.

Every answer gives the mechanism, not just the fact. That is deliberate. A memorized fact has no second layer, and the interviewer’s second question is where most candidates stop. Where a number appears, the derivation appears with it. The highest-value cards carry a follow-up they’ll ask with its answer attached.

The map: seven mechanisms, six answers

Before the cards, here is the shape of the first section. The diagram below has mechanisms on the left and interview answers on the right. Arrows mean “this mechanism is why that answer is true.” Read it once now; you do not need to memorize it.

flowchart LR
    A["Causal attention"] --> C["Prompt caching<br/>render order"]
    K["KV cache<br/>prefill vs decode"] --> C
    K --> P["Output costs ~5x<br/>TTFT vs TPOT"]
    Q["Quadratic attention<br/>history resent"] --> G["Cost growth<br/>context management"]
    U["U-shaped recall"] --> G
    E["Lossy embeddings"] --> R["Hybrid search<br/>cross-encoder rerank"]
    M["Logit masking"] --> S["Structured output<br/>tool calling"]
    F["Batched float<br/>non-determinism"] --> V["N=3 eval gating"]

    style C fill:#2d6a4f,color:#fff
    style G fill:#2d6a4f,color:#fff
    style P fill:#40916c,color:#fff
    style R fill:#40916c,color:#fff
    style S fill:#40916c,color:#fff
    style V fill:#40916c,color:#fff

How to read the diagram. Each arrow below is stated in full as a card later in this section, so treat this as a table of contents rather than something to absorb now.

• Causal attention → prompt caching / render order. Causal attention is the rule that a token may look at earlier tokens and never at later ones. That is what forces prompt caching to be a prefix match, and therefore what makes render order — the order in which your request is assembled before it is sent — decide your cache hit rate.
• KV cache → prompt caching, and → output pricing / latency. The model stores the key and value vectors of every token it has already processed. That store creates two phases of work with different physics, which is what explains both prompt caching and why output costs roughly 5x what input costs, with time to first token (TTFT) and time per output token (TPOT) moving under separate levers.
• Quadratic attention → cost growth / context management. Attention compute grows with the square of the token count. Combine that with the fact that an agent resends its whole history every turn and you get cost growth, which is why context management is a skill and not a setting.
• U-shaped recall → cost growth / context management. Retrieval accuracy is highest at the beginning and end of a long context and lowest in the middle. That pushes toward the same design conclusions as quadratic attention.
• Lossy embeddings → hybrid search / rerank. Embeddings preserve meaning and discard exact wording, which is why serious retrieval needs keyword search alongside vector search plus a reranking pass.
• Logit masking → structured output / tool calling. The runtime zeroes out any token that would break your schema, so valid shape is enforced rather than requested.
• Batched float arithmetic → N=3 eval gating. Summation order depends on which other requests share your batch, so identical inputs can produce different outputs — which is why eval gating in continuous integration runs each case three times and takes the majority.

Seven mechanisms on the left generate the six answers on the right. Six is the count of the boxes on the right, not the left, because two of the answers take two mechanisms each.

The two darker green boxes are those two: causal attention and the KV cache both feed prompt caching / render order, and quadratic attention and U-shaped recall both feed cost growth / context management. The four lighter green boxes each follow from a single mechanism.

If you can reconstruct the arrows, you can answer follow-ups this file never anticipated.

Every mechanism in the first section is stated in full below. Chapter 00 — LLM internals carries the longer derivations if you want to go deeper on any of them.

1. LLM internals

Start inside the large language model (LLM) itself: how text becomes tokens, what a single forward pass produces, why the KV cache exists, and why nearly every cost, latency, and caching rule you will be asked about is a consequence of that cache.

What is a token, and why does the count depend on the model? A token is a subword unit — a chunk of text, usually a few characters long. Tokens are produced by a tokenizer, and the tokenizer is trained with byte-pair encoding (BPE): an algorithm that starts from individual characters and repeatedly merges the most frequent adjacent pair until it has a fixed-size vocabulary.

The result is that common words end up as a single token and rare strings split into several. ERR_4021 becomes four tokens: ERR, _, 40, 21.

The tokenizer ships as part of the model, so the same text has different token counts on different models. Never reuse a count from one model to budget another, and never use tiktoken for Claude — it is OpenAI’s tokenizer and undercounts prose by roughly 15-20%, code by more. Treat that range as a direction rather than a constant. It is a property of whichever two tokenizers you happen to compare, not a fixed conversion, so measure it on your own text if you must estimate at all. The rule of thumb is ~4 characters per token for English prose, worse for code and much worse for non-Latin scripts. When it matters, call count_tokens against the model you will actually use.

Follow-up they’ll ask: “Why does that matter for retrieval?” — Because a rare identifier fragments into low-information pieces, which is the first half of why dense embedding search misses exact error codes. The second half is in the embeddings card further down this section, and the fix is in Retrieval rag.

What happens in one forward pass? A forward pass takes a sequence of tokens as input and produces exactly one new token as output. In between, four things happen:

1. Each token is turned into an embedding — a list of numbers representing that token.
2. Those vectors run through N layers of attention plus a feed-forward network (FFN, the per-token transformation that sits between attention layers).
3. The final layer emits logits: one raw, unnormalized score for every entry in the vocabulary.
4. Softmax converts those scores into probabilities that sum to 1, and one token is sampled from that distribution.

One pass produces exactly one token, so a 500-token response is 500 sequential passes, each taking the previous output as part of its input. That loop, and the fact that each pass has to wait for the one before it, is where essentially all generation latency lives.

What is the KV cache, and why does it exist? Inside attention, every token is projected into three vectors: a Query (what this token is looking for), a Key (what this token offers to others searching), and a Value (what it contributes when it is matched).

To generate token n+1, the model needs the Key and Value of every token from 0 to n. Recomputing all of them at every step would make generating n tokens cost O(n^3) — growing with the cube of the length. So once a token has been processed, its K and V are kept in GPU memory (the memory attached to the graphics processor the model runs on) and never recomputed. That store is the KV cache.

Every performance property of an LLM API — the network service you call to run the model — is downstream of this cache: the prefill/decode asymmetry, output pricing, and prompt caching all fall out of it.

Prefill vs. decode — what is the difference and why do I care? Serving a single request happens in two phases, and they have completely different physics. The table compares them row by row; the row to fixate on is Bottleneck, because everything else follows from it.

You care because the two phases move under different levers. Prompt caching attacks prefill and does nothing for decode; shortening the output attacks decode and does nothing for TTFT. Naming which term you are attacking is the difference between an answer and a guess.

Follow-up they’ll ask: “So how do you cut latency on a long-context agent?” — TTFT is dominated by prefilling a long history, so cache the prefix. TPOT is set by how many tokens you ask for, so prefer an edit tool over a write tool, and stream the response so the user sees tokens as they arrive.

Why do output tokens cost roughly 5x input tokens? This is not a pricing whim; the ratio tracks a real difference in what the two phases cost to run. Prefill is one efficient parallel matrix multiplication over all input tokens at once. Decode is a sequential, memory-bound crawl in which every single token re-reads a growing cache. Output costs more because generating a token is genuinely more expensive than reading one.

Concretely, on claude-opus-5 the price is $5/MTok (per million tokens) for input and $25/MTok for output — exactly the 5x ratio. The practical consequence is that shortening output is a bigger win than shortening input, which is the entire argument for diff-based editing in a coding agent.

Why is prompt caching a prefix match rather than a general cache? Because attention is causal, meaning token i attends to tokens 0 through i and never forward to anything later. That has a direct consequence: token i’s K and V depend only on itself and everything before it.

So if tokens 0 through j of your request are identical to a previous request, their K and V are bit-for-bit identical and can be reused. The moment token j+1 differs, every token after it sits in a different preceding context, so it has different K and V, so nothing downstream is reusable. That one sentence generates every caching rule you will ever need.

Follow-up they’ll ask: “Why do reads cost ~10% and writes 1.25x?” — A cache read skips the prefill arithmetic entirely but still pays to move the stored K and V into GPU memory, so it is much cheaper but not free. A cache write pays normal prefill plus the cost of persisting the cache. Break-even is two requests. Count in multiples of what one uncached request’s input would cost: two uncached requests cost 2.0, while a cached pair costs 1.25 for the write plus 0.1 for the read, which is 1.35 — cheaper from the second request on.

Why does one changed byte invalidate everything after it? It is the same mechanism, run forward. Say a timestamp changes a token at position 30. Every token from 30 onward now sits in a different preceding context, so it has different K and V, so nothing from position 30 on is reusable — even though 99% of the text is unchanged.

The same mechanism also bounds the damage: content before the change is still cached and still free. That is why datetime.now() belongs at the end of the last user message and never in the system prompt.

Why does quality degrade in long contexts? Two distinct effects get lumped together under the label “context rot,” and naming them separately is most of the answer.

The first is attention dilution. Softmax normalizes attention weights so they sum to 1 across all positions, so when 200k tokens are competing, the weight any single token can receive is bounded by the competition.

The second is positional bias. Measured retrieval accuracy across position follows a U-shaped recall curve — strong at the start and the end, weakest in the middle. That is a learned artifact of training data in which important content clusters at the beginnings and ends of documents.

The design consequences follow directly: restate the goal near the end of the context, and offload material rather than stuffing it in.

Follow-up they’ll ask: “So a 1M-token window solves context management?” — No. Cost still scales with tokens, attention is still quadratic, and mid-window recall still degrades. A larger window raises the ceiling; it does not change the shape of the curve.

What does an embedding actually store, and what does it lose? An embedding is a fixed-length vector — a list of numbers — produced by a model trained so that texts with similar meaning land close together under cosine similarity, a closeness measure based on the angle between two vectors.

The key property is that it is a lossy compression optimized for meaning, not identity. Pooling a whole passage into ~1,024 floats — averaging its per-token vectors into a single one — necessarily discards detail, and the training objective decided which detail survives: semantic gist, not rare literals.

Lossy embeddings are therefore why the query "why am I getting ERR_4021" retrieves passages about errors in general. BM25 is the exact complement — a keyword-scoring function that weights each term by inverse document frequency, so a rare term gets a high weight precisely because it is rare.

Why isn’t temperature=0 deterministic? Setting the temperature to 0 means always taking the highest-scoring token (the argmax), so the output should be reproducible — and it almost is.

The gap is that floating-point addition is not associative: adding the same numbers in a different order can change the last bit of the result. GPU kernels — the low-level routines that perform the arithmetic — sum in an order that depends on batch composition, that is, on whose requests happened to be batched alongside yours. Batched float addition in a different order gives different last-bit rounding, which occasionally flips the argmax, and once one token differs the sequences diverge.

Never build a system whose correctness depends on identical output across runs. Cache by input hash when you need stability, and assert on properties rather than exact strings in tests. This is also why eval gating in continuous integration (CI) — blocking a merge on your evaluation suite — needs an N=3 majority: run each case three times and take the verdict that wins, instead of comparing against one exact expected string.

How does structured output actually work? The mechanism is logit masking, also called constrained decoding. At each decode step, before a token is sampled, the runtime sets the logit of every token that would make the output invalid under your schema to negative infinity, which makes its probability exactly zero. (Your schema here is the JSON — JavaScript Object Notation — description of the fields you require.) Sampling then draws only from what survives.

So invalid output is not merely unlikely — it is impossible by construction. Two consequences follow. Never write a JSON-repair retry loop, because there is nothing to repair. And remember that fields are generated in schema order, so a reasoning field must be declared before the score it justifies, or you get a rationalization of a number the model already committed to.

Follow-up they’ll ask: “Then why do I still get semantically wrong output?” — Masking enforces shape, not intent. {"date": {"type": "string"}} makes "next Tuesday" perfectly valid. Every constraint you do not declare is a constraint the model can violate at zero cost.

Is tool calling a separate engine? No. Tool schemas are serialized into the prompt like everything else, the model emits a structured call under the same constrained decoding described above, and the runtime parses that call and sets stop_reason: "tool_use". It is tokens in, tokens out, the whole way through. Saying this correctly signals that you understand the API rather than having memorized its surface.

Why is attention quadratic, and does it matter to me? With n tokens, attention computes a score between every pair of tokens, so the Q x K^T matrix is n x n. Doubling the context roughly quadruples the attention compute.

Quadratic attention matters because context length is not a free dial: a 200k-token window does not cost 2x a 100k one on the attention term, and “just put everything in context” stops being a good answer well before you reach the model’s documented limit.

2. Agent fundamentals

From the model, move up one level: what an agent actually is, how its loop runs and terminates, and — the question most candidates answer badly — when you should refuse to build one.

What is an AI agent, and how does it differ from a simple LLM call? A single LLM call maps a prompt to an output once and stops. An agent runs a loop in which the model decides the next step and when to stop, based on tool results it observes along the way.

Stated formally: a workflow computes next_step = f(current_step) where f is code you wrote, while an agent computes next_step = model(history, tools) where f is a forward pass.

The dividing line isn’t tool use — a workflow can call tools too — it is who owns the control flow. That substitution is what buys adaptivity and what costs predictability: you cannot enumerate the possible paths, so you cannot test them exhaustively, so every guarantee has to come from invariants that hold no matter which path runs.

Follow-up they’ll ask: “Then how do you test one?” — You test the outcome strictly and the trajectory loosely. Assert that a forbidden tool was never called and that a call budget was respected; do not assert an exact sequence of steps, because the path is legitimately non-deterministic.

What’s the difference between a workflow and an agent? In a workflow you write the control flow in code, so the path is fixed and knowable at design time. In an agent the model chooses the path at runtime.

The honest argument for preferring a workflow is cost rather than simplicity. Let P be the fixed prompt and a the tokens each step adds.

• A workflow with N stages sends N x (P + a) tokens. That is linear in N.
• An agent resends its entire history every turn, so over n turns it sends n x P + a x n(n-1)/2 tokens. That is quadratic in n.

Be careful how you quote that second formula. At n = 20 the history term is a x 190, and 190 (or 200, if you round n^2/2) is the coefficient on the per-turn delta a — it is not a number of calls.

Worked example. Take P = 6,000, a = 1,200, n = 20. The agent sends 20 x 6,000 + 1,200 x 190 = 120,000 + 228,000 = 348,000 tokens. Twenty independent single calls would send 20 x (6,000 + 1,200) = 144,000. So the agent costs 348,000 / 144,000 = 2.4x — not 10x, because the fixed prefix P here is five times the per-turn delta a.

That ratio is entirely set by P/a. When a large fixed prefix dominates, the agent is barely above 1x; when the deltas dominate, it approaches n/2, about 10x at n = 20. Prefer a workflow whenever you can draw the flowchart.

What is an agent loop, and how does it decide when to stop? The loop has four steps that repeat:

1. You call the model.
2. If the response carries stop_reason == "tool_use", you execute the requested tools.
3. You append the entire assistant message plus every tool result to the history.
4. You call the model again.

It stops on end_turn, on a verifiable goal predicate (a condition your code can check, such as “the test suite passes”), on a step cap, on a budget ceiling, on a loop detector, or on a human veto.

Prefer the verifiable predicate, because end_turn only means the model sampled a stop token — a statement about the token distribution, not about the world. “Tests pass” is checkable; “the model says it’s done” is a vibe, and it fails silently and confidently.

Follow-up they’ll ask: “Which stop reasons do people forget to handle?” — max_tokens, which returns HTTP 200 — the success status — with silently truncated output; refusal, where the content may be an empty list so resp.content[0] raises IndexError; and pause_turn, where the loop exits early with a partial answer and no error. All three fail quietly.

Reactive vs. proactive agents? A reactive agent runs in response to a user message. A proactive agent is triggered by a schedule, a webhook (an inbound call from another system), or a watcher, with no human waiting on the result.

A proactive agent is a distributed system with an LLM inside it. It needs idempotency keys, retries, durable state, dead-letter handling for messages that never succeed, and alerting, because a failure may be seen by nobody for hours.

The specific hazard is at-least-once delivery meeting a non-idempotent agent: your queue redelivers the message after a timeout, the agent runs a second time, and a second email goes out. The fix is the same as in any distributed system — an idempotency key (a unique id for the unit of work, meaning running it twice has the same effect as running it once) checked and written before the irreversible act, not after.

What is harness engineering? The harness is all the code around the model: prompt assembly and ordering, tool dispatch, authorization, retries, context trimming, budget accounting, loop detection, and tracing.

A better model with a bad harness loses to a worse model with a good one. Concretely, a harness that lets tool output flood the context hits quality collapse around turn 30 no matter which model you use, because that failure is positional — it comes from where content sits in the window, not from how smart the model is.

Authorization in particular cannot live in the prompt, because a prompt rule is advisory, and 1% non-compliance on a destructive action is unacceptable.

Follow-up they’ll ask: “Your agent is worse in production than in the demo — where do you look?” — Not the prompt. Pull traces and diff five hashes: the prompt, the tool set, the model id, the index version, and the harness version. Then check cache_read_input_tokens and the distribution of stop_reason values. Prompt tuning is what you do after the diff has told you where the two environments diverged.

When should you not build an agent? Run four checks, and one “no” means drop down a tier of autonomy.

• Complexity: can you specify the steps in advance? If yes, it is a workflow.
• Value: does the outcome justify 5-20x the tokens and quadratic growth in them?
• Viability: is the model actually good at this task type today?
• Cost of error: can mistakes be caught and reversed, through tests, review, or an undo?

The common interview trap describes a fixed four-step pipeline and asks you to “design an agent for it”; the correct answer is a workflow with an agent escape hatch for the ~5% of inputs the pipeline cannot classify.

3. Design patterns

Each of the named architectures — ReAct (reason and act), Plan-and-Execute, reflection, Evaluator–Optimizer, Reflexion, orchestration, and code generation — comes with the mechanical reason it costs what it costs, so you can choose between them out loud rather than reciting a list.

Explain the ReAct architecture. ReAct is short for reason and act, and it interleaves the two: thought, then tool call, then observation, then repeat until done.

Modern native tool calling is ReAct. The thought is the model’s reasoning text, the action is a tool_use block emitted under constrained decoding, and the observation is the tool_result you send back. You no longer hand-prompt the old Thought:/Action:/Observation: text format.

The cost is 5-20x a single call and is unbounded without a step cap. ReAct is the only pattern in the catalog whose context growth is quadratic — which is also why it is the pattern that most needs guards.

Follow-up they’ll ask: “Why does it loop?” — Each iteration appends a (call, result) pair to the transcript. After three near-identical pairs, the transcript has become a few-shot demonstration — worked examples the model imitates — of repeating yourself, so a fourth identical call is the high-probability continuation rather than a mistake. The loop reinforces itself, and waiting for the model to notice is not a strategy.

What is Plan-and-Execute? You plan all the steps up front with a strong model, then execute them with a cheaper one, replanning only when a step fails.

It is cheaper than ReAct over the same horizon for two mechanical reasons. Each executor step runs on a short context — the plan plus one step — so there is no quadratic term. And following an explicit instruction is a much easier task than deciding what to do next, so a smaller model suffices. That puts it at roughly 3-8x a single call versus ReAct’s 5-20x.

Add a recon phase of one or two cheap reads before planning, or the plan is written against an environment nobody looked at. Cap replans at ~3, because a third failed plan means the goal is wrong, not the plan.

ReAct vs. Plan-and-Execute — when each? Use ReAct when step N+1 genuinely depends on what step N returns, so no plan could have been written in advance.

Use Plan-and-Execute for long horizons where ReAct’s context growth is the bottleneck, or where a human should approve the work before anything runs — a plan is a reviewable artifact and a ReAct trajectory is not.

A hybrid — plan coarsely, then run ReAct within each step — is usually the best interview answer, and it is what most production coding agents actually do.

What is agent reflection? Reflection means generate, then critique with a separate call and a fresh context, then revise.

The fresh context is the load-bearing part. If you ask the same conversation “is this good?”, the draft is already in its context, so the model is now predicting the continuation of a conversation in which it just produced that draft — sycophancy (the tendency to agree with whatever is in front of it) and recency both push toward approval. A separate call sees the draft as an input to judge rather than as something it just wrote, for the same reason a code reviewer should not be the author.

Cost is 2N+1 calls for N rounds, not 2N. The first draft is generated before the loop begins, then each round adds one critique call and one revision call. So a 3-round cap is 2 x 3 + 1 = 7 calls, not 6. Rounds 1-2 capture most of the gain.

Follow-up they’ll ask: “What if it oscillates?” — Vague criteria produce a round 3 that undoes round 2. Cap the rounds, make the evaluator return structured per-criterion verdicts instead of prose, and — most important — if a criterion can be checked in code, check it in code. An LLM judge is the fallback for genuinely subjective dimensions, not the default.

What is an Evaluator-Optimizer loop? It is reflection with the criteria written down in advance and a structured verdict coming back. The loop runs like this:

1. One call generates a draft.
2. A second call — fresh context, so it never sees the conversation that produced the draft — judges it against a named list of criteria and returns a per-criterion pass/fail plus the specific issues.
3. The generator revises against those issues.
4. Repeat until the judge passes it or a round cap stops it.

Cost is 2N+1 calls for N rounds, for the same reason as reflection: the first draft is generated before the loop begins, so a 3-round cap is 7 calls and not 6.

Two things decide whether it works. The criteria have to be articulable, because a judge without written criteria is a vibe with a schema. And anything checkable in code should be checked in code rather than asked of the judge.

The loop must return both the draft and whether it passed — a loop that hands back only the draft is indistinguishable from one that succeeded.

It is built three times over, in pseudocode, LangGraph, and the raw SDK, in chapter 02a, which also shows the variant that judges every draft it might return: that one pairs generates with evaluates and costs exactly 2N, and the +1 you drop is precisely the draft nobody read.

What’s the difference between Reflexion and Evaluator-Optimizer? Reflexion adds memory across attempts: the lesson survives the retry and informs future tasks. Evaluator-Optimizer improves a single artifact within a single task and keeps nothing afterwards.

Reflexion only pays off if the lessons transfer, which requires the next task to be drawn from a similar distribution. Its characteristic failure is a lesson store filling with unfalsifiable junk. “Be more careful with edge cases” is pure token tax; “orders_v2 is canonical; orders is a stale view kept for reporting” is specific, falsifiable, and not obvious from the code.

Every stored lesson is a permanent context tax on every future run, so cap the store and let it evict.

What is agent orchestration? Orchestration is coordinating multiple steps or agents toward one goal: decomposing the task, dispatching the pieces, aggregating the results, and resolving conflicts between them.

It can be code-driven, which makes it a workflow, or model-driven, which makes it an orchestrator agent — and the distinction is exactly whether the number and content of the subtasks are decided at design time or at runtime. Orchestrator-worker, where one coordinator hands scoped briefs to several workers, is the common shape.

Code-generating vs. tool-calling agents? A tool-calling agent picks from a fixed set of typed functions, which makes every action auditable, gateable, and one round trip long.

A code-generating agent writes a script that composes actions with loops and conditionals. That is far more expressive, and intermediate results never enter the context — a token win and a latency win at once, since three chained lookups collapse into a single round trip.

The trade is safety. Code generation needs a real isolation boundary: a container, a non-root user, dropped operating-system privileges, no host filesystem, network off by default, and a cap on wall-clock time (meaning real elapsed seconds). Tool calling does not.

4. Tools

Tools are where an agent touches the world: how a model actually calls one, how to write a tool description that fires at the right time, what MCP — the Model Context Protocol — does and does not standardize, and how the cost arithmetic of computer use works.

What is tool use / function calling, and how does it enable agents? You describe each tool as a name, a description, and a JSON schema for its arguments. Those schemas are serialized into the prompt; the model emits a structured call under constrained decoding; the runtime sets stop_reason: "tool_use"; you execute the call and return a tool_result block.

That is what lets a model act rather than only emit text — and the agent loop exists precisely because tool results feed the next decision. There is no separate “function calling engine” anywhere in the stack.

How do you design and define tools for an agent? Names disambiguate, schemas constrain, and the description is a prompt that decides when the tool fires — it is the highest-leverage text in your system after the system prompt itself. State the trigger condition and the boundary (“do not use for X; use Y instead”), not just what the tool does.

Give one tool one job. Return structure rather than prose, so the next step can branch on it.

5-15 tools is comfortable; past ~20, selection accuracy degrades from schema volume alone, because thousands of tokens of near-identical JSON dilute the attention available to any one of them. Both figures are practitioner rules of thumb, not a published threshold. Nothing changes at exactly 20, and the real number depends on how distinct your descriptions are, so treat 20 as the point where you start measuring selection accuracy rather than the point where it breaks.

Follow-up they’ll ask: “How do you know a description is bad?” — Build a confusion matrix of expected tool versus actually called tool across your eval set. Overlapping descriptions show up as one specific off-diagonal pair; sheer schema volume shows up as diffuse off-diagonal spray. Two different diagnoses, two different fixes.

Bash vs. dedicated tools? A general shell tool gives you breadth, and dedicated tools give you control.

A shell string is opaque: the harness cannot gate it, render it for a human, audit it, or parallelize it, because it cannot know what the string will do until it runs. Start with bash for reach, and promote an action into its own typed tool the moment you need any of those four properties.

What is MCP, and how does it standardize tool integration? MCP is the Model Context Protocol, an open protocol for exposing tools, resources, and prompts so that any MCP-speaking agent can use any MCP server — N+M integrations instead of N x M. That is the whole value proposition, and it is a real one.

What it does not solve is selection accuracy, authorization, or cost. And a third-party MCP server is an untrusted input source whose tool descriptions land directly in your context, which makes it a prompt-injection surface and a supply-chain dependency at the same time.

How do you handle 50+ tools? Use tool search: mark tools with defer_loading: true so their schemas stay out of the prompt, and let the model query the catalog when it needs something.

Matching schemas are then appended to the context rather than swapped in. Appending matters mechanically, because it leaves the existing prefix byte-identical and the prompt cache survives, whereas swapping tools in and out would invalidate the whole prefix on every query.

The alternatives are a cheap router that loads a subset per request, or namespacing many tools into fewer, broader ones with an enum parameter to pick the variant. Fix overlapping descriptions before reaching for any of this: infrastructure does nothing for two tools that genuinely mean the same thing.

How do you build a code execution agent safely? You need a real isolation boundary: a container, running as a non-root user, with operating-system privileges dropped, no host filesystem mounted, network off by default with an explicit allowlist, caps on CPU, memory, and wall-clock time, and a fresh instance per session.

Never call exec() on model output inside your own process, and do not rely on RestrictedPython or an allowlist over the abstract syntax tree — both are escapable, and “escapable” here means arbitrary code execution holding your service’s credentials.

How do computer-use agents work? The loop is: capture a screenshot, send it to the model, receive an action back (click(x,y), type, key, scroll), execute that action, capture a new screenshot, and repeat. Every turn ships an image, so cost per step is high and roughly constant.

The two tiers. Quote the tier you are on when you give the arithmetic, because these numbers move with the model. There are exactly two tiers to know (Why images dominate computer use cost tabulates both). The high-resolution tier is claude-opus-5, Opus 4.7 and later, and Sonnet 5 and later; the standard tier is every older model.

How an image is priced. Images are billed in 28x28-pixel patches, one token per patch, so an image costs ceil(w/28) x ceil(h/28). On the high-resolution tier the caps are 2,576 px on the long edge and 4,784 tokens per image. A 1080p screenshot is under both caps, so it is not downscaled, and it costs:

ceil(1920/28) x ceil(1080/28) = 69 x 39 = 2,691 tokens.

Why a run costs more than one image. Each screenshot is resent on every subsequent turn until you trim it. So by turn t the request carries t images, and a 20-step run with no trimming pays 1 + 2 + ... + 20 = 210 image-slots:

210 x 2,691 = 565,110 image tokens.

What trimming buys. Keep only the last 3 images and turn t carries min(t, 3) of them. Summing that over 20 turns gives 1 + 2 + 3 x 18 = 57 image-slots — the multiplier is 57 rather than 60 because the first two turns do not yet have three images to keep. That is:

57 x 2,691 = 153,387 tokens, and 565,110 / 153,387 = 3.7x fewer image tokens.

The standard tier gets the same ratio. There the caps are 1,568 px on the long edge and 1,568 tokens per image, so 1080p is downscaled and each image costs 1,560 tokens instead. The reduction is still 3.7x, because the ratio is 210 / 57 — set by the trimming rule, not by the price of an image.

Computer use is the fallback for when no API exists.

Follow-up they’ll ask: “What else does that trimming break?” — Nothing, provided you keep a text log of what each removed screenshot showed. Drop the pixels, keep the sentence. The agent needs to remember that it clicked “Submit”; it does not need the 1,500 tokens that proved it.

What are Agent Skills? A Skill is a folder containing a SKILL.md file — instructions, plus optional scripts the agent can run. Only the short description sits in the context permanently; the model reads the full file when the description tells it the Skill is relevant.

That is progressive disclosure for instructions, the same idea defer_loading applies to tools, and for the same reason: anything permanently in context is a permanent tax on every turn, and attention is a finite pool.

What is OKF (Open Knowledge Format)? OKF is an emerging specification for packaging domain knowledge — facts, ontologies (structured vocabularies of concepts and their relations), procedures — into a portable, model-agnostic file rather than burying it in a prompt or a vendor-specific store.

The motivation is the same as MCP’s, one layer up: MCP standardizes actions, and OKF aims at knowledge. It is early, and you should say so when you mention it.

5. Memory and context

Next, everything the agent knows beyond the immediate prompt: the four memory types, how to order a context window, and the two techniques — compaction and offloading — that decide whether a long-running agent stays affordable.

What is AI agent memory? Agent memory is everything the agent knows beyond the current prompt. Split it two ways: what persists within a session, which is context engineering, and what persists across sessions, which is storage.

A large context window removes neither need. Cost scales with the tokens you send whether you use them well or not, and mid-window recall degrades long before you reach the documented limit.

What are the types of agent memory? There are four:

• Working memory is the current context window.
• Episodic memory is what happened — past runs, actions, and outcomes.
• Semantic memory is what is true — facts, preferences, entities.
• Procedural memory is how to do things — skills and learned lessons.

Short-term memory is the working kind; long-term memory is the other three. The useful test when someone asks where a piece of information belongs is: does it need to survive a process restart, and does it need to still be true tomorrow?

What is context engineering? Context engineering is deciding what goes into the window and in what order. Stable content goes first — tools, then system, then messages, which is the order the API renders in by construction — and volatile content goes last.

That ordering is not a style preference; it is forced by causal attention. Caching is a prefix match, so anything you want cached has to be positionally first, and anything that changes has to come after the last cache breakpoint — the marker that says “cache everything up to here.”

Context engineering is therefore also what sets your cache hit rate, which makes it a cost lever and not just a quality one.

Follow-up they’ll ask: “How do you verify it’s working?” — Read usage.cache_read_input_tokens on the second call of a session. Zero means something is invalidating the prefix. Nonzero writes with zero reads means your breakpoint is moving between calls. Then diff two rendered payloads and find the first differing byte.

How does context compaction work? When the transcript approaches the window limit, you summarize the early turns and continue from the summary.

Keep the goal, the decisions and why they were made, open items, and unresolved errors. Drop verbose tool output, superseded drafts, and abandoned dead ends.

Compaction improves quality as well as cost, because it moves your instructions out of the mid-window dead zone where recall is weakest and back near the end where it is strong.

Server-side compaction returns a compaction block that you must append back into the conversation — extracting only the text silently loses the state, and nothing errors when you get it wrong.

Compaction vs. context editing? Compaction summarizes the prefix into shorter prose; context editing clears stale tool results or thinking blocks outright, leaving nothing behind.

Use editing when the old output is simply irrelevant, such as a directory listing from turn 3. Use compaction when the narrative matters, such as why you rejected approach A.

Note that both invalidate the cache from the edit point forward, so batch them into occasional large trims rather than trimming a little every turn.

What’s the single best technique for keeping context small? The single most effective technique is offloading. Write large tool outputs to disk and keep a one-line pointer in the context — "wrote 4,812 rows to /tmp/analysis.json (columns: id, region, revenue)".

It changes the growth rate, not just the constant. Truncation makes each turn cheaper but the context still grows with the task; offloading stops it from growing with the task at all. The agent re-reads only what it turns out to need, and a pointer sitting at the end of the window outperforms 60k tokens of file content buried in the middle.

Follow-up they’ll ask: “Doesn’t the agent lose information?” — Only information it can retrieve on demand, which is exactly the trade you want. Keep the pointer descriptive enough to decide whether re-reading is worth it: row count, column names, and the path. A useless pointer such as "saved output to a file" forces the agent to re-read everything, which is worse than not offloading at all.

How do you implement state management in complex workflows? Checkpoint after every step rather than at the end, and store artifacts by reference — paths and ids — rather than as blobs of content inlined into the state.

The state should hold the messages, the task queue, the artifact paths, the budget ledger, and the error history.

Mark a task in-progress before executing it, so that a resume verifies what happened rather than blindly re-executing it; this is the same idempotency problem as in any distributed system. A crash at minute 39 of a 40-minute run should cost you one step, not the whole run.

How do you handle multi-modal inputs and outputs? Images, PDFs, and audio enter the model as content blocks alongside text, so the capability is rarely the hard part.

Cost is the design constraint. On the high-resolution tier (claude-opus-5, Opus 4.7 and later, Sonnet 5 and later), the most any single image can cost is 4,784 tokens, and a plain 1080p screenshot is 2,691 tokens. That is well above the ~1.5k figure often quoted, which comes from the standard tier of older models, where the same screenshot is downscaled and costs 1,560 — see Tools for the patch arithmetic. On top of that, every image is re-sent on every turn until you trim it. So trim old images and downsample when fidelity is not needed.

Transcribe audio first and treat the transcript as the artifact of record, because transcription errors propagate silently into every downstream step, which is a far worse failure mode than a loud one.

6. Retrieval (RAG)

Retrieval-augmented generation (RAG) means fetching relevant text and putting it in the prompt so the model answers from it. The recurring questions are why one retriever is never enough, how to chunk, when retrieval is the wrong tool entirely, and how to evaluate the whole thing in a way that tells you which half is broken.

How does agentic RAG differ from classic RAG? Classic RAG retrieves once through a fixed pipeline and then answers. Agentic RAG makes retrieval a tool the model calls when it decides it needs to — possibly several times with refined queries, possibly not at all.

The ability to skip retrieval matters more than the ability to repeat it. Classic RAG injects irrelevant passages into “hi” and “what’s 2+2”, which costs tokens and actively degrades the answer by surrounding it with distractors.

Why do you need hybrid search? Because the two families of retriever fail in opposite directions, and the failure is mechanical rather than incidental.

Dense embeddings are a lossy compression optimized for semantic gist, so they find paraphrases and miss exact literals — error codes, function names, product SKUs (stock-keeping unit codes) — whose tokens carry little semantic weight and get washed out when the passage is pooled into one vector.

BM25 weights a rare term higher precisely because it is rare, so it nails the literals and misses the paraphrase.

Hybrid search runs both, fuses the two ranked lists with reciprocal rank fusion (which scores each document by the reciprocal of its rank in each list, so no score calibration between systems is needed), and then reranks the survivors with a cross-encoder.

Follow-up they’ll ask: “Why is a reranker better than just retrieving more?” — A bi-encoder embeds the query and the document separately, which is exactly what makes the index precomputable and also what makes it blind to how the two interact. A cross-encoder feeds the query and document through the model together, so full attention runs across the pair. That is far more accurate and far too slow to run over a whole corpus, meaning the full document collection. Hence the standard shape: retrieve 50 cheaply, rerank those 50 expensively, keep 5.

How should you chunk? Chunk on structure — headings, functions, sections — rather than on fixed character counts, because a boundary drawn at 500 characters lands mid-sentence and mid-idea.

Overlap between chunks exists so that a fact split across a boundary still appears whole somewhere; respect the structure and you need less of it.

Contextual chunking — prepending a document-level summary to each chunk before embedding it — is the biggest single-technique win reported, and the reason is straightforward: a chunk that says “it supports up to 100 requests per minute” is unretrievable unless the chunk also says what “it” is.

RAG vs. tool call vs. fine-tuning vs. long context?

These four get confused constantly. The table sorts them by the question you should ask first — what kind of thing are you trying to give the model? — and the “Why” column is the sentence to say out loud.

The mistake to avoid saying out loud is reaching for fine-tuning to fix factual errors. It does not fix them, and it is the most expensive available way to not fix them.

What is GraphRAG and when is it worth it? GraphRAG builds a graph of entities and the relationships between them across the corpus, and answers by traversing that graph instead of by similarity alone.

It answers global questions — “what themes recur across all our incident reports?” — that top-k retrieval, which returns only the k chunks most similar to the query, structurally cannot. No single chunk contains the answer, and similarity search only ever returns chunks.

The cost is a heavy indexing pass up front and staleness on a corpus that changes often. It is worth it when the questions are genuinely aggregate, and overkill when they are lookups.

How do you evaluate a RAG system? Split retrieval from generation and diagnose in that order.

For retrieval, measure Recall@k (what fraction of the needed passages appear in the top k) and mean reciprocal rank (MRR, how high the first correct passage sits). For generation, measure faithfulness to the retrieved passages and relevance to the question. End to end, measure citation accuracy.

If Recall@10 is 0.4, no prompt change will fix the answers — the passage was never in the context, and everything downstream is an argument about the wrong layer. One number partitions the problem, which is why it is the first thing to measure.

7. Multi-agent

Multi-agent questions turn on when a second agent earns its cost, what a subagent actually is, how agents exchange information, and the specific ways multi-agent systems make things worse.

Single-agent vs. multi-agent — when do you need a second agent? You need a second agent when one agent’s context floods with tool output it does not need to keep. Subagents each burn a big window of their own and return a small summary, so the orchestrator never sees the noise.

The compression ratio is the value. Work it out: 6 workers reading 60k tokens each is 6 x 60,000 = 360,000 tokens explored. They return 6 summaries of ~800 tokens, which is 6 x 800 = 4,800 tokens the orchestrator has to hold. That is 360,000 / 4,800 = 75x compression. A single agent physically cannot read 360k tokens of source material and still reason well over it.

If your compression ratio is under ~5x, you are paying 4-15x for orchestration and buying nothing with it.

Follow-up they’ll ask: “Where does 4-15x come from?” — For w workers each running s steps, the orchestrator pays one decompose call, plus w summaries in, plus one synthesis call; each worker separately pays a loop that is quadratic in s over its own window. The total is dominated by w times the worker loop. With 4-6 workers at 40-60k tokens each you land in the 4-15x range, and the levers are worker count, worker step budget, and worker model tier.

What are subagents? Subagents are agents spawned by another agent with a scoped task and their own isolated context window.

They return summaries rather than transcripts — that is the definition, not an implementation detail, because a subagent that returns its transcript has bought you nothing.

In LangGraph the clean form is a compiled subgraph exposed to the parent as a tool, which makes the isolation structural rather than a convention someone has to remember to follow.

How do agents communicate? There are three mechanisms.

• Message passing is explicit and auditable, but relayed content costs tokens twice — once in the sender’s output and again in the receiver’s input.
• Shared state is cheap but needs locking when several agents write.
• The filesystem keeps large artifacts out of everyone’s context entirely.

The filesystem is underrated: write report_a.md, return one line saying where it is. A2A (Agent-to-Agent) is the emerging cross-organization standard for this, analogous to what MCP does for tools.

When does multi-agent make things worse? Overlapping scopes cause duplicated work; concurrent writers corrupt shared state; and a synthesizer that averages contradictions produces something true of neither input — "the rate limit is in the hundreds of requests per minute" is the canonical useless output.

Read-only fan-out — running many agents in parallel — is much safer than write fan-out. Parallel research is fine, while parallel editing needs filesystem or version-control isolation per agent.

Fix the synthesizer by instructing it to surface conflicts with their provenance — which worker said what, and from which source — rather than reconcile them, and give its output schema a conflicts field so it has somewhere to put them.

What makes a good subagent brief? Four things: an objective stated in one checkable sentence, an explicit scope boundary saying what not to touch, an output contract that is a schema rather than “report back”, and a budget.

“Research the auth module” handed to three workers produces three overlapping reports; a scoped brief produces three pieces that compose. The scope boundary is the part people skip, and it is the one that prevents duplicated work structurally rather than by luck.

8. Reliability and safety

Reliability comes down to what to do when things fail, where guardrails have to live to actually hold, the security model of an agent that reads untrusted text, and how to keep a human in the loop without leaking resources.

How do you handle agent failures and implement error recovery? Split failures by class, because the two classes need opposite treatment.

Retry infrastructure failures inside the harness — HTTP 429 rate limits, 503 unavailable, timeouts — using exponential backoff with jitter, meaning each retry waits longer and by a randomized amount. Jitter matters on its own: without it, N concurrent clients retry in lockstep and re-spike the service that just failed.

Return semantic failures to the model as a tool_result with is_error: true. The model usually self-corrects within one turn, because "Error: city 'Pariss' not found. Did you mean 'Paris'?" is better in-context instruction than anything you could write in the system prompt.

Track a consecutive-failure budget separately from the step cap: five failures in a row means the environment is broken, not the reasoning.

Follow-up they’ll ask: “Which errors should you never retry?” — Any 4xx other than 429. A 400 means your request is malformed, and it will be malformed again the next time; retrying it burns budget and delays the real fix.

How do you implement guardrails to prevent harmful actions? Guardrails live in the harness, not in the prompt: an authorization check before every tool call, sandboxed execution, output validation, and loop and budget guards. Prompt rules are advisory — 1% non-compliance on a destructive action is unacceptable.

For irreversible actions there is a ladder, strongest first:

1. Read-only credentials against a replica (a copy of the database), where a DROP is not denied but impossible.
2. Production credentials simply absent from the environment.
3. Soft delete instead of hard delete.
4. A two-phase commit in which apply_change requires an id that only an approver can mint.
5. Explicit human confirmation.
6. Per-action-class rate limits.

Reach for the highest rung you can afford.

What are the security risks of agentic systems? The list is prompt injection, where text the agent reads carries instructions it then follows; data exfiltration, where private data leaves the system; over-privileged tools; unsafe code execution; cross-tenant leakage, where one customer’s data reaches another; and supply-chain risk from third-party MCP servers.

The framing that ties them together is that the agent’s authority should never exceed the trust level of the least-trusted content it reads.

The mechanism behind injection specifically is that there is no privileged channel: the system prompt, the user message, a tool result, and a retrieved document all become one flat sequence of tokens, and attention runs over all of it uniformly. Role labels are conventions the model was trained to weight, not an enforcement boundary.

What is the lethal trifecta? The lethal trifecta is access to private data, plus exposure to untrusted content, plus the ability to communicate externally.

Any two of the three are usually survivable; all three together form an exfiltration channel — untrusted text instructs the agent to read secrets and send them out.

Break one leg, typically with an egress allowlist that fixes the set of outbound destinations, or by splitting reading and sending across two agents that do not share both capabilities. You cannot make injection impossible; you can make a successful injection harmless, and that reframing is the answer they are listening for.

Follow-up they’ll ask: “Doesn’t wrapping untrusted content in tags fix it?” — It reduces the success rate and is worth doing. It is not a boundary, because the tags are also just tokens in the same flat sequence. Rank it fourth, behind the egress allowlist, capability splitting, and human confirmation on irreversible externally-triggered actions.

What is the human-in-the-loop pattern, and when is it needed? It comes in four flavors: approve, which gates a call before it runs; edit, which lets a human fix the arguments; review, which checks the output after the fact; and escalate, which hands the whole task over.

You need it for irreversible actions, for high blast radius (how much damage one bad action can do), for high cost, and for low model confidence.

Two engineering requirements get forgotten: durable state across the pause, because the process will restart while you wait, and a mandatory timeout branch, because “waiting for approval” forever is a resource leak and a silently dropped task.

How do you manage token consumption in long-running workflows? Keep a budget ledger that charges the real usage numbers after every call, including the cache multipliers — writes at 1.25x and reads at 0.10x, because charging reads at full price makes caching look worthless in your own dashboard.

Warn at 80% of the budget and hard stop at 100%. Add a separate task budget that the model can see, so it wraps up gracefully instead of being cut off mid-edit.

A budget-exhausted run must return partial results and state what is missing — the most dangerous line of code in an agent is the one that returns success on the exhausted path.

9. Evaluation and production

In production the questions become: how to test something whose output changes between identical runs, how to make an LLM judge trustworthy, what to watch once it is live, and the order in which to attack cost.

How do you evaluate and test AI agents? Evaluate the outcome strictly and the trajectory loosely: assert that a forbidden tool was never called and that a call budget was respected, but not that the exact sequence of steps matched, because the path is legitimately non-deterministic.

Use code assertions wherever a criterion can be checked in code, and a calibrated LLM judge only for genuinely subjective dimensions.

Start with 20 hand-written cases and grow the set from production failures, so that every incident becomes a permanent case. A suite that only measures what you already thought to check cannot surprise you, and surprising you is its only job.

Follow-up they’ll ask: “Your evals pass and users still complain — what’s wrong?” — Usually distribution mismatch: you sampled test cases from your mental model of the users, and the complaints are falsifying that model. Hand-grade 100 real production inputs and compare the score to the suite’s number. The size of the gap tells you how much to trust the suite at all.

How do you make an LLM judge trustworthy? Put the reasoning field before the score in the schema, because fields generate in order under constrained decoding, so a score-first schema produces a number followed by a rationalization of it.

Use discrete scales with defined anchors for each level rather than a continuous 1-10, which no two graders interpret alike.

Score one dimension per judge, because a single schema scoring five dimensions correlates them all to whichever one is generated first.

Calibrate against ~50 human labels and report the agreement rate before you trust any of the numbers.

And know the biases you are fighting: position bias (order of the candidates), length bias (longer looks better), and self-preference (a model prefers its own output).

How do you gate CI on a non-deterministic system? Run each case N=3 times and take the majority verdict. That is necessary because even at temperature 0 the output is not reproducible across runs: GPU kernels sum in an order that depends on which requests are batched together, and floating-point addition is not associative.

Gate on aggregate success across the suite rather than per case, except for safety cases, which are individually blocking.

Track a moving baseline and alert on drops that exceed the observed noise. Version the prompt, the tool set, the model id, and the index, so a regression becomes a one-minute diff instead of a week of guessing.

Follow-up they’ll ask: “Why is N=3 enough?” — It is not a statistical guarantee, it is a noise filter sized to the flake rate you actually observe. Measure the per-case pass rate across 10 repeats first; if a case flips 30% of the time, an N=3 majority still flips too often and the case itself is underspecified. Fix the case rather than raising N.

What do you monitor in production? Track task success rate, verified rather than self-reported; cost per completed task; p50 and p95 latency, the median and the 95th percentile; escalation rate; safety violations, which must be zero; and cache hit rate.

Cache hit rate is the most useful derived metric — it moves before cost does and tells you immediately whether your prompt architecture survived a deploy.

Track self-reported success and verified success as two separate series, because the gap between them is your real error rate.

Follow-up they’ll ask: “Why cost per completed task rather than per call?” — Because a cheaper agent that fails more often is more expensive overall. Per-call cost improves every time you make quality worse, which makes it an actively misleading optimization target.

How does prompt caching work, and why does it matter so much for agents? Prompt caching stores the KV cache for a prefix of your request so that prefill can be skipped on the next request that shares it. The match runs over tools, then system, then messages, in that order, and a cache_control breakpoint marks everything before it as cacheable.

Reads bill at ~10% of the input price and writes at 1.25x, so break-even is two requests: 1.25 + 0.1 = 1.35 against 2.0 uncached, in multiples of one uncached request’s input cost.

In an agent loop the conversation prefix grows and stays stable, which is the ideal shape for a prefix cache — it only ever appends, so the previous request’s prompt is a genuine prefix of this one.

Quote your own number rather than a range. The “3-10x” you will hear is a spread across tasks, and its top end is the asymptotic ceiling on the cached prefix alone, not what a whole task moves by, because output tokens and cache writes are untouchable (Prompt caching the highest leverage lever).

Worked example. Take a 20-turn agent with a P = 6,000-token prefix that adds a = 1,200 tokens per turn. This uses chapter 09’s convention — turn 1 carries no delta, so the history term is a x n(n-1)/2; never mix that with the n(n+1)/2 convention inside one calculation.

The block below has one line per token category. Read the right-hand column as “how many tokens, and at what price multiplier.” 0.10x and 1.25x are the cache read and write multipliers from the paragraph above.

uncached  20 x 6,000 + 1,200 x 190           = 120,000 + 228,000 = 348,000 tokens
reads     19 x 6,000 + 1,200 x 171           = 114,000 + 205,200 = 319,200 @ 0.10x
writes     6,000 (first prefix) + 19 x 1,200 =   6,000 +  22,800 =  28,800 @ 1.25x
check     319,200 + 28,800 = 348,000   <- reads + writes must equal the uncached total

Line by line: uncached is what you send with no caching at all. reads is everything served from cache — 19 of the 20 turns re-read the prefix, and the deltas of earlier turns get re-read 1 + 2 + ... + 18 = 171 times. writes is everything written into the cache once: the 6,000-token prefix on turn 1, plus each turn’s 1,200-token delta on the 19 turns that add one.

Now price it:

effective 319,200 x 0.10 + 28,800 x 1.25 = 31,920 + 36,000 = 67,920 tokens

348,000 / 67,920 = 5.1x, and it is an input-only ratio rather than the bill.

Two details are where people go wrong. The check line, because caching changes the price of a token and never the number of them — if your reads and writes do not sum back to the uncached total, you have double-counted. And the 6,000-token first write, because turn 1 has to write the prefix somebody later reads.

Fold output back in at chapter 09’s reference rate of 400 output tokens per call, priced at $5/MTok input and $25/MTok output:

• Uncached: 348,000 tokens x $5/MTok = $1.740 input, plus 20 x 400 = 8,000 output tokens x $25/MTok = $0.200 — $1.94 total.
• Cached: 319,200 x $0.50/MTok = $0.160 reads, plus 28,800 x $6.25/MTok = $0.180 writes, plus the same $0.200 output — $0.54 total.

That is 1.94 / 0.54 = 3.6x on the whole task, with output now 37% of the cached bill and caching unable to touch a token of it.

Follow-up they’ll ask: “Why is it model-scoped?” — Different weights produce different K and V for identical tokens, so there is no way to share a cache across models. That is also why routing to a cheaper model mid-conversation discards the whole cached prefix and often loses money overall.

What silently breaks prompt caching? The usual causes are datetime.now() or a UUID (a freshly generated unique id) in the system prompt, json.dumps without sort_keys=True so key order varies between runs, a per-user tool list, switching models mid-conversation, editing the system prompt mid-session, or a prefix shorter than the minimum cacheable length.

The last one produces no error at all — just cache_creation_input_tokens: 0 — so “there’s no exception” is not evidence that caching is working.

Verify with cache_read_input_tokens: zero across repeated calls means something is invalidating the prefix, and the fastest way to find it is to diff two rendered request payloads and look at the first differing byte.

How do you reduce agent latency? Find the dominant term first: prefill, which sets TTFT; decode, which sets TPOT; or tool execution time. Tool time usually dominates, and people usually tune the prompt instead.

Return parallel tool calls in one user message, because splitting the results across several messages turns the transcript into a demonstration of answering serially, and the model quietly stops emitting parallel calls with no error anywhere.

Use programmatic tool calling — letting the model write code that chains several tools in one go — to collapse round trips, caching to cut prefill, and streaming to cut perceived latency. Prefetch the lookup that 90% of sessions start with.

And counterintuitively, a higher reasoning effort setting sometimes reduces wall-clock time by producing fewer turns, since each turn costs a full round trip.

In what order do you optimize cost? Work down this list in order:

1. Measure.
2. Fix caching.
3. Offload and truncate tool output.
4. Cut the number of calls per task.
5. Cut tokens per call.
6. Route to cheaper models.
7. Tune reasoning effort.
8. Batch anything that is not interactive.

Never start at model routing. Downgrading an uncached agent buys ~2x and costs accuracy, while fixing caching costs nothing in quality at all.

Give a number you measured rather than the range — the example above is 5.1x on input and 3.6x on the whole task, and chapter 09’s own reference task lands at 2.9x (Prompt caching the highest leverage lever); the “3-10x” rule of thumb is a spread across tasks whose top end is the ceiling on the cached prefix alone.

The first four items are free in quality terms and the last three are not, and stating that boundary out loud is most of the answer.

10. Frameworks

Last, frameworks: what the two most common ones actually give you, what they cost you in visibility, and the honest case for using none at all.

How does LangChain work? LangChain is a library of composable abstractions — models, prompts, retrievers, tools, output parsers — wired together with LCEL, the LangChain Expression Language, a pipe syntax for chaining steps. It is good for standard pipelines and fast prototyping.

The cost is that its abstraction layers sit between you and the rendered prompt, which is exactly the thing you need to inspect when caching breaks or context grows, so you end up debugging the framework’s serialization instead of your own logic. Its agent executors have largely been superseded by LangGraph.

How does LangGraph work? LangGraph models an agent as an explicit state machine: a typed state object, nodes that update it, and conditional edges that decide where control goes next. Cycles are first-class, which is what makes agent loops natural to express rather than bolted on.

The real value is the checkpointer, the component that saves the state after every node, not the graph abstraction. It gives you durable state, resume after a crash, human-in-the-loop pauses that survive a process restart, and time-travel debugging. thread_id is the key you resume against.

Without a checkpointer, a crash at minute 39 of a 40-minute run loses everything.

When would you use no framework at all? Most of the time, for an agent built on your own tools.

The core loop is ~30 lines against the raw SDK — the vendor’s client library — and writing it yourself keeps full control over the three things that actually determine agent quality: the order in which context is assembled, which sets your cache hit rate; error handling, meaning which failures go back to the model and which get retried in code; and budget accounting.

Reach for a framework when you specifically want its checkpointing, its state machine, or its integrations — not by default, and not because “agents need a framework.”

The five sentences worth memorizing verbatim

These five come up in almost every round, and saying them precisely is worth more than any amount of surrounding detail. Each one compresses a mechanism from the sections above into a single line you can deliver without hesitating.

1. “Caching stores the KV cache, and because attention is causal, a token’s K and V depend only on itself and everything before it — so a shared prefix is reusable, and any change invalidates everything after it, and only after it.”
2. “The dividing line between a workflow and an agent isn’t tool use. It’s who decides step N+1.”
3. “end_turn means the model sampled a stop token. That’s a statement about the token distribution, not about the world.”
4. “Multi-agent buys context isolation. If your compression ratio is under 5x, you’re paying 4-15x for nothing.”
5. “I wouldn’t put that in the prompt. Prompt rules are advisory, and 1% non-compliance on a destructive action is unacceptable.”

Next: 14 — Code Lab.