Rapid-Fire Q&A

In this lesson, we’ll answer the questions AI-engineering work keeps circling back to: what a token is, what happens in one forward pass, when you should not build an agent. Each answer gives the mechanism and its consequence, not just the fact, because the mechanism is what lets you reason about the next case instead of memorizing this one. By the end you’ll be able to give the crisp version of each answer out loud, and derive the ones you have not memorized from the handful of mechanisms they all share.

Every entry stands on its own. Concepts are defined where they first appear, and where a longer derivation lives elsewhere a link points to it, but you never have to click away to follow the entry.

The map: seven mechanisms, six consequences

Almost every cost, latency, and caching rule in the first section is a consequence of a handful of low-level facts about how the model runs. The diagram maps them: a mechanism on the left, the practical rule it forces on the right. An arrow means “this mechanism is why that rule is true.”

flowchart LR
    A["Causal attention"] --> C["Prompt caching<br/>render order"]
    K["KV cache<br/>prefill vs decode"] --> C
    K --> P["Output costs ~5x<br/>TTFT vs TPOT"]
    Q["Quadratic attention<br/>history resent"] --> G["Cost growth<br/>context management"]
    U["U-shaped recall"] --> G
    E["Lossy embeddings"] --> R["Hybrid search<br/>cross-encoder rerank"]
    M["Logit masking"] --> S["Structured output<br/>tool calling"]
    F["Batched float<br/>non-determinism"] --> V["N=3 eval gating"]

Seven mechanisms produce six rules, because two rules take two mechanisms each: causal attention and the KV cache both feed prompt caching, and quadratic attention and U-shaped recall both feed cost growth. Each arrow is spelled out as an entry below. LLM internals carries the longer derivations.

LLM internals

Start inside the large language model (LLM) itself: how text becomes tokens, what a single forward pass produces, why the KV cache exists, and why nearly every cost, latency, and caching rule is a consequence of that cache.

What is a token, and why does the count depend on the model? A token is a subword unit, a chunk of text usually a few characters long. Tokens are produced by a tokenizer trained with byte-pair encoding (BPE): an algorithm that starts from individual characters and repeatedly merges the most frequent adjacent pair until it has a fixed-size vocabulary. Common words end up as a single token, while rare strings split into several: ERR_4021 becomes four tokens: ERR, _, 40, 21.

The tokenizer ships as part of the model, so the same text has different token counts on different models. Never reuse a count from one model to budget another, and never use tiktoken for Claude: it is OpenAI’s tokenizer and undercounts prose by roughly 15-20%, code by more. That range is a direction, not a constant, since it depends on which two tokenizers you compare. The rule of thumb is ~4 characters per token for English prose, worse for code and much worse for non-Latin scripts. When it matters, call count_tokens against the model you will actually use.

Why does this matter for retrieval? A rare identifier fragments into low-information pieces, which is the first half of why dense embedding search misses exact error codes. The second half is in the embeddings entry below, and the fix is in the retrieval section further down.

What happens in one forward pass? A forward pass takes a sequence of tokens as input and produces exactly one new token as output. In between, four things happen:

Each token is turned into an embedding, a list of numbers representing that token.
Those vectors run through N layers of attention plus a feed-forward network (FFN, the per-token transformation between attention layers).
The final layer emits logits: one raw, unnormalized score for every entry in the vocabulary.
Softmax converts those scores into probabilities that sum to 1, and one token is sampled from that distribution.

One pass produces exactly one token, so a 500-token response is 500 sequential passes, each taking the previous output as part of its input. That loop, and the fact that each pass must wait for the one before it, is where essentially all generation latency lives.

What is the KV cache, and why does it exist? Inside attention, every token is projected into three vectors: a Query (what this token is looking for), a Key (what this token offers to others searching), and a Value (what it contributes when matched).

To generate token n+1, the model needs the Key and Value of every token from 0 to n. Recomputing all of them at every step would make generating n tokens cost O(n^3). So once a token has been processed, its K and V are kept in GPU memory (the memory attached to the graphics processor the model runs on) and never recomputed. That store is the KV cache.

Every performance property of an LLM API, the network service you call to run the model, is downstream of this cache: the prefill/decode asymmetry, output pricing, and prompt caching all fall out of it.

Prefill vs. decode: what is the difference and why does it matter? Serving a single request happens in two phases with completely different physics. The row that drives everything else is Bottleneck.

Phase property	Prefill	Decode
What it does	Processes the whole prompt at once	Produces one output token at a time
Parallelism	All input tokens go through together	Strictly sequential, one token after another
Bottleneck	Compute, raw arithmetic throughput on the GPU	Memory bandwidth, it re-reads the whole KV cache for every token
Throughput	Thousands of tokens per second	Tens of tokens per second
What it determines	Time to first token (TTFT)	Time per output token (TPOT)

The two phases move under different levers. Prompt caching attacks prefill and does nothing for decode; shortening the output attacks decode and does nothing for TTFT. Identify which term dominates before choosing a fix.

So how do you cut latency on a long-context agent? TTFT is dominated by prefilling a long history, so cache the prefix. TPOT is set by how many tokens you ask for, so prefer an edit tool over a write tool, and stream the response so the user sees tokens as they arrive.

Why do output tokens cost roughly 5x input tokens? The ratio tracks a real difference in what the two phases cost to run. Prefill is one efficient parallel matrix multiplication over all input tokens at once. Decode is a sequential, memory-bound crawl in which every token re-reads a growing cache. Output costs more because generating a token is genuinely more expensive than reading one.

On claude-opus-5 the price is $5/MTok (per million tokens) for input and $25/MTok for output, exactly the 5x ratio. The practical consequence is that shortening output is a bigger win than shortening input, which is the entire argument for diff-based editing in a coding agent.

Why is prompt caching a prefix match rather than a general cache? Because attention is causal: token i attends to tokens 0 through i and never forward. So token i’s K and V depend only on itself and everything before it.

If tokens 0 through j of your request are identical to a previous request, their K and V are bit-for-bit identical and can be reused. The moment token j+1 differs, every token after it sits in a different preceding context, so it has different K and V, and nothing downstream is reusable. Every caching rule follows from this.

Why do cache reads cost ~10% and writes 1.25x? A read skips the prefill arithmetic but still pays to move the stored K and V into GPU memory, so it is much cheaper but not free; a write pays normal prefill plus the cost of persisting the cache. Break-even is two requests: a cached pair costs 1.25 (write) + 0.1 (read) = 1.35 against 2.0 for two uncached requests, so it is cheaper from the second request on.

Why does one changed byte invalidate everything after it? Same mechanism, run forward. Say a timestamp changes a token at position 30. Every token from 30 onward now sits in a different preceding context, so it has different K and V, and nothing from position 30 on is reusable, even though 99% of the text is unchanged. The same mechanism bounds the damage: content before the change is still cached and still free. That is why datetime.now() belongs at the end of the last user message and never in the system prompt.

Why does quality degrade in long contexts? Two distinct effects get lumped together as “context rot,” and separating them is most of the explanation.

The first is attention dilution. Softmax normalizes attention weights to sum to 1 across all positions, so when 200k tokens compete, the weight any single token can receive is bounded by the competition.

The second is positional bias. Measured retrieval accuracy across position follows a U-shaped curve, strong at the start and end, weakest in the middle. That is a learned artifact of training data in which important content clusters at the beginnings and ends of documents.

The design consequences follow directly: restate the goal near the end of the context, and offload material instead of stuffing it in.

Does a 1M-token window solve context management? No. Cost still scales with tokens, attention is still quadratic, and mid-window recall still degrades. A larger window raises the ceiling; it does not change the shape of the curve.

What does an embedding store, and what does it lose? An embedding is a fixed-length vector produced by a model trained so that texts with similar meaning land close together under cosine similarity, a closeness measure based on the angle between two vectors.

The key property is that it is a lossy compression optimized for meaning, not identity. Pooling a whole passage into ~1,024 floats, averaging its per-token vectors into one, necessarily discards detail, and the training objective decided which detail survives: semantic gist, not rare literals.

That is why the query "why am I getting ERR_4021" retrieves passages about errors in general. BM25 is the exact complement, a keyword-scoring function that weights each term by inverse document frequency, so a rare term gets a high weight precisely because it is rare.

Why isn’t temperature=0 deterministic? Setting temperature to 0 means always taking the highest-scoring token (the argmax), so the output should be reproducible, and it almost is.

The gap is that floating-point addition is not associative: adding the same numbers in a different order can change the last bit of the result. GPU kernels sum in an order that depends on batch composition, that is, on whose requests happened to be batched alongside yours. That different order gives different last-bit rounding, which occasionally flips the argmax, and once one token differs the sequences diverge.

Never build a system whose correctness depends on identical output across runs. Cache by input hash when you need stability, and assert on properties, not exact strings, in tests. This is also why eval gating in continuous integration (CI) uses an N=3 majority: run each case three times and take the verdict that wins, instead of comparing against one exact expected string.

How does structured output work? The mechanism is logit masking, also called constrained decoding. At each decode step, before a token is sampled, the runtime sets the logit of every token that would make the output invalid under your schema to negative infinity, making its probability exactly zero. (The schema here is the JSON, JavaScript Object Notation, description of the fields you require.) Sampling then draws only from what survives.

So invalid output is not merely unlikely, it is impossible by construction. Two consequences: never write a JSON-repair retry loop, because there is nothing to repair; and fields are generated in schema order, so a reasoning field must be declared before the score it justifies, or you get a rationalization of a number the model already committed to.

Then why do I still get semantically wrong output? Masking enforces shape, not intent. {"date": {"type": "string"}} makes "next Tuesday" perfectly valid. Every constraint you do not declare is one the model can violate at zero cost.

Is tool calling a separate engine? No. Tool schemas are serialized into the prompt like everything else, the model emits a structured call under the same constrained decoding, and the runtime parses that call and sets stop_reason: "tool_use". It is tokens in, tokens out, the whole way through.

Why is attention quadratic? With n tokens, attention computes a score between every pair, so the Q x K^T matrix is n x n. Doubling the context roughly quadruples the attention compute. Context length is not a free dial: a 200k-token window does not cost 2x a 100k one on the attention term, and “just put everything in context” stops being a good answer well before the model’s documented limit.

Agent fundamentals

From the model, move up one level: what an agent actually is, how its loop runs and terminates, and when you should refuse to build one.

What is an AI agent, and how does it differ from a simple LLM call? A single LLM call maps a prompt to an output once and stops. An agent runs a loop in which the model decides the next step and when to stop, based on tool results it observes along the way.

Formally, a workflow computes next_step = f(current_step) where f is code you wrote, while an agent computes next_step = model(history, tools) where f is a forward pass.

The dividing line is not tool use, since a workflow can call tools too; it is who owns the control flow. That substitution buys adaptivity and costs predictability: you cannot enumerate the possible paths, so you cannot test them exhaustively, so every guarantee has to come from invariants that hold no matter which path runs.

Then how do you test one? Test the outcome strictly and the trajectory loosely. Assert that a forbidden tool was never called and that a call budget was respected; do not assert an exact sequence of steps, because the path is legitimately non-deterministic.

What’s the difference between a workflow and an agent, in cost? In a workflow you write the control flow in code, so the path is fixed and knowable at design time. In an agent the model chooses the path at runtime. The honest argument for preferring a workflow is cost, not simplicity. Let P be the fixed prompt and a the tokens each step adds.

A workflow with N stages sends N x (P + a) tokens: linear in N.
An agent resends its entire history every turn, so over n turns it sends n x P + a x n(n-1)/2 tokens: quadratic in n.

The multiplier is set entirely by P/a. When a large fixed prefix dominates, the agent is barely above 1x; when the per-turn deltas dominate, it approaches n/2, about 10x at n = 20. A concrete point on that curve: with P = 6,000, a = 1,200, n = 20, the agent costs about 2.4x twenty independent single calls, low because the prefix is five times the delta. Prefer a workflow whenever you can draw the flowchart.

What is an agent loop, and how does it decide when to stop? Four steps repeat:

You call the model.
If the response carries stop_reason == "tool_use", you execute the requested tools.
You append the entire assistant message plus every tool result to the history.
You call the model again.

It stops on end_turn, on a verifiable goal predicate (a condition your code can check, such as “the test suite passes”), on a step cap, on a budget ceiling, on a loop detector, or on a human veto.

Prefer the verifiable predicate, because end_turn only means the model sampled a stop token, a statement about the token distribution, not about the world. “Tests pass” is checkable; “the model says it’s done” is not, and it fails silently.

Which stop reasons get forgotten? max_tokens, which returns HTTP 200 (success) with silently truncated output; refusal, where the content may be an empty list so resp.content[0] raises IndexError; and pause_turn, where the loop exits early with a partial answer and no error. All three fail quietly.

Reactive vs. proactive agents? A reactive agent runs in response to a user message. A proactive agent is triggered by a schedule, a webhook (an inbound call from another system), or a watcher, with no human waiting on the result.

A proactive agent is a distributed system with an LLM inside it. It needs idempotency keys, retries, durable state, dead-letter handling for messages that never succeed, and alerting, because a failure may be seen by nobody for hours.

The specific hazard is at-least-once delivery meeting a non-idempotent agent: the queue redelivers a message after a timeout, the agent runs a second time, and a second email goes out. The fix is an idempotency key (a unique id for the unit of work, so running it twice has the same effect as running it once) checked and written before the irreversible act, not after.

What is harness engineering? The harness is all the code around the model: prompt assembly and ordering, tool dispatch, authorization, retries, context trimming, budget accounting, loop detection, and tracing.

A better model with a bad harness loses to a worse model with a good one. A harness that lets tool output flood the context hits quality collapse around turn 30 no matter which model you use, because that failure is positional: it comes from where content sits in the window, not from how smart the model is.

Authorization in particular cannot live in the prompt, because a prompt rule is advisory, and 1% non-compliance on a destructive action is unacceptable.

Your agent is worse in production than in the demo, where do you look? Not the prompt. Pull traces and diff five hashes: the prompt, the tool set, the model id, the index version, and the harness version. Then check cache_read_input_tokens and the distribution of stop_reason values. Prompt tuning comes after the diff has told you where the two environments diverge.

When should you not build an agent? Run four checks; one “no” means drop down a tier of autonomy.

Complexity: can you specify the steps in advance? If yes, it is a workflow.
Value: does the outcome justify 5-20x the tokens and quadratic growth in them?
Viability: is the model actually good at this task type today?
Cost of error: can mistakes be caught and reversed, through tests, review, or an undo?

A common mistake: given a fixed four-step pipeline, building “an agent for it.” The right design is a workflow with an agent escape hatch for the ~5% of inputs the pipeline cannot classify.

Design patterns

Each named architecture, ReAct (reason and act), Plan-and-Execute, reflection, Evaluator-Optimizer, Reflexion, orchestration, and code generation, comes with the mechanical reason it costs what it costs, which is what lets you choose between them.

Explain the ReAct architecture. ReAct interleaves reasoning and acting: thought, then tool call, then observation, then repeat until done.

Modern native tool calling is ReAct. The thought is the model’s reasoning text, the action is a tool_use block emitted under constrained decoding, and the observation is the tool_result you send back. You no longer hand-prompt the old Thought:/Action:/Observation: text format.

The cost is 5-20x a single call and is unbounded without a step cap. ReAct is the only pattern here whose context growth is quadratic, which is also why it most needs guards.

Why does it loop? Each iteration appends a (call, result) pair to the transcript. After three near-identical pairs, the transcript has become a few-shot demonstration of repeating yourself, so a fourth identical call is the high-probability continuation, not a mistake. The loop reinforces itself, and waiting for the model to notice is not a strategy.

What is Plan-and-Execute? You plan all the steps up front with a strong model, then execute them with a cheaper one, replanning only when a step fails.

It is cheaper than ReAct over the same horizon for two reasons. Each executor step runs on a short context, the plan plus one step, so there is no quadratic term. And following an explicit instruction is far easier than deciding what to do next, so a smaller model suffices. That puts it at roughly 3-8x a single call versus ReAct’s 5-20x.

Add a recon phase of one or two cheap reads before planning, or the plan is written against an environment nobody looked at. Cap replans at ~3, because a third failed plan means the goal is wrong, not the plan.

ReAct vs. Plan-and-Execute: when each? Use ReAct when step N+1 genuinely depends on what step N returns, so no plan could have been written in advance. Use Plan-and-Execute for long horizons where ReAct’s context growth is the bottleneck, or where a human should approve the work before anything runs, since a plan is a reviewable artifact and a ReAct trajectory is not.

A hybrid, plan coarsely then run ReAct within each step, is usually best, and it is what most production coding agents do.

What is agent reflection? Reflection means generate, then critique with a separate call and a fresh context, then revise.

The fresh context is load-bearing. If you ask the same conversation “is this good?”, the draft is already in its context, so the model is predicting the continuation of a conversation in which it just produced that draft; sycophancy (the tendency to agree with whatever is in front of it) and recency both push toward approval. A separate call sees the draft as an input to judge, for the same reason a code reviewer should not be the author.

Cost is 2N+1 calls for N rounds, not 2N: the first draft is generated before the loop begins, then each round adds one critique and one revision. So a 3-round cap is 7 calls, not 6. Rounds 1-2 capture most of the gain.

What if it oscillates? Vague criteria produce a round 3 that undoes round 2. Cap the rounds, make the evaluator return structured per-criterion verdicts instead of prose, and, most important, if a criterion can be checked in code, check it in code. An LLM judge is the fallback for genuinely subjective dimensions, not the default.

What is an Evaluator-Optimizer loop? It is reflection with the criteria written down in advance and a structured verdict coming back:

One call generates a draft.
A second call, fresh context, judges it against a named list of criteria and returns a per-criterion pass/fail plus the specific issues.
The generator revises against those issues.
Repeat until the judge passes it or a round cap stops it.

Cost is 2N+1 calls for N rounds, for the same reason as reflection, so a 3-round cap is 7 calls.

Two things decide whether it works. The criteria have to be articulable, because a judge without written criteria only adds a schema to an unstated opinion. And anything checkable in code should be checked in code, not asked of the judge. The loop must return both the draft and whether it passed; a loop that hands back only the draft is indistinguishable from one that succeeded.

It is built three times over, in pseudocode, LangGraph, and the raw SDK, in building an evaluator-optimizer, which also shows the variant that judges every draft it might return: that one costs exactly 2N, and the +1 you drop is precisely the draft nobody read.

What’s the difference between Reflexion and Evaluator-Optimizer? Reflexion adds memory across attempts: the lesson survives the retry and informs future tasks. Evaluator-Optimizer improves a single artifact within a single task and keeps nothing afterward.

Reflexion only pays off if the lessons transfer, which requires the next task to be drawn from a similar distribution. Its characteristic failure is a lesson store filling with unfalsifiable junk. “Be more careful with edge cases” is pure token tax; “orders_v2 is canonical; orders is a stale view kept for reporting” is specific, falsifiable, and not obvious from the code. Every stored lesson is a permanent context tax on every future run, so cap the store and let it evict.

What is agent orchestration? Orchestration is coordinating multiple steps or agents toward one goal: decomposing the task, dispatching the pieces, aggregating the results, and resolving conflicts. It can be code-driven, which makes it a workflow, or model-driven, which makes it an orchestrator agent; the distinction is exactly whether the number and content of the subtasks are decided at design time or at runtime. Orchestrator-worker, where one coordinator hands scoped briefs to several workers, is the common shape.

Code-generating vs. tool-calling agents? A tool-calling agent picks from a fixed set of typed functions, which makes every action auditable, gateable, and one round trip long.

A code-generating agent writes a script that composes actions with loops and conditionals. That is far more expressive, and intermediate results never enter the context, a token win and a latency win at once, since three chained lookups collapse into a single round trip.

The trade is safety. Code generation needs a real isolation boundary: a container, a non-root user, dropped operating-system privileges, no host filesystem, network off by default, and a cap on wall-clock time. Tool calling does not.

Tools

Tools are where an agent touches the world: how a model actually calls one, how to write a description that fires at the right time, what MCP (Model Context Protocol) does and does not standardize, and how the cost arithmetic of computer use works.

What is tool use / function calling, and how does it enable agents? You describe each tool as a name, a description, and a JSON schema for its arguments. Those schemas are serialized into the prompt; the model emits a structured call under constrained decoding; the runtime sets stop_reason: "tool_use"; you execute the call and return a tool_result block.

That is what lets a model act, not only emit text, and the agent loop exists precisely because tool results feed the next decision. There is no separate “function calling engine” anywhere in the stack.

How do you design and define tools for an agent? Names disambiguate, schemas constrain, and the description is a prompt that decides when the tool fires; it is the highest-leverage text in your system after the system prompt itself. State the trigger condition and the boundary (“do not use for X; use Y instead”), not just what the tool does. Give one tool one job. Return structure, not prose, so the next step can branch on it.

5-15 tools is comfortable; past ~20, selection accuracy degrades from schema volume alone, because thousands of tokens of near-identical JSON dilute the attention available to any one of them. Both figures are practitioner rules of thumb, not a published threshold: nothing changes at exactly 20, and the real number depends on how distinct your descriptions are. Treat 20 as the point where you start measuring selection accuracy.

How do you know a description is bad? Build a confusion matrix of expected tool versus actually called tool across your eval set. Overlapping descriptions show up as one specific off-diagonal pair; sheer schema volume shows up as diffuse off-diagonal spray. Two different diagnoses, two different fixes.

Bash vs. dedicated tools? A general shell tool gives you breadth; dedicated tools give you control. A shell string is opaque: the harness cannot gate it, render it for a human, audit it, or parallelize it, because it cannot know what the string will do until it runs. Start with bash for reach, and promote an action into its own typed tool the moment you need any of those four properties.

What is MCP, and how does it standardize tool integration? MCP is the Model Context Protocol, an open protocol for exposing tools, resources, and prompts so that any MCP-speaking agent can use any MCP server: N+M integrations instead of N x M. That is the whole value proposition.

What it does not solve is selection accuracy, authorization, or cost. And a third-party MCP server is an untrusted input source whose tool descriptions land directly in your context, which makes it a prompt-injection surface and a supply-chain dependency at once.

How do you handle 50+ tools? Use tool search: mark tools with defer_loading: true so their schemas stay out of the prompt, and let the model query the catalog when it needs something. Matching schemas are then appended to the context instead of swapped in. Appending matters mechanically, because it leaves the existing prefix byte-identical and the prompt cache survives, whereas swapping tools in and out would invalidate the whole prefix on every query.

The alternatives are a cheap router that loads a subset per request, or namespacing many tools into fewer, broader ones with an enum parameter to pick the variant. Fix overlapping descriptions before reaching for any of this: infrastructure does nothing for two tools that genuinely mean the same thing.

How do you build a code execution agent safely? You need a real isolation boundary: a container, running as a non-root user, with operating-system privileges dropped, no host filesystem mounted, network off by default with an explicit allowlist, caps on CPU, memory, and wall-clock time, and a fresh instance per session.

Never call exec() on model output inside your own process, and do not rely on RestrictedPython or an allowlist over the abstract syntax tree; both are escapable, and “escapable” here means arbitrary code execution holding your service’s credentials.

How do computer-use agents work? The loop is: capture a screenshot, send it to the model, receive an action back (click(x,y), type, key, scroll), execute it, capture a new screenshot, repeat. Every turn ships an image, so cost per step is high and roughly constant.

There are two pricing tiers, and the numbers move with the model (both are tabulated in why images dominate computer use cost). The high-resolution tier is claude-opus-5, Opus 4.7 and later, and Sonnet 5 and later; the standard tier is every older model.

Images are billed in 28x28-pixel patches, one token per patch, so an image costs ceil(w/28) x ceil(h/28). On the high-resolution tier the caps are 2,576 px on the long edge and 4,784 tokens per image. A 1080p screenshot is under both caps, so it is not downscaled and costs ceil(1920/28) x ceil(1080/28) = 2,691 tokens. On the standard tier the caps are 1,568 px and 1,568 tokens, so 1080p is downscaled and each image costs 1,560 tokens instead.

The real cost is that each screenshot is resent on every subsequent turn until you trim it, so a 20-step run pays for a growing pile of images. Keeping only the last 3 images cuts the image-token bill by 3.7x, and the ratio is set by the trimming rule, not by the price of an image, so it holds on both tiers. Computer use is the fallback for when no API exists.

What else does trimming break? Nothing, provided you keep a text log of what each removed screenshot showed. Drop the pixels, keep the sentence. The agent needs to remember that it clicked “Submit”; it does not need the 1,500 tokens that proved it.

What are Agent Skills? A Skill is a folder containing a SKILL.md file, instructions plus optional scripts the agent can run. Only the short description sits in context permanently; the model reads the full file when the description tells it the Skill is relevant. That is progressive disclosure for instructions, the same idea defer_loading applies to tools, and for the same reason: anything permanently in context is a permanent tax on every turn, and attention is a finite pool.

What is OKF (Open Knowledge Format)? OKF is an emerging specification for packaging domain knowledge, facts, ontologies (structured vocabularies of concepts and their relations), and procedures, into a portable, model-agnostic file instead of burying it in a prompt or a vendor-specific store. The motivation is MCP’s, one layer up: MCP standardizes actions, OKF aims at knowledge. It is early, and worth flagging as such.

Memory and context

Everything the agent knows beyond the immediate prompt: the four memory types, how to order a context window, and the two techniques, compaction and offloading, that decide whether a long-running agent stays affordable.

What is AI agent memory? Agent memory is everything the agent knows beyond the current prompt. Split it two ways: what persists within a session, which is context engineering, and what persists across sessions, which is storage. A large context window removes neither need: cost scales with the tokens you send whether you use them well or not, and mid-window recall degrades long before you reach the documented limit.

What are the types of agent memory? There are four:

Working memory is the current context window.
Episodic memory is what happened: past runs, actions, and outcomes.
Semantic memory is what is true: facts, preferences, entities.
Procedural memory is how to do things: skills and learned lessons.

Short-term memory is the working kind; long-term memory is the other three. The useful test for where a piece of information belongs: does it need to survive a process restart, and does it need to still be true tomorrow?

What is context engineering? Context engineering is deciding what goes into the window and in what order. Stable content goes first, tools, then system, then messages, which is the order the API renders in by construction, and volatile content goes last.

That ordering is not a style preference; it is forced by causal attention. Caching is a prefix match, so anything you want cached has to be positionally first, and anything that changes has to come after the last cache breakpoint, the marker that says “cache everything up to here.” Context engineering therefore sets your cache hit rate, which makes it a cost lever and not just a quality one.

How do you verify it’s working? Read usage.cache_read_input_tokens on the second call of a session. Zero means something is invalidating the prefix. Nonzero writes with zero reads means your breakpoint is moving between calls. Then diff two rendered payloads and find the first differing byte.

How does context compaction work? When the transcript approaches the window limit, you summarize the early turns and continue from the summary. Keep the goal, the decisions and why they were made, open items, and unresolved errors. Drop verbose tool output, superseded drafts, and abandoned dead ends.

Compaction improves quality as well as cost, because it moves your instructions out of the mid-window dead zone where recall is weakest and back near the end where it is strong. Server-side compaction returns a compaction block that you must append back into the conversation; extracting only the text silently loses the state, and nothing errors when you get it wrong.

Compaction vs. context editing? Compaction summarizes the prefix into shorter prose; context editing clears stale tool results or thinking blocks outright, leaving nothing behind. Use editing when the old output is simply irrelevant, such as a directory listing from turn 3. Use compaction when the narrative matters, such as why you rejected approach A. Both invalidate the cache from the edit point forward, so batch them into occasional large trims instead of trimming a little every turn.

What’s the single best technique for keeping context small? Offloading. Write large tool outputs to disk and keep a one-line pointer in the context, "wrote 4,812 rows to /tmp/analysis.json (columns: id, region, revenue)".

It changes the growth rate, not just the constant. Truncation makes each turn cheaper but the context still grows with the task; offloading stops it from growing with the task at all. The agent re-reads only what it turns out to need, and a pointer at the end of the window outperforms 60k tokens of file content buried in the middle.

Doesn’t the agent lose information? Only information it can retrieve on demand, which is exactly the trade you want. Keep the pointer descriptive enough to decide whether re-reading is worth it: row count, column names, path. A useless pointer such as "saved output to a file" forces the agent to re-read everything, which is worse than not offloading at all.

How do you implement state management in complex workflows? Checkpoint after every step, not at the end, and store artifacts by reference, paths and ids, instead of as blobs inlined into the state. The state should hold the messages, the task queue, the artifact paths, the budget ledger, and the error history.

Mark a task in-progress before executing it, so a resume verifies what happened instead of blindly re-executing it; this is the same idempotency problem as in any distributed system. A crash at minute 39 of a 40-minute run should cost you one step, not the whole run.

How do you handle multi-modal inputs and outputs? Images, PDFs, and audio enter the model as content blocks alongside text, so the capability is rarely the hard part. Cost is the design constraint. On the high-resolution tier (claude-opus-5, Opus 4.7 and later, Sonnet 5 and later), the most any single image can cost is 4,784 tokens, and a plain 1080p screenshot is 2,691 tokens; on the standard tier of older models the same screenshot is downscaled to 1,560 (see the patch arithmetic in the tools section above). On top of that, every image is re-sent on every turn until you trim it, so trim old images and downsample when fidelity is not needed.

Transcribe audio first and treat the transcript as the artifact of record, because transcription errors propagate silently into every downstream step, a far worse failure mode than a loud one.

Retrieval (RAG)

Retrieval-augmented generation (RAG) means fetching relevant text and putting it in the prompt so the model answers from it. The recurring questions: why one retriever is never enough, how to chunk, when retrieval is the wrong tool, and how to evaluate the whole thing so you know which half is broken.

How does agentic RAG differ from classic RAG? Classic RAG retrieves once through a fixed pipeline and then answers. Agentic RAG makes retrieval a tool the model calls when it decides it needs to, possibly several times with refined queries, possibly not at all.

The ability to skip retrieval matters more than the ability to repeat it. Classic RAG injects irrelevant passages into “hi” and “what’s 2+2”, which costs tokens and actively degrades the answer by surrounding it with distractors.

Why do you need hybrid search? Because the two families of retriever fail in opposite directions, and the failure is mechanical.

Dense embeddings are a lossy compression optimized for semantic gist, so they find paraphrases and miss exact literals, error codes, function names, product SKUs (stock-keeping unit codes), whose tokens carry little semantic weight and get washed out when the passage is pooled into one vector. BM25 weights a rare term higher precisely because it is rare, so it nails the literals and misses the paraphrase.

Hybrid search runs both, fuses the two ranked lists with reciprocal rank fusion (which scores each document by the reciprocal of its rank in each list, so no score calibration between systems is needed), and then reranks the survivors with a cross-encoder.

Why is a reranker better than just retrieving more? A bi-encoder embeds the query and the document separately, which makes the index precomputable but blind to how the two interact. A cross-encoder feeds the query and document through the model together, so full attention runs across the pair: far more accurate, and far too slow to run over a whole corpus. Hence the standard shape: retrieve 50 cheaply, rerank those 50 expensively, keep 5.

How should you chunk? Chunk on structure, headings, functions, sections, not fixed character counts, because a boundary at 500 characters lands mid-sentence and mid-idea. Overlap between chunks exists so a fact split across a boundary still appears whole somewhere; respect the structure and you need less of it.

Contextual chunking, prepending a document-level summary to each chunk before embedding it, is the biggest single-technique win reported, and the reason is straightforward: a chunk that says “it supports up to 100 requests per minute” is unretrievable unless the chunk also says what “it” is.

RAG vs. tool call vs. fine-tuning vs. long context? These four get confused constantly. Sort them by the question to ask first: what kind of thing are you trying to give the model?

Approach	When to use it	Why
RAG	Knowledge changes and answers must be cited	The index updates without retraining the model
Tool call	There is a system of record	Order status is a database query, not a similarity search
Fine-tuning	You need a behavior or an output format	It teaches style, not facts, and does not fix hallucination
Long context	The corpus is stable and under ~100k tokens	Put it all in the prompt and cache it; reads bill at ~10%

The mistake to avoid is reaching for fine-tuning to fix factual errors. It does not fix them, and it is the most expensive available way to not fix them.

What is GraphRAG and when is it worth it? GraphRAG builds a graph of entities and their relationships across the corpus, and answers by traversing that graph instead of by similarity alone. It answers global questions (“what themes recur across all our incident reports?”) that top-k retrieval, which returns only the k chunks most similar to the query, structurally cannot: no single chunk contains the answer, and similarity search only ever returns chunks.

The cost is a heavy indexing pass up front and staleness on a corpus that changes often. It is worth it when the questions are genuinely aggregate, and overkill when they are lookups.

How do you evaluate a RAG system? Split retrieval from generation and diagnose in that order. For retrieval, measure Recall@k (what fraction of the needed passages appear in the top k) and mean reciprocal rank (MRR, how high the first correct passage sits). For generation, measure faithfulness to the retrieved passages and relevance to the question. End to end, measure citation accuracy.

If Recall@10 is 0.4, no prompt change will fix the answers: the passage was never in the context, and everything downstream is an argument about the wrong layer. One number partitions the problem, which is why it is the first thing to measure.

Multi-agent

Multi-agent questions turn on when a second agent earns its cost, what a subagent actually is, how agents exchange information, and the specific ways multi-agent systems make things worse.

Single-agent vs. multi-agent: when do you need a second agent? You need a second agent when one agent’s context floods with tool output it does not need to keep. Subagents each burn a big window of their own and return a small summary, so the orchestrator never sees the noise.

The compression ratio is the value. Six workers reading 60k tokens each explore 360,000 tokens; they return six ~800-token summaries, so the orchestrator holds about 4,800 tokens, roughly 75x compression. A single agent physically cannot read 360k tokens of source material and still reason well over it. If your compression ratio is under ~5x, you are paying 4-15x for orchestration and buying nothing.

Where does 4-15x come from? For w workers each running s steps, the orchestrator pays one decompose call, plus w summaries in, plus one synthesis call; each worker separately pays a loop that is quadratic in s over its own window. The total is dominated by w times the worker loop. With 4-6 workers at 40-60k tokens each you land in the 4-15x range, and the levers are worker count, worker step budget, and worker model tier.

What are subagents? Subagents are agents spawned by another agent with a scoped task and their own isolated context window. They return summaries, not transcripts; that is the definition, not an implementation detail, because a subagent that returns its transcript has bought you nothing. In LangGraph the clean form is a compiled subgraph exposed to the parent as a tool, which makes the isolation structural, not a convention someone has to remember.

How do agents communicate? Three mechanisms:

Message passing is explicit and auditable, but relayed content costs tokens twice, once in the sender’s output and again in the receiver’s input.
Shared state is cheap but needs locking when several agents write.
The filesystem keeps large artifacts out of everyone’s context entirely.

The filesystem is underrated: write report_a.md, return one line saying where it is. A2A (Agent-to-Agent) is the emerging cross-organization standard for this, analogous to what MCP does for tools.

When does multi-agent make things worse? Overlapping scopes cause duplicated work; concurrent writers corrupt shared state; and a synthesizer that averages contradictions produces something true of neither input ("the rate limit is in the hundreds of requests per minute" is the canonical useless output).

Read-only fan-out, running many agents in parallel, is much safer than write fan-out. Parallel research is fine; parallel editing needs filesystem or version-control isolation per agent. Fix the synthesizer by instructing it to surface conflicts with their provenance, which worker said what and from which source, instead of reconciling them, and give its output schema a conflicts field to put them in.

What makes a good subagent brief? Four things: an objective stated in one checkable sentence, an explicit scope boundary saying what not to touch, an output contract that is a schema, not “report back”, and a budget. “Research the auth module” handed to three workers produces three overlapping reports; a scoped brief produces three pieces that compose. The scope boundary is the part people skip, and the one that prevents duplicated work structurally, not by luck.

Reliability and safety

Reliability comes down to what to do when things fail, where guardrails have to live to actually hold, the security model of an agent that reads untrusted text, and how to keep a human in the loop without leaking resources.

How do you handle agent failures and implement error recovery? Split failures by class, because the two classes need opposite treatment.

Retry infrastructure failures inside the harness, HTTP 429 rate limits, 503 unavailable, timeouts, using exponential backoff with jitter, meaning each retry waits longer and by a randomized amount. Jitter matters on its own: without it, N concurrent clients retry in lockstep and re-spike the service that just failed.

Return semantic failures to the model as a tool_result with is_error: true. The model usually self-corrects within one turn, because "Error: city 'Pariss' not found. Did you mean 'Paris'?" is better in-context instruction than anything you could write in the system prompt.

Track a consecutive-failure budget separately from the step cap: five failures in a row means the environment is broken, not the reasoning.

Which errors should you never retry? Any 4xx other than 429. A 400 means your request is malformed, and it will be malformed again next time; retrying it burns budget and delays the real fix.

How do you implement guardrails to prevent harmful actions? Guardrails live in the harness, not the prompt: an authorization check before every tool call, sandboxed execution, output validation, and loop and budget guards. Prompt rules are advisory, and 1% non-compliance on a destructive action is unacceptable.

For irreversible actions there is a ladder, strongest first:

Read-only credentials against a replica (a copy of the database), where a DROP is not denied but impossible.
Production credentials simply absent from the environment.
Soft delete instead of hard delete.
A two-phase commit in which apply_change requires an id that only an approver can mint.
Explicit human confirmation.
Per-action-class rate limits.

Reach for the highest rung you can afford.

What are the security risks of agentic systems? Prompt injection, where text the agent reads carries instructions it then follows; data exfiltration, where private data leaves the system; over-privileged tools; unsafe code execution; cross-tenant leakage, where one customer’s data reaches another; and supply-chain risk from third-party MCP servers.

The framing that ties them together: the agent’s authority should never exceed the trust level of the least-trusted content it reads.

The mechanism behind injection is that there is no privileged channel. The system prompt, the user message, a tool result, and a retrieved document all become one flat sequence of tokens, and attention runs over all of it uniformly. Role labels are conventions the model was trained to weight, not an enforcement boundary.

What is the lethal trifecta? The lethal trifecta is access to private data, plus exposure to untrusted content, plus the ability to communicate externally. Any two are usually survivable; all three form an exfiltration channel, untrusted text instructing the agent to read secrets and send them out.

Break one leg, typically with an egress allowlist that fixes the set of outbound destinations, or by splitting reading and sending across two agents that do not share both capabilities. You cannot make injection impossible; you can make a successful injection harmless.

Doesn’t wrapping untrusted content in tags fix it? It reduces the success rate and is worth doing, but it is not a boundary, because the tags are also just tokens in the same flat sequence. Rank it fourth, behind the egress allowlist, capability splitting, and human confirmation on irreversible externally-triggered actions.

What is the human-in-the-loop pattern, and when is it needed? It comes in four flavors: approve, which gates a call before it runs; edit, which lets a human fix the arguments; review, which checks the output after the fact; and escalate, which hands the whole task over. You need it for irreversible actions, for high blast radius (how much damage one bad action can do), for high cost, and for low model confidence.

Two engineering requirements get forgotten: durable state across the pause, because the process will restart while you wait, and a mandatory timeout branch, because “waiting for approval” forever is a resource leak and a silently dropped task.

How do you manage token consumption in long-running workflows? Keep a budget ledger that charges the real usage numbers after every call, including the cache multipliers, writes at 1.25x and reads at 0.10x, because charging reads at full price makes caching look worthless in your own dashboard. Warn at 80% of the budget and hard stop at 100%. Add a separate task budget the model can see, so it wraps up gracefully instead of being cut off mid-edit. A budget-exhausted run must return partial results and state what is missing; the most dangerous line of code in an agent is the one that returns success on the exhausted path.

Evaluation and production

In production the questions become: how to test something whose output changes between identical runs, how to make an LLM judge trustworthy, what to watch once it is live, and the order in which to attack cost.

How do you evaluate and test AI agents? Evaluate the outcome strictly and the trajectory loosely: assert that a forbidden tool was never called and that a call budget was respected, but not that the exact sequence of steps matched, because the path is legitimately non-deterministic. Use code assertions wherever a criterion can be checked in code, and a calibrated LLM judge only for genuinely subjective dimensions. Start with 20 hand-written cases and grow the set from production failures, so every incident becomes a permanent case. A suite that only measures what you already thought to check will never catch the failures you did not anticipate.

Evals pass and users still complain, what’s wrong? Usually distribution mismatch: you sampled test cases from your mental model of the users, and the complaints falsify that model. Hand-grade 100 real production inputs and compare the score to the suite’s number; the size of the gap tells you how much to trust the suite.

How do you make an LLM judge trustworthy? Put the reasoning field before the score in the schema, because fields generate in order under constrained decoding, so a score-first schema produces a number followed by a rationalization of it. Use discrete scales with defined anchors for each level, not a continuous 1-10, which no two graders interpret alike. Score one dimension per judge, because a single schema scoring five dimensions correlates them all to whichever one is generated first. Calibrate against ~50 human labels and report the agreement rate before you trust any of the numbers. And know the biases you are fighting: position bias (order of the candidates), length bias (longer looks better), and self-preference (a model prefers its own output).

How do you gate CI on a non-deterministic system? Run each case N=3 times and take the majority verdict. That is necessary because even at temperature 0 the output is not reproducible across runs: GPU kernels sum in an order that depends on which requests are batched together, and floating-point addition is not associative. Gate on aggregate success across the suite, not per case, except for safety cases, which are individually blocking. Track a moving baseline and alert on drops that exceed the observed noise. Version the prompt, the tool set, the model id, and the index, so a regression becomes a one-minute diff instead of a week of guessing.

Why is N=3 enough? It is not a statistical guarantee, it is a noise filter sized to the flake rate you actually observe. Measure the per-case pass rate across 10 repeats first; if a case flips 30% of the time, an N=3 majority still flips too often and the case itself is underspecified. Fix the case instead of raising N.

What do you monitor in production? Track task success rate, verified, not self-reported; cost per completed task; p50 and p95 latency, the median and the 95th percentile; escalation rate; safety violations, which must be zero; and cache hit rate. Cache hit rate is the most useful derived metric: it moves before cost does and tells you immediately whether your prompt architecture survived a deploy. Track self-reported success and verified success as two separate series, because the gap between them is your real error rate.

Why cost per completed task instead of per call? Because a cheaper agent that fails more often is more expensive overall. Per-call cost improves every time you make quality worse, which makes it an actively misleading optimization target.

How does prompt caching work, and why does it matter so much for agents? Prompt caching stores the KV cache for a prefix of your request so that prefill can be skipped on the next request that shares it. The match runs over tools, then system, then messages, in that order, and a cache_control breakpoint marks everything before it as cacheable.

Reads bill at ~10% of the input price and writes at 1.25x, so break-even is two requests: 1.25 + 0.1 = 1.35 against 2.0 uncached, in multiples of one uncached request’s input cost.

In an agent loop the conversation prefix grows and stays stable, the ideal shape for a prefix cache, because it only ever appends, so the previous request’s prompt is a genuine prefix of this one.

The headline “3-10x” is a spread across tasks, and its top end is the ceiling on the cached prefix alone, not what a whole task moves by, because output tokens and cache writes are untouchable. A worked 20-turn example in the production and cost chapter lands at about 5x on input but only about 3.6x on the whole bill once output is folded back in at the standard rate, with output making up roughly 37% of the cached bill and caching unable to touch a token of it. Measure your own number instead of quoting the range.

Why is caching model-scoped? Different weights produce different K and V for identical tokens, so there is no way to share a cache across models. That is also why routing to a cheaper model mid-conversation discards the whole cached prefix and often loses money overall.

What silently breaks prompt caching? The usual causes: datetime.now() or a UUID (a freshly generated unique id) in the system prompt, json.dumps without sort_keys=True so key order varies between runs, a per-user tool list, switching models mid-conversation, editing the system prompt mid-session, or a prefix shorter than the minimum cacheable length. The last one produces no error at all, just cache_creation_input_tokens: 0, so “there’s no exception” is not evidence that caching is working. Verify with cache_read_input_tokens: zero across repeated calls means something is invalidating the prefix, and the fastest way to find it is to diff two rendered request payloads and look at the first differing byte.

How do you reduce agent latency? Find the dominant term first: prefill, which sets TTFT; decode, which sets TPOT; or tool execution time. Tool time usually dominates, and people usually tune the prompt instead.

Return parallel tool calls in one user message, because splitting the results across several messages turns the transcript into a demonstration of answering serially, and the model quietly stops emitting parallel calls with no error anywhere.

Use programmatic tool calling, letting the model write code that chains several tools in one go, to collapse round trips; caching to cut prefill; and streaming to cut perceived latency. Prefetch the lookup that 90% of sessions start with. And counterintuitively, a higher reasoning effort setting sometimes reduces wall-clock time by producing fewer turns, since each turn costs a full round trip.

In what order do you optimize cost? Work down this list in order:

Measure.
Fix caching.
Offload and truncate tool output.
Cut the number of calls per task.
Cut tokens per call.
Route to cheaper models.
Tune reasoning effort.
Batch anything that is not interactive.

Never start at model routing. Downgrading an uncached agent buys ~2x and costs accuracy, while fixing caching costs nothing in quality. The first four items are free in quality terms and the last three are not; that boundary is the whole point of the ordering.

Frameworks

Finally, frameworks: what the two most common ones actually give you, what they cost you in visibility, and the honest case for using none at all.

How does LangChain work? LangChain is a library of composable abstractions, models, prompts, retrievers, tools, output parsers, wired together with LCEL, the LangChain Expression Language, a pipe syntax for chaining steps. It is good for standard pipelines and fast prototyping. The cost is that its abstraction layers sit between you and the rendered prompt, which is exactly the thing you need to inspect when caching breaks or context grows, so you end up debugging the framework’s serialization instead of your own logic. Its agent executors have largely been superseded by LangGraph.

How does LangGraph work? LangGraph models an agent as an explicit state machine: a typed state object, nodes that update it, and conditional edges that decide where control goes next. Cycles are first-class, which is what makes agent loops natural to express, not bolted on. The real value is the checkpointer, the component that saves the state after every node, not the graph abstraction. It gives you durable state, resume after a crash, human-in-the-loop pauses that survive a process restart, and time-travel debugging; thread_id is the key you resume against. Without a checkpointer, a crash at minute 39 of a 40-minute run loses everything.

When would you use no framework at all? Most of the time, for an agent built on your own tools. The core loop is ~30 lines against the raw SDK, the vendor’s client library, and writing it yourself keeps full control over the three things that actually determine agent quality: the order in which context is assembled, which sets your cache hit rate; error handling, meaning which failures go back to the model and which get retried in code; and budget accounting. Reach for a framework when you specifically want its checkpointing, its state machine, or its integrations, not by default, and not because “agents need a framework.”

Conclusion

Five sentences compress the load-bearing mechanisms from this lesson. Everything else follows from them.

Caching stores the KV cache, and because attention is causal, a token’s K and V depend only on itself and everything before it, so a shared prefix is reusable and any change invalidates everything after it, and only after it.
The dividing line between a workflow and an agent isn’t tool use; it is who decides step N+1.
end_turn means the model sampled a stop token, a statement about the token distribution, not about the world.
Multi-agent buys context isolation; if your compression ratio is under 5x, you are paying 4-15x for nothing.
Prompt rules are advisory, and 1% non-compliance on a destructive action is unacceptable, so authorization and guardrails belong in the harness.