LLM Internals for Agent Builders — cheat sheet — AI Agent System Design

The handful of mechanisms inside a model that change your engineering choices.

The request

Every call sends the whole conversation in fixed order: tools → system → messages. The API is stateless, so turn N re-sends turns 1..N−1.
One user message plus the model’s reply is one turn; an agent is a model calling tools in a loop.

Attention and the KV cache

Attention lets each token read every earlier token, so cost is O(n²) in sequence length.
The KV cache stores each token’s Key/Value once, so a token is not re-processed every decode step.
Prompt caching reuses the cache for an unchanged prefix, so put the stable part (tools, system) first and the changing part last; a cache hit is roughly 10× cheaper and faster on the prefill.

Why long context hurts quality

Attention dilution: to hold 90% of attention on one token against n competitors the score gap must be g = ln(9n) — it grows with the log of n, not linearly (n = 4 → 3.6; n = 50,000 → 13).
It is not free: a fixed gap that won 0.9 at n = 4 collapses to ~0.0007 at n = 50,000, and 99% correct per layer compounds to 0.99^80 ≈ 0.45.
Lost in the middle: recall is U-shaped — strong at the start and end, down 20–40 points in the middle. Restate instructions near the end; compaction and offloading move facts out of the trough.

recall  ●                          ●     ← high at both ends
        · ●                    ●  ·
        ·    ●                ●   ·
        ·        ● ● ● ● ●        ·       ← middle trough (−20 to −40 pts)
        └──────────────────────────
        start        middle        end

Embeddings and retrieval

An embedding maps text to a fixed vector (e.g. 1,024 floats); a dimension is one learned scale, not a human label.
Pooling averages the per-token vectors into one, so a rare token gets drowned out — the reason dense search can miss a needle like ERR_4021.
Similarity is the angle, not the distance: cosine = (a·b) / (‖a‖‖b‖), in [−1, 1]. Length is divided out, so meaning wins over word count.

Sampling

Softmax turns scores into probabilities; temperature T divides the logits. T → 0 is greedy (argmax) and deterministic; higher T is more random.
top_p (nucleus): keep the smallest set of tokens whose probability sums past p, discard the rest.
temperature = 0 is near-deterministic, not guaranteed identical (floating-point, batching).

Structured output and cost

JSON mode and grammars constrain decoding token-by-token (illegal tokens are masked), so valid output is a guarantee, not a request.
Cost = input tokens × price + output tokens × price; output is far pricier, and agent cost grows quadratically with turns because each turn re-sends the whole history — which is why prompt caching and compaction matter.