Rapid-Fire Q&A — cheat sheet — AI Agent System Design

Almost every cost, latency, and caching rule falls out of a handful of low-level facts about how the model runs; know the mechanism and the consequence follows.

Five load-bearing mechanisms

KV cache stores each token’s Key and Value so they are never recomputed; every LLM API performance property is downstream of it.
Causal attention: token i attends only to 0..i, so its K/V depend on itself plus everything before. Caching is a prefix match; any change invalidates everything after it, and only after it.
Prefill vs. decode: prefill processes the whole prompt in parallel (compute-bound, sets TTFT); decode emits one token at a time (memory-bandwidth-bound, sets TPOT). One forward pass = exactly one token.
Quadratic attention: the Q x K^T matrix is n x n; doubling context ~4x the compute. Plus U-shaped recall (strong at start/end, weak in the middle).
Logit masking (constrained decoding): invalid tokens set to -inf before sampling, so invalid structured output is impossible by construction, not just unlikely.

Numbers to know

Token rule of thumb: ~4 chars/token English prose, worse for code. Tokenizer ships with the model, so counts differ; never use tiktoken for Claude (undercounts ~15-20%).
Output ~5x input cost. claude-opus-5: $5/MTok in, $25/MTok out.
Cache reads ~10% of input price; writes 1.25x. Break-even at 2 requests: 1.25 + 0.1 = 1.35 < 2.0.
Tools: 5-15 comfortable; measure selection accuracy past ~20.
Reflection / Evaluator-Optimizer cost: 2N+1 calls for N rounds (first draft precedes the loop); 3 rounds = 7 calls.
Multi-agent: value = compression ratio; under ~5x you pay 4-15x for nothing.
Computer use images: ceil(w/28) x ceil(h/28) tokens (28x28 patches). 1080p = 2,691 tokens high-res tier, 1,560 downscaled standard tier. Keeping last 3 images cuts image bill ~3.7x.

Workflow vs. agent

Workflow: next_step = f(current_step), code owns control flow, path fixed at design time, linear token cost N x (P+a).
Agent: next_step = model(history, tools), model owns control flow, resends full history each turn: quadratic n x P + a x n(n-1)/2.
Dividing line is who decides step N+1, not tool use.
Don’t build an agent if you fail any of: complexity (can you specify steps?), value (justifies 5-20x tokens?), viability (model good at it?), cost of error (reversible?).

Loop, stopping, harness

Agent loop: call model → if stop_reason == tool_use run tools → append full assistant message + all tool results → call again.
Stop on a verifiable predicate (“tests pass”), not end_turn (only means a stop token was sampled).
Quiet failures: max_tokens (HTTP 200, truncated), refusal (empty content, IndexError), pause_turn (partial answer).
The harness (prompt assembly, auth, retries, trimming, budget, loop detection) beats a better model with a bad harness. Authorization lives in the harness, never the prompt: 1% non-compliance on a destructive action is unacceptable.

Retrieval, safety, gotchas

Hybrid search: dense embeddings (lossy, semantic gist) miss literals; BM25 weights rare terms high and nails them. Fuse with reciprocal rank fusion, rerank with a cross-encoder. Standard shape: retrieve 50, rerank, keep 5.
Split RAG eval: retrieval (Recall@k, MRR) before generation (faithfulness, relevance). Recall@10 = 0.4 means no prompt change helps.
Lethal trifecta: private data + untrusted content + external communication = exfiltration channel. Break one leg (egress allowlist, or split read/send agents).
No privileged channel: system prompt, user message, tool result, retrieved doc are one flat token sequence; role labels are learned weights, not enforcement.
Cost optimization order: measure → fix caching → offload/truncate tool output → cut calls → cut tokens/call → cheaper models → tune effort → batch. Never start at model routing.
Caching silently breaks on: datetime.now()/UUID in system prompt, json.dumps without sort_keys=True, per-user tool list, mid-conversation model switch. Verify with cache_read_input_tokens.

Cheapest reusable rules

change one byte  --> everything after it re-prefills (before it stays free)
put datetime.now() at END of last user message, never in system prompt
temperature=0 is NOT deterministic (batched float rounding) --> assert properties, N=3 eval
reasoning field BEFORE score in schema (fields generate in order)