Code Lab — cheat sheet — AI Agent System Design

An agent is a ~25-line loop around an LLM, plus nine guards that stop it running forever, overspending, leaking one tenant’s data to another, or reporting success on a failed run.

The core loop

Nothing is remembered server-side; the full context (system, tools, history) is resent every call. Each iteration does four things:

serialize (history + tools) -> prefill (reuse cached K,V) -> decode reply
   -> if tool_use: run tool, append result, loop; else return answer

Token ~4 chars; prices quoted per MTok.
Prefill reads the prompt in parallel, computes K and V per token; decode emits one token at a time.
Causal attention: a token sees earlier tokens only, so one changed byte near the start invalidates every cached K,V after it.
The message array is the agent’s memory. Every exercise-1 bug is a bug in how that array is built.

stop_reason handling

Value	Meaning	Harness must
`end_turn`	model finished	take text as answer
`tool_use`	wants tools run	run all, return all results in ONE message, loop
`max_tokens`	cut off mid-sentence	fail loudly (not a short answer)
`refusal`	declined	handle; `content` may be `[]`, so `content[0]` crashes

Check max_tokens and refusal before end_turn: a truncated answer looks like an answer, a refusal may carry no text block.

Exercise 1: the five plumbing failures

Appending only content[0].text -> loses tool_use block -> 400 on next tool_result.
Missing a result for one of N parallel calls -> unanswered id -> 400.
Splitting N results across N messages -> no error, but transcript teaches “one tool at a time” -> agent drifts ~3x slower.
Letting a tool exception escape -> kills run; instead return Error: {e} with is_error: true so the model can fix it.
No iteration cap -> the single most expensive line you can fail to write. Use for _ in range(max_steps), never while True.

The ten exercises (loop + nine guards)

#	Guard	Defends against
1	Tool-calling loop	the loop having no natural stop
2	Retry + backoff + jitter	infra failures (429/5xx); jitter breaks synchronized retry herds; never retry 4xx except 429
3	Budget ledger	runaway spend; hard `stop` + soft `warn` at 80% (injected once)
4	Loop detector	3 loop shapes: identical repeat (`hash(tool,args)`), A/B cycle (`hash(state)`), no-progress (metric); nudge first, halt only if it trips again
5	Semantic cache	repeat questions in different words; false hits worse than misses
6	Mini eval harness	measuring agents: assert properties, not exact output/path
7	Prompt cache audit	silent cache breaks (invalidator in prefix, prefix too short)
8	Context offloading	quadratic context growth; dump big results to files, keep a pointer
9	Trajectory assertions	grading the route, not just the answer; return a list of violations
10	Token-accounting decorator	attributing spend per call-site; four price terms kept separate

Prices and cache math

claude-opus-5 $5/$25, claude-sonnet-5 $3/$15, claude-haiku-4-5 $1/$5 (in/out per MTok).
Cache write = 1.25x input rate; cache read = 0.10x. Charging reads at full rate makes caching look worthless.
Cost per call = in*p_in + out*p_out + cache_write*p_in*1.25 + cache_read*p_in*0.10.
Because history is resent every turn, total input over n turns ~ n*P + a*n^2/2. The n^2 term means ~4x cost per doubling of turns once it dominates.
Report median and p95, never the mean; cost per task has a long tail.

Ordering + gotchas (integrated loop)

Semantic cache before context assembly (a hit skips the model).
Authorize before execute (use an allow-list; unknown tool -> denied).
Offload before loop guard (keeps state/args hashes stable).
Loop guard before budget (a loop is recoverable via nudge; a budget stop is terminal).
Budget check after charging the call.
Goal predicate after end_turn (end_turn is the model’s opinion; the harness verifies).
tenant is injected at dispatch, never in a tool schema, so cross-tenant reads can’t be expressed.
Every terminal path except success returns a partial with a reason — no exceptions escape the harness.
Semantic cache threshold starts high (0.98): worst false pair sits at cos 0.97.
Path containment: use realpath + commonpath, not startswith/abspath (defeats sibling-prefix and symlink escapes).
Streamed usage arrives on the final message_delta; reading resp.usage records a silent zero.