Most agent bugs are harness bugs, not model bugs: the model faithfully continues a context your code built wrong, so localize by finding where your intended context and actual context first diverge.
The five-part method
Answer these five in order; the common mistake is skipping to step 4.
1. Signal what observable metric/log line reveals it?
2. Mechanism the causal step, not a restatement of the symptom
3. Diagnostic one command/field that splits causes ~in half
4. Fix cheapest-in-QUALITY true fix first (not cheapest to build)
5. Verify decide beforehand what moves, and what must stay true- “Cheap” means cheap in quality, not engineering hours.
- A verify metric must move for the right reason: a loop detector that trips then fails every run renamed the bug, it did not fix it.
Bisect the harness (symptom not in the catalogue)
Get one good run and one bad run on the same input; every rung is a diff. Stop at the first that differs; prompt is last because it is the only rung a diff cannot settle.
| # | Rung | Rules out |
|---|---|---|
| i | Diff the two full serialized payloads | Templating, retrieval, trimming, tool assembly |
| ii | stop_reason, usage, is_error, content blocks | Silent truncation / tool failure / cache loss |
| iii | Calls per turn, results per turn, args-hash histogram | Repetition, dropped results, lost parallelism |
| iv | Five hashes: prompt, tool set, model id, index version, harness version | Environment drift, stale index, mid-session model switch |
| v | Read the prompt, change one thing, re-run evals | Nothing — which is why it is last |
Numbers that recur
- Cache multipliers (on input price): read
0.10x, write1.25x. Ceiling saving1.0/0.10 = 10x; a token flipped from read to write-and-never-read costs1.25/0.10 = 12.5x(worse than no cache). - Quadratic history: input(n) = P + n·a; total = n·P + a·n(n+1)/2. Double the length, ~4x the bill. Context ceiling ≈ n/2 (~10x at turn 20).
- Model routing cheaper tier is
3/5 = 0.6xprice → only1/0.6 = 1.67xsaving, not “halves the bill.” - Multi-agent buys only context isolation; worth it above ~5x compression per worker, clearly worth 20x+, useless under 5x.
- Birthday bound: 4 hex (16 bit) hash collides at ~7.3% for 100 items, ~49.6% for 300 — full digest in the guard, truncate only for printout.
- ~20 cases = order-of-magnitude “do it by hand today,” not a power calc.
First diagnostic per symptom
| Symptom | Look at first | Mechanism |
|---|---|---|
| Loops to step cap | Args-hash histogram | Transcript is a few-shot demo of itself |
| Tools disagree, answer matches neither | Was right value in context? | No precedence in the forward pass |
| Cost grows faster than turns | cache_read_input_tokens on call 2 | Quadratic resent history |
| Cost flat but 10x high | cache_read_input_tokens = 0 | Broken cache prefix (one changed byte) |
| A few tasks cost 10-50x | Cost histogram, not mean | Model can’t see its own spend |
KeyError on missing tool | Prompt capabilities vs TOOLS names | Prompt read as the action space |
| Irreversible action ran | What can the credential express? | Prompt rules are advisory |
| Wrong tool chosen | Confusion matrix | Description overlap + schema volume |
| Slow | Span breakdown (prefill/decode/tools) | Two latency terms; tools usually dominate |
| Malformed / wrong args | 20 calls read by hand vs intent | Undeclared constraints are unenforced |
| Confident but wrong | Recall@10 (0.4 retrieval vs 0.9 generation) | Embeddings lossy for rare literals |
| Worse in prod | Five-hash diff | Rendered tokens differ, source doesn’t |
| Cross-tenant leak | Identity fields in tool schemas | Model output is not a trust boundary |
| Obeyed retrieved doc | Lethal-trifecta check | No privileged channel in attention |
| Evals pass, users complain | Grade 100 real inputs by hand | You chose the eval distribution |
| Says “done,” isn’t | Run the goal predicate yourself | end_turn is a token, not a fact |
| Output cut mid-sentence | stop_reason (= max_tokens, HTTP 200) | Decode hit the cap |
| Lost parallel calls | Results per assistant turn | Message-array shape is a demonstration |
| Valid JSON, wrong content | Schema field order | Fields generate in order |
| Recall dies after rebuild | Can a document find itself? | Vectors are model-specific |
| Same prompt, different output | Diff 20 runs (near-tied tokens) | Batched float non-associativity |
Rules and gotchas
- Enforce guarantees in code, not the prompt: precedence, budgets, tenant isolation, irreversible-action approval. A prompt only shifts probabilities; the tail is where production lives.
- Loops come in three shapes: A identical repeat (hash args, trip at 3), B A/B/A cycle (hash state, trip at 2), C zero progress (track a monotone metric). A detector for one is blind to the others. Feed the trip back to the model before halting; halt only on a second trip.
- Propose-then-apply: the guarantee is that
approvedis set by a path the model cannot call (approveis not a tool), NOT that the id is unguessable — the model is handed the id. Read-only credentials beat every prompt rule. - Tenant leak fix is to delete the identity parameter and inject it at dispatch, not guard it.
- Reasoning fields first in a schema:
scorebeforereasoningrationalizes; the swap is the largest free quality win in structured output. - Verify cost per completed task, not per call. Track self-reported vs verified success; the gap is your real error rate.
- Never return success on the budget-exhausted / truncated / predicate-failed path — return a partial result plus what’s missing.
- Temperature 0 is not deterministic and there is no knob (sampling params removed). Fix the assertions (semantics, not string equality), don’t hunt the randomness.