Customer Support Agent

A support agent answers from product docs, acts on accounts, and hands off to humans; cost and safety are decided by cheap routing in front of the agent, not by the agent itself.

Architecture: deterministic front, three lanes

• Harness: ordinary program code (loop, tool executor, permission checks) that no prompt can reach. Every hard rule lives here.
• Router: cheapest-model classifier that answers nothing, only sorts into a lane. Three lanes: single RAG call (FAQ), agent loop (account action), human (angry/VIP/legal).
• Deterministic if rules run before any model call and can send a ticket straight to a human.

message
  -> deterministic escalation rules (no model)  --match--> human queue
  -> router (Haiku)  --FAQ--> 1 RAG call
                     --account--> agent loop (gated tools)
                     --angry/VIP/legal--> human
  -> Grounded? --yes--> reply + citations   --no--> escalate

Cost: routing is the dominant lever

• Traffic mix assumed: 60% FAQ, 30% account, 10% human. Lanes differ 16x in cost ($0.0057 FAQ vs $0.0903 account).
• Blended model cost $0.0343/ticket = $343/day at 10k tickets. Account lane (30% of traffic) = 82% of the model bill.
• Prompt caching: reads bill at 0.1x input rate, first write 1.25x. Only the stable ~16k policy corpus is cached; volatile context sits after the breakpoint.
• Minimum cacheable prefix floors differ by model (512/1024/2048/4096); Haiku’s is highest, so the FAQ prefix silently never caches.
• Human handoff dominates everything: 12% escalation x $6/handoff = $7,200/day, ~20x the model bill. Cutting escalation 12%->8% saves $2,400/day vs $172/day from halving the whole model bill.

Safety lives in the harness, not the prompt

• Escalation rules run before the model: enterprise, legal language, open dispute, >=6 turns unresolved, falling sentiment (<-0.5), LTV >$10k. First match wins; returns a rule id (R01–R06) for audit. Rules read the account, not the message, so injection has nothing to act on.
• Identity from the session, never the model: delete customer_id from every tool schema so there is no value to validate. A field in the schema is a field the model will fill. Model output is untrusted (it’s a function of the untrusted message).
• Where an identifier cannot be removed (order_id on update_shipping/issue_refund), assert ownership and emit a security event on mismatch.
• Refund ceiling is cumulative $50, comparison >=. A per-call check is not a ceiling: one ticket runs many tool_use blocks over 8 rounds, so 100x $49.99 = $4,999 all “in policy”. A RefundLedger tracks per-order and per-ticket totals.
• Denials are a value (False, reason), not an exception; malformed args (missing key, "50.00", None, negative) all deny with a reason. Returned as is_error: true so the model can recover/escalate.
• Injection defense: escape <, >, & first (& before <, or double-encoding), then wrap in <customer_message> so the customer cannot close the delimiter.

Confidence: measure, don’t ask

• Self-reported confidence is near-worthless: the answer is already in context (causal attention), logits encode next-token uncertainty not truth, and the model was never trained to calibrate it. ~82% of replies say “high”, spanning ~18 points.
• Retrieval margin (top1 − top2 score) gives ~43 points of separation with mass spread across bins, and costs nothing extra.
• Gate on structural (computed in code) or external (separate judging call) signals: citation entailment, every-claim-cited, retrieval margin/score, search_docs called at all.

Metric: effective deflection

• Deflection alone is gameable (close everything -> 100%). Pair it with reopen rate (same issue re-filed within 7 days).
• effective deflection = deflection x (1 − reopen). 72% deflection at 19% reopen is really 58% and still cost a $6 handoff each time.
• Reopen lags 7 days, so gate releases on the offline eval suite; watch reopen as the weekly check.

Gotchas

• stop_reason == "refusal" is an HTTP 200 with empty content, not an exception. Check it before reading the reply, and escalate (don’t retry).
• Step budget range(8): eight rounds without an answer is a human’s case by definition.
• Reindexed docs keep serving from cache up to 5 min; stamp docs_index_version on every reply and invalidate on reindex.
• Safety eval rows require 100% (cross-tenant, cumulative cap, malformed-arg denial, delimiter escape); quality rows (Recall@5 > 0.95) do not.
• Cross-tenant disclosure raises no error and no odd log line — its detector is the other customer.