Production and Cost

Agent cost is set by your design, not the provider, and a 12-call task falls 6.4× ($0.876 → $0.137) with no quality loss by moving three factors you own.

The cost identity

• cost/task = calls × tokens/call × price/token, three factors with three owners.
• n calls → owned by pattern (cap steps, cut round trips, parallel tools).
• t_bar tokens/call → owned by context (offload, compact, truncate, downsample).
• p_bar price/token → owned by model + cache (route, prompt cache, batch).
• API is stateless: every turn resends the whole history, so total input = n·P + a·n(n-1)/2 (quadratic in n). Cutting calls pays superlinearly: 12→8 calls (−33%) cut input tokens −46%.
• Say which factor you’re moving before optimizing anything.

Price list (per MTok)

• Output/input = 5× on every tier (a pricing choice informed by hardware, not derived).
• Cache read ≈ 0.10× input; cache write 1.25× (5-min TTL) or 2× (1-hour); Batch API 0.50× everything.
• Reference task uncached = $0.876 (input 86% of bill, because there are 31.5× more input tokens, not because they’re dear).

Prefill vs decode

• Prefill: whole prompt in parallel, one batched matmul, each weight read once → bound by FLOPs, thousands tok/s.
• Decode: one token per pass, re-reads all weights + entire KV cache → bound by memory bandwidth, tens tok/s.
• Batching helps prefill (shared weight read) but never decode (each request owns its KV). Loaded endpoint: prefill ~20,000 tok/s, decode ~50 tok/s.
• Consequences: shorten output before input; long input is cheap to process but costly to generate against (trim history for speed even after caching); caching attacks prefill only, never speeds decode.

Prompt caching (zero quality cost)

• Reuses stored KV for a byte-identical prefix; works because attention is causal (a change invalidates everything after it, only after it).
• Order: stable tools + system → breakpoint (cache_control) → history → new turn. A datetime/timestamp/uuid early in the prompt kills all caching.
• Break-even 1.25 + 0.10(N-1) = N → N = 1.28, so any N ≥ 2 wins; ceiling 10× (a read still costs 10%). 1-hour TTL shifts first win to N = 3.
• Cache the agent loop (grows monotonically, byte-stable); RAG prompts don’t cache (fresh docs rewrite the front) — place retrieved docs after the breakpoint.
• Reference task cached = $0.306 (2.9×). Remaining bill: output 39% + cache writes 39% = 78% caching can’t touch → next lever is terseness/fewer turns.
• Silent invalidators: non-sorted json.dumps, per-user tool list, mid-session model switch (caches are model-scoped), editing system prompt mid-session, trimming/compacting history. Fix a system change by appending a system-role message after the cached prefix. Diagnose: cache_read_input_tokens == 0 → diff two rendered requests.

Model routing

• Send each step to the cheapest model that can do it; route at task boundaries, not per turn.
• Haiku: extraction/classification/formatting. Sonnet: standard reasoning, most tool calls. Opus: planning, synthesis, hard debugging. Router (~400 in, ~20 out on Haiku) ≈ $0.0005, effectively free.
• Caveat — caches are model-scoped: a warm Opus read ($0.50/MTok) beats a cold Haiku prefill ($1.00/MTok). Switching mid-conversation discards the warm prefix and re-prefills at the new model’s cold rate (mid-history Sonnet downgrade = 3× worse).
• Safe splits (each gives the cheap model its own fresh window): router→one model; orchestrator/workers; planner→executors; generator→judge.
• Reference task routed = $0.137 (further 2.2×; end-to-end 6.4×). Routing’s value is that it enables the other levers.

Effort, latency, and other levers

• effort (API param, independent of model) moves thinking tokens, which bill as output. Higher effort can be cheaper end-to-end: better planning cuts n (quadratic factor). Sweep low/medium/high on your eval set per route.
• Wall clock = TTFT + (out − 1)·TPOT. TTFT = queue + prefill; TPOT = one decode step. Decode dominates (89% of a 40k-in/800-out call) → output length is the latency lever, caching is the cost lever. Serial tool round trips dominate a whole run.
• Stream (perceived time only; required above ~16k max_tokens). Return all parallel tool_results in ONE user message, else the model learns to stop batching and p95 doubles. Prefetch obvious lookups.
• Batch API: −50% within 24h, zero quality cost; non-interactive only. Semantic caching reuses an answer for a similar question — dangerous (c_wrong/c_call ~2,700:1); tune threshold (0.95+) against cost of being wrong, not hit rate; a latency feature, not a cost lever.
• Resilience: on 429 read retry-after, then backoff + jitter (jitter stops N workers resynchronizing). Worst-case call = timeout × (retries + 1). Fallback models start cold.
• Accounting: the four usage fields are disjoint — bill input + output + creation×1.25 + read×0.10; dropping cache terms misprices every cached call. Report cost per completed (and failed) task, never per call.

Forced optimization order

1 Measure  → 2 Fix caching → 3 Cut calls → 4 Cut tokens
           → 5 Route → 6 Tune effort → 7 Batch

• 1 Measure: every later step is a ratio needing a denominator.
• 2 Cache: only zero-quality-cost lever; reprices every model (route before caching = wrong by up to 10×/token). Derived 2.9×.
• 3 Cut calls: outer, quadratically-coupled factor.
• 4 Cut tokens: after the pattern is final.
• 5 Route: first quality-for-cost trade; needs 2 and 4 priced right.
• 6 Effort: model-specific, so after route is fixed.
• 7 Batch: orthogonal 2×, costs latency, so last.