Production and Cost

In this lesson, we’ll take one 12-call agent from $0.876 per task down to $0.137, a 6.4× cut with no quality loss, and derive every reduction from the hardware and pricing underneath it. By the end you’ll be able to read an agent’s bill as three factors you own, name which one to move first, and defend the order out loud.

An agent’s running cost reads as the model provider’s bill, but nearly all of it is set by your own design decisions, and it can usually be cut several-fold without making the agent worse.

An agent here is a program that calls a large language model (LLM) in a loop, feeding each result back in so the model can decide the next step. Inference is the act of running the model on a prompt to produce output; it is the only thing you pay the provider’s API for, so your inference cost is your agent cost.

The loop is why cost matters. One agent task is not one model call, it is ten or twenty, so anything that makes a single call cheaper or faster multiplies across every step of every run. One fact about how a GPU reads its own memory ends up dictating which line of your agent to change first, which is why GPU behavior, inference optimization, and agent design are one stack, not three separate fields.

A token is the unit a model reads, writes, and bills in: a word or word-fragment, roughly three-quarters of an English word, so 1,000 tokens is about 750 words. Prices are quoted per million tokens (MTok).

The price list

Every dollar figure below comes from these three models, and each row gives the cost to send a million tokens in, the cost to get a million back, and the largest prompt the model accepts.

Input tokens are everything you send: system prompt, tool definitions, the whole conversation so far, and the new turn. Output tokens are what the model generates in reply. Context is the largest prompt the model accepts at all.

Three multipliers sit on top of these list prices, each one a lever a later section pulls:

• Cache read: ~10% of the input price. A token the API has already processed in a prior prompt bills at a tenth.
• Cache write: 1.25× the input price for a five-minute time-to-live (TTL, how long the cached copy survives), or 2× for a one-hour TTL. Putting something in the cache costs a premium; reading it back repays you.
• Batch API: 50% of everything, for work you do not need answered immediately.

The output/input ratio is exactly 5× on every tier. That is not three independent decisions that happened to agree; it is one hardware asymmetry showing up three times, derived in the prefill-vs-decode section.

The running example

One worked example threads through this lesson: a 12-call agent that starts at $0.876 per task and ends at $0.137, a 6.4× reduction with no quality change on the hard path (the tasks that still run every step on the strongest model). Model routing is where traffic splits into a hard path and a cheap one; until then, every task is a hard-path task.

The cost identity

One equation governs the whole chapter, and every later section moves one of its terms. The cost of one task splits into exactly three multiplied factors, each owned by a design decision of yours, none by the provider.

flowchart TD
    T["cost per task<br/>= calls x tokens/call x price/token"] --> A["calls per task<br/>owner: PATTERN"]
    T --> B["tokens per call<br/>owner: CONTEXT"]
    T --> C["price per token<br/>owner: MODEL + CACHE"]

    A --> A1["step caps, fewer round trips,<br/>parallel over serial tools"]
    B --> B1["offload, compact, truncate,<br/>downsample images"]
    C --> C1["route to cheaper tiers,<br/>prompt caching, batch API"]

    style A fill:#40916c,color:#fff
    style B fill:#40916c,color:#fff
    style C fill:#2d6a4f,color:#fff

• Calls per task is set by the pattern, the shape of the agent loop chosen in Agent Design Patterns. You move it by capping steps, cutting round trips, and running tools in parallel.
• Tokens per call is set by context, how much text goes in each prompt. You shrink it by offloading large tool outputs to files, compacting history, truncating, and downsampling images (Memory and Context Management).
• Price per token is set by model and cache: routing to cheaper tiers, prompt caching, and the Batch API.

Written out, the exact bill is a sum over calls, and the shape you reason with is its mean-field form:

cost(task) = SUM over calls c of [ in_tokens(c) x price_in + out_tokens(c) x price_out ]
           ~ n x t_bar x p_bar

That is n calls, t_bar average tokens per call, p_bar average dollars per token. The three factors have three different owners: n is an architecture decision, t_bar a context decision, p_bar a procurement decision. “Make it cheaper” is never one conversation; you have to say which factor you are moving.

The factors are coupled

The API is stateless: it remembers nothing between calls, so every turn resends the entire conversation so far. That single fact makes t_bar a function of n, not a free parameter.

Let P be the stable prefix (system prompt plus tool definitions, which never change) and a the tokens each turn appends. Call t sends P + (t-1)a input tokens. Summing over n calls, the prefix appears n times and the deltas add up to a·n(n-1)/2:

total input over n calls = n.P + a.n(n-1)/2      <- quadratic in n

The second term grows with n², and it is the one people miss: every delta you leave in gets resent by every call after it.

Cutting calls pays superlinearly. On the reference agent, P = 6,000 tokens and a = 1,200 tokens per turn:

n = 12:  12 x 6,000 + 1,200 x 66  = 151,200 input tokens
n =  8:   8 x 6,000 + 1,200 x 28  =  81,600 input tokens

A 33% cut in calls (12→8) produced a 46% cut in input tokens (151,200→81,600). The extra 13 points came from the quadratic term. This is the single best argument for fixing the pattern before fixing the prompt.

The reference task, uncached

The same agent generates 400 output tokens per call and runs entirely on claude-opus-5. Output totals 12 × 400 = 4,800 tokens:

input   151,200 x $5/1M   = $0.756
output    4,800 x $25/1M  = $0.120
                            -------
                            $0.876 per task

Every optimization below is measured against $0.876, where input is 86% of the total, not because input tokens are expensive (they are the cheap half) but because there are 31.5× more of them.

Prefill vs decode

Why are output tokens the expensive half? Why can caching never speed up generation? Why does trimming a long prompt help latency even after caching fixed the price? One hardware asymmetry answers all three. Serving one request happens in two phases with completely different performance characteristics.

flowchart TD
    subgraph PF["PREFILL - the whole prompt, one shot"]
        P1["all input tokens in parallel,<br/>one large batched matmul,<br/>each weight read ONCE for the whole prompt"]
        P3["bottleneck: GPU FLOPs<br/>thousands of tok/s"]
        P1 --> P3
    end
    subgraph DC["DECODE - one token at a time"]
        D1["compute Q for ONE new token"]
        D2["re-read every weight<br/>+ the ENTIRE KV cache"]
        D3["emit 1 token, append its K,V<br/>bottleneck: memory bandwidth, tens of tok/s"]
        D1 --> D2 --> D3 --> D1
    end
    PF --> DC

    style PF fill:#2d6a4f,color:#fff
    style DC fill:#bc6c25,color:#fff

Prefill reads every input token in parallel as one batched matmul (matrix multiplication, the operation a GPU does in bulk). Because the whole prompt goes through together, each model weight is read once for the whole prompt, not once per token, so the load is amortized across every token. The limit is raw arithmetic throughput (GPU FLOPs, floating-point operations per second), and prefill runs at thousands of tokens per second.

Decode is a loop that runs once per output token. Each pass computes a query vector Q for the new token, re-reads every weight and the entire KV cache, emits one token, and appends its key K and value V. The KV cache is the model’s stored summary of every token seen so far; producing even one new token requires streaming that whole cache from memory. So decode is limited by memory bandwidth (how fast bytes move into the compute units), not arithmetic, and runs at tens of tokens per second.

The asymmetry in one sentence: prefill amortizes one pass over the weights across the whole prompt, while decode pays a full pass over the weights and the entire KV cache to produce a single token. That is what output pricing tracks.

The decode ceiling

To see that decode is bandwidth-bound, size the cache and divide by bandwidth. For a 70B-class dense transformer using grouped-query attention (L = 80 layers, H = 8 key/value heads, head dimension d = 128, stored as 2-byte bf16 numbers):

KV per token  = 2 (K and V) x 80 x 8 x 128 x 2 bytes  ~ 320 KB
50,000-token cache                                     ~ 16 GB
time per token = 16 GB / HBM bandwidth (~3 TB/s)       ~ 5 ms  ->  ~190 tok/s

That is a hard ceiling: even with infinite arithmetic, the bytes still have to move. The roundings do not matter because the claim is about the order of magnitude, hundreds of tokens per second, not thousands and not tens.

On a real, shared production endpoint the observed numbers differ: prefill rises to roughly 20,000 tok/s and decode falls to roughly 50 tok/s. Batching is why. Concurrent requests share a single pass over the weights, so the weight read is split many ways and prefill saturates the arithmetic. But every request owns its KV cache and must stream it again for every token, so batching cannot help decode; loading an endpoint drives decode down and prefill up. That unamortizable KV read is what output tokens are priced on.

How far the hardware gets you toward 5×

The hardware establishes two things you can defend. First, output must cost more than input, by a large factor, because decode’s per-token work (a full pass over weights plus the whole KV cache) dwarfs prefill’s amortized pass. Second, the multiplier should be roughly constant across a vendor’s tiers, because all three models run the same two-phase algorithm on the same class of accelerator, which is exactly why the ratio is identical across the price table.

What the hardware does not give you is the number 5. The raw per-request throughput gap is 10× to 100×, far above 5×. Pricing lands lower because batching amortizes the weight read (shrinking the gap) and list prices carry commercial margin (a choice, not a physical constant). So 5× is a pricing decision informed by the hardware asymmetry, not a quantity derivable from it. The usable conclusion survives anyway: every rule that depends on the ratio needs only “output is several times more expensive and sets the latency floor.”

The three consequences you will use

The first row is the same principle in a different costume: a coding agent that rewrites a whole file (write) generates every line as output, while one that emits a small patch (edit) generates only the changed lines, for a fraction of the decode.

Prompt caching

Exactly one optimization costs nothing in quality: prompt caching, which attacks prefill and only prefill. It stores the KV cache for a prefix (the leading stretch of the prompt, counted from its first byte) so prefill can be skipped on the next request that starts with the same bytes.

It works because attention is causal: a token attends only to itself and the tokens before it, so a token’s K and V depend only on the text up to that point. Two consequences follow, and they are the whole of prompt caching:

1. A shared prefix is reusable: identical leading bytes produce identical K/V, so the second request loads them instead of recomputing them.
2. A change invalidates everything after it, and only after it.

The cache key is the literal bytes of the prompt. “Same prefix” means byte-identical, not merely equivalent in meaning.

Prompt ordering

Put the volatile part at the back and the prompt caches; put it at the front and it never caches.

flowchart LR
    subgraph GOOD["Cacheable"]
        A1["tools (stable)"] --> A2["system (stable)"] --> AC{{breakpoint}} --> A3["history"] --> A4["new turn"]
    end
    subgraph BAD["Never caches"]
        B1["system + timestamp"] --> B2["tools"] --> B3["history"]
    end

    style AC fill:#2d6a4f,color:#fff
    style B1 fill:#9d0208,color:#fff

The breakpoint is an explicit marker telling the API “cache everything up to here.” Place it after the stable tools and system prompt and before the history and new turn (the parts that grow every request). A system + timestamp block at the very front breaks caching entirely, because every request differs a few dozen tokens in and nothing behind that survives.

In code, the breakpoint is a cache_control marker on the last block you want cached:

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=8192,
    system=[{"type": "text", "text": BIG_STABLE_PROMPT,
             "cache_control": {"type": "ephemeral"}}],   # tools + system cached
    tools=TOOLS,                                          # sorted, deterministic
    messages=messages,
)
print(resp.usage.cache_read_input_tokens)   # 0 across repeats -> an invalidator

Break-even

Caching is not free: the first request pays a write premium, later requests pay a small read. Measuring everything in units of one uncached prefill (uncached = 1.0, write = 1.25, read = 0.10), break-even for N requests sharing a prefix is 1.25 + 0.10(N-1) = N, which solves to N = 1.28. So any N ≥ 2 wins:

• N = 2: cached 1.35 vs uncached 2.00, a 32% saving.
• N = 10: cached 2.15 vs uncached 10.00, 4.7×.
• N → ∞: 10×, the ceiling, because a read still costs 10%.

The one-hour TTL costs 2× to write, which shifts break-even to N = 2.11, so its first win is N = 3. Pick the long TTL only for prefixes you know get re-hit across a gap longer than five minutes, such as a nightly batch over a shared corpus, not a chat session.

There is also a floor. The minimum cacheable prefix is model-dependent and is not ordered by generation; below it, caching silently does nothing (no error, cache_creation_input_tokens: 0):

Why the agent conversation is the ideal target

A prefix is worth caching only if it grows monotonically (only ever appends, never rewrites) and stays byte-stable. An agent loop has both: turn t’s message array is turn t-1’s array plus an append, and tool schemas and history are immutable once written.

A retrieval-augmented generation (RAG) prompt, one that searches a document store each turn and pastes the results in, has neither. Fresh documents arrive every turn and land near the front, so the prefix is rewritten instead of appended to. That is why the agent loop is where caching pays several-fold and a retrieval prompt is where it pays almost nothing. The fix, when you do use retrieval, is to place retrieved documents after the breakpoint.

The reference task, cached

Give the 12-call agent a rolling breakpoint that moves to the end of what has been written on every call. Then each call reads whatever existed at the previous call and writes only its 1,200 fresh tokens:

reads  = 132,000 tokens @ 0.10x     ->  132,000 x $5/1M x 0.10 = $0.066
writes =  19,200 tokens @ 1.25x     ->   19,200 x $5/1M x 1.25 = $0.120
output =   4,800 tokens             ->    4,800 x $25/1M       = $0.120
check:   132,000 + 19,200 = 151,200 = total input  OK          -------
                                                               $0.306 per task

Always write the check line: reads plus writes must equal the 151,200 input tokens from the cost identity, because caching changes the price of each token, never the number.

$0.876 → $0.306, a 2.9× cut for zero quality change (4.1× on the input half alone). Reading the new bill apart shows where the remaining money sits:

Output and cache writes are 78% of the bill that caching cannot touch. That is your signal that the next lever is shorter outputs and fewer turns, not more caching. Re-derive this split after every optimization; it tells you what to do next.

Silent invalidators

Every entry below changes a byte in the prefix, so by the causal-attention argument everything after that byte stops being reusable. The right column is how early the change lands, and therefore how much is destroyed.

The system-prompt row has a fix worth knowing: append a {"role": "system", ...} message to messages[] instead of editing the top-level system field. It sits after the cached prefix, so the history survives, and it still carries operator authority.

To diagnose: if cache_read_input_tokens == 0 across repeated requests that should share a prefix, dump the rendered bytes for two consecutive calls and diff them. The first differing byte is the bug. That is a five-minute fix; the win depends on your own N and prefix share (4.7× at N = 10 on the cached prefix, approaching 10×, or 2.9× on this lesson’s whole task once untouchable output and writes fold back in). Quote your own number, not the “3-10×” range.

Model routing

Caching cut the price of the tokens you resend; routing cuts the price of the model serving them, because not every step needs the smartest model. Send each step to the cheapest model that can do it, with one caveat that makes the naive version backfire.

flowchart TD
    T([Task]) --> C{"Router - Haiku<br/>~400 in, ~20 out"}
    C -->|"extraction, classification,<br/>formatting"| H["Haiku 4.5<br/>$1 / $5"]
    C -->|"standard reasoning,<br/>most tool calls"| S["Sonnet 5<br/>$3 / $15"]
    C -->|"planning, synthesis,<br/>hard debugging"| O["Opus 5<br/>$5 / $25"]

    style C fill:#bc6c25,color:#fff
    style H fill:#2d6a4f,color:#fff
    style S fill:#40916c,color:#fff
    style O fill:#95d5b2,color:#000

Haiku handles work where the answer is already in the input (extraction, classification, formatting). Sonnet handles standard reasoning and most tool calls. Opus handles planning, synthesis, and hard debugging. The router is a small model call that classifies the task and names a destination.

The router has to earn its own cost. It reads ~400 tokens and emits ~20 on Haiku, so 400 × $1/1M + 20 × $5/1M = $0.0005, five hundredths of a cent, 0.16% of the $0.306 cached task. Effectively free; only the shares and branch costs matter.

To see what routing is worth, suppose 60% of tasks are simple lookups that resolve in 3 calls on Haiku (same P and a, 800 output tokens each). Priced the same cached way, that path costs about $0.024. Blending it with the 40% hard path at $0.306:

blended = $0.0005 + 0.60 x $0.024 + 0.40 x $0.306  ~  $0.137 per task

$0.306 → $0.137, a further 2.2×. End to end, $0.876 → $0.137, a 6.4× reduction with no change to the hard path. For a second data point, the Customer Support Agent case study attributes 2.7× to routing alone on the model bill and 16.3× to routing, tiering, and caching together. Routing’s real leverage is that it enables the other levers, not that it beats them.

The caveat: caches are model-scoped

The naive rule (“use the cheap model whenever the turn looks easy”) loses money, because a cached token and a fresh token are priced completely differently. Re-price each model by its effective input cost (list price times cache multiplier):

A warm Opus prefix (already cached, read at 10%) is half the price of a cold Haiku prefix (nothing cached, whole prompt prefilled at full price). Caches are model-scoped: the K/V Opus computed are invalid for Haiku, because they came from different weights. So switching models mid-conversation throws the warm prefix away and re-prefills the entire history at the new model’s cold rate.

At turn 20 of the reference agent (28,800 accumulated tokens), staying on warm Opus costs about $0.031, switching to cold Sonnet costs about $0.092 (3× worse), and switching to cold Haiku is about $0.031, a wash while giving up two tiers of intelligence. The bigger model is cheaper because it kept its cache, and the gap widens every turn as the discarded prefix grows.

Route at task boundaries, not per turn. These splits all respect that, because each gives the cheap model its own fresh window (its own conversation, warming its own prefix):

Effort and thinking

Routing decides which model answers; a second dial decides how much it reasons before answering. That dial is effort, an API request parameter independent of the model. It moves the number of thinking tokens the model generates (internal reasoning produced before the visible answer), which bill as output tokens, the expensive half. So raising effort raises the expensive half of the bill.

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    thinking={"type": "adaptive"},          # model decides per request how much to reason
    output_config={"effort": "medium"},
    messages=messages,
)

Higher effort can be cheaper end to end. Effort raises t_bar, but a model that plans better takes fewer steps, lowering n, the factor with the quadratic coupling. On the reference agent, dropping 12 calls to 8 removes ~69,600 input tokens, worth about $0.35 saved; buying that with, say, 300 extra thinking tokens on each of the 8 surviving calls costs about $0.06. That is a ~6:1 win for the more expensive setting. Put both sides in dollars before assuming low effort is cheap, and measure cost per completed task, never per call.

Effort defaults carried over from another model are almost never right. Sweep low, medium, and high across your evaluation set (the fixed collection of tasks with known-good answers) and pick a level per route.

Latency

A cheaper agent is not automatically a faster one; latency and cost respond to different levers. A request’s wall-clock time splits into three stages and two metrics.

flowchart LR
    U([User]) --> Q["Queue<br/>rate limits, admission<br/>sets TTFT"]
    Q --> P["Prefill<br/>~ input tokens<br/>sets TTFT"]
    P --> D["Decode<br/>~ output tokens<br/>sets TPOT"]
    D --> R([Response])

    style P fill:#2d6a4f,color:#fff
    style D fill:#bc6c25,color:#fff

TTFT (time to first token) is how long the user stares at nothing; it is queue time plus prefill time. TPOT (time per output token) is how fast text appears once it starts; it is one decode step. Total time is TTFT + (output_tokens - 1) × TPOT, the -1 because the first token arrives at TTFT by definition.

On a concrete call, 40,000 input and 800 output tokens at the loaded-endpoint rates (prefill ~20,000 tok/s, decode ~50 tok/s):

TTFT  = 40,000 / 20,000     = 2.0 s
TPOT  = 1 / 50              = 20 ms
total = 2.0 + 799 x 0.020   ~ 18 s     (decode is 89% of it)

That is two seconds of prefill against sixteen of decode, so the levers that move the clock are the ones that touch decode:

Caching only buys 9% here for a reason worth spelling out. Caching 36,000 of the 40,000 input tokens does not make prefill free; the cached K/V still stream from memory, just ~10× faster than recomputing them. Pricing the fresh 4,000 tokens at 20,000 tok/s and the cached 36,000 at ~200,000 tok/s gives a new TTFT of ~0.4 s, a 5× cut on a term that was only 11% of the wall clock. The same cache hit cuts the bill by 74% ($0.22 → $0.058). Caching is the cost lever; output length is the latency lever. Conflating them is why people report “we added caching and it didn’t feel faster” — and they are right that it would not.

Where agent time actually goes

Zoom out from one call to a whole run, and the dominant contributor is not any single call but the number of calls.

Programmatic tool calling is where the model writes a short script that calls several tools itself, so the whole sequence runs in one round trip instead of one per tool. The reference agent’s 12 calls at 18 seconds each is 12 × 18 = 216 seconds, three and a half minutes. The outer loop dominates everything inside it, which is why “reduce round trips” outranks every per-call optimization on latency as it does on cost.

Three practical wins

1. Stream. Streaming sends tokens as they are generated. It does not reduce total time, only perceived time: the user sees output after TTFT instead of after the full decode. It also avoids HTTP timeouts and is required above roughly 16,000 tokens of max_tokens (a non-streamed request holds the connection for the whole decode, and 16,000 / 50 tok/s = 320 seconds of silence on the wire).

2. Emit all parallel tool calls in one assistant turn, and return all results in ONE user message. The failure is silent:

   # WRONG - three user messages, one result each
   for tu in tool_uses:
       messages.append({"role": "user", "content": [
           {"type": "tool_result", "tool_use_id": tu.id, "content": run(tu)}]})
   
   # RIGHT - one user message, all results
   messages.append({"role": "user", "content": [
       {"type": "tool_result", "tool_use_id": tu.id, "content": run(tu)}
       for tu in tool_uses]})

   The model is a next-token predictor conditioned on the transcript. The wrong version writes a transcript where every turn holds exactly one call, so the model learns that turns contain one call, copies that pattern, and stops batching. Nothing errors, but p95 latency (the slowest 5% of requests) doubles, and every extra round trip means an extra full-history prefill.

3. Prefetch the obvious. If 90% of sessions start with the same lookup, fire it before the model asks, converting a serial TTFT + tool into a parallel one.

Everything on at once

In production the levers are pulled together. This is the shape to copy: streaming (messages.stream), the cache_control marker, the effort setting, and the usage read that feeds cost accounting.

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    system=[{"type": "text", "text": SYSTEM,
             "cache_control": {"type": "ephemeral"}}],
    thinking={"type": "adaptive", "display": "summarized"},
    output_config={"effort": "high"},
    tools=TOOLS,
    messages=messages,
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()

u = final.usage
print(f"\nin={u.input_tokens} out={u.output_tokens} "
      f"cache_read={u.cache_read_input_tokens} stop={final.stop_reason}")

display: "summarized" is a user-experience decision. The default omits thinking text, so on a high-effort call the user gets TTFT, then a long silent gap while thinking tokens decode, then output, and streaming cannot help because there is nothing to stream during the gap. Summarized thinking fills it.

Batch and semantic caching

Two cost levers live outside the request path. One is a pure win if you can tolerate delay; the other is more dangerous than it looks.

The Batch API

The Batch API takes work you submit now and returns it within 24 hours at 50% off. It halves p_bar, so the reference task drops $0.306 → $0.153 at zero quality cost, which is rare. The entire cost is paid in latency, which makes it a product decision, not an engineering one. Use it for anything not user-facing (overnight evaluation runs, bulk classification, backfills), never for anything interactive.

Semantic caching

A semantic cache is a different animal. Prompt caching reuses computation for byte-identical text and can never change an answer; a semantic cache reuses an answer for a question that merely resembles an old one, and that can absolutely change the answer. “Resembles” is measured with embeddings (vectors positioned so similar-meaning texts sit near each other) against a similarity threshold you pick.

def cached_answer(q: str, threshold: float = 0.97) -> str | None:
    hit = vector_cache.search(embed(q), k=1)
    if hit and hit.score >= threshold and not hit.stale:
        return hit.answer
    return None

The temptation is to tune the threshold by hit rate. That is the wrong objective, because the two failure modes are wildly asymmetric. Expected loss is P_miss × c_call + P_false_hit × c_wrong, where a miss is a question the cache should have answered and didn’t (costing one displaced call, ~$0.003 on a Haiku FAQ answer) and a false hit is a stored answer served for a question it was not right for (costing a re-contact, an escalation, and trust, call it ~$8 for a support ticket).

That ratio is about 2,700:1: one false hit undoes 2,700 saved calls. Over 10,000 queries, a loose threshold (0.92, ~40% hit rate, ~3% of them false) nets roughly -$948, while a tight one (0.97, ~18% hit rate, ~0.1% false) nets roughly -$9. Loosening by 5 points turns a $9 rounding error into a $948 loss. (Those rates are illustrative; measure your own.)

The reason the shape always favors the tight setting: loosening the threshold admits questions in order of decreasing similarity, so the false-hit rate rises faster than the hit rate. Two conclusions:

1. Tune the threshold against c_wrong, not against hit rate. A dashboard showing “cache hit rate up to 40%” is reporting the wrong number.
2. Semantic caching is a latency feature that happens to save a little money, not a cost lever. Prompt caching saved a derived $0.570 per task at zero risk; the tight semantic setting nets -$9 across 10,000 queries with a tail of wrong answers. If someone proposes semantic caching as a cost fix, prompt caching is the larger and safer win.

In practice: start at 0.95 or above, exclude anything user-specific or time-sensitive, give entries a TTL, and log every hit with its similarity score so you can audit the tail.

Rate limits and resilience

Sooner or later the API says no. The question is how to retry without turning one spike into a self-sustaining one.

flowchart TD
    R([Request]) --> A{429?}
    A -->|no| OK([Response])
    A -->|yes| B["Read retry-after"]
    B --> C["Backoff + jitter"]
    C --> D{"Attempts < N?"}
    D -->|yes| R
    D -->|no| F{Fallback?}
    F -->|yes| G["Smaller model or<br/>degraded mode"]
    F -->|no| E([Fail with context])

    style G fill:#bc6c25,color:#fff

A request either succeeds or returns 429 (“too many requests”). On a 429 you read retry-after (the header telling you how long to wait), then apply backoff (wait longer after each failure, typically doubling) and jitter (a random offset on that wait) before retrying. Once attempts are exhausted, take a fallback path or fail with context (surface what was being attempted, not a bare exception).

The SDK already retries 408, 409, 429, and 5xx with backoff (max_retries=2). Add three things:

• Jitter. Without it, N workers that all hit a 429 at once back off by the same deterministic amount, wake together, and reproduce the spike. Jitter breaks the synchronization.
• A graceful degradation path. A smaller model, a cached answer, or an honest “busy, try again.” The model-routing caveat applies: a fallback model starts cold, so budget the full cold prefill.
• Awareness that timeouts multiply. Worst-case wall clock per call is timeout × (max_retries + 1). A 60-second timeout with 2 retries is 180 seconds for one call, and the reference agent makes 12 of them, so the worst case for the task is 36 minutes.

Different model tiers draw on separate rate-limit pools, so shifting traffic between them does not inherit headroom. Check the target tier’s limits before you migrate volume.

Cost accounting

None of the derivations above survive contact with production unless you can measure them. Instrument from day one; you cannot retrofit this after the bill arrives.

PRICE = {
    "claude-opus-5":   {"in": 5e-6,  "out": 25e-6},
    "claude-sonnet-5": {"in": 3e-6,  "out": 15e-6},
    "claude-haiku-4-5":{"in": 1e-6,  "out": 5e-6},
}

def cost_usd(model: str, u) -> float:
    p = PRICE[model]
    return (u.input_tokens * p["in"]
            + u.output_tokens * p["out"]
            + (u.cache_creation_input_tokens or 0) * p["in"] * 1.25
            + (u.cache_read_input_tokens or 0) * p["in"] * 0.10)

def task_report(calls: list) -> dict:
    total  = sum(cost_usd(c.model, c.usage) for c in calls)
    reads  = sum(c.usage.cache_read_input_tokens or 0 for c in calls)
    writes = sum(c.usage.cache_creation_input_tokens or 0 for c in calls)
    fresh  = sum(c.usage.input_tokens for c in calls)
    out_cost = sum(c.usage.output_tokens * PRICE[c.model]["out"] for c in calls)
    return {
        "cost_per_task": total,
        "calls": len(calls),
        "cache_hit_rate": reads / max(reads + writes + fresh, 1),  # low -> fix caching
        "output_share": out_cost / max(total, 1e-12),              # > 0.35 -> be terser
    }

The subtlety is that the four usage fields are disjoint: input_tokens excludes the cached tokens, so cost_usd simply adds all four instead of adjusting one by another. Getting this wrong is the most common cost-dashboard bug, and it fails both ways. Double-counting (adding cache fields on top of input_tokens as if it were the total) makes spend look worse than it is. Dropping the cache terms makes it look better: on the reference task, where every input token is a read or a write, input_tokens is 0, so a dashboard that ignores the cache fields reports $0.12 against a true $0.306. Writes belong in the cache_hit_rate denominator too, because a written token is one that did not hit the cache and billed at 1.25×.

Report cost per completed task, not per call. An agent making 20 cheap calls to fail is worse than one making 5 expensive calls to succeed, and per-call dashboards hide that. Track four numbers; the last is the one nobody instruments:

Optimization order

Every lever is now on the table, so the remaining question is sequence, and the sequence is forced: each step changes the inputs to the next.

flowchart TD
    M["1. Measure"] -->|"no denominator, no ratio"| C["2. Fix caching"]
    C -->|"reprices every model"| N["3. Cut calls per task"]
    N -->|"outer factor, quadratic"| T["4. Cut tokens per call"]
    T -->|"pattern is now final"| R["5. Route models"]
    R -->|"first quality trade"| E["6. Tune effort"]
    E -->|"last: costs latency"| B["7. Batch what isn't interactive"]

    style C fill:#2d6a4f,color:#fff
    style N fill:#2d6a4f,color:#fff
    style R fill:#bc6c25,color:#fff

Steps 2 and 3 are the free wins. Step 5 is where the character of the list changes: the first quality-for-cost trade. Do these in order, and stop when it is cheap enough. Do not start at step 5, because downgrading the model on an uncached, chatty agent costs accuracy and, on a long history, can cost money too (a mid-conversation Sonnet downgrade works out at 3× worse). The 10× you hear quoted for caching is the asymptotic ceiling on the cached prefix alone, not what a whole task moves by; lead with the derived 2.9×.

Conclusion

• One identity governs everything: cost per task = calls × tokens/call × price/token. The three factors have three owners (pattern, context, model+cache), and calls and tokens are coupled quadratically because the stateless API resends the whole history every turn.
• Output tokens cost several times more than input and set the latency floor, because decode re-reads all weights and the entire KV cache per token while prefill amortizes one pass over the whole prompt. The identical 5× ratio across tiers is a pricing decision informed by that asymmetry, not derived from it.
• Fix in a forced order: measure, then cache (free, 2.9× here), then cut calls (quadratic leverage), then cut tokens, then route (first quality trade), then effort, then batch.
• Prompt caching is the safe several-fold win; semantic caching is a latency feature with a dangerous tail; routing is worth most because it enables the other levers. Route at task boundaries, because caches are model-scoped and a warm prefix is worth more than a cheaper tier.
• Instrument cost per completed (and failed) task from day one; the four usage fields are disjoint, and dropping the cache terms silently misprices every cached call.

One line to remember: cost per task is calls × tokens/call × price/token, so before you optimize anything, say out loud which of those three you are about to move.

Cheat sheet

Further reading

• Anthropic, “Prompt caching”: the breakpoint mechanism, TTLs, and minimum-prefix rules.
• Anthropic, “Message Batches”: the 50%-off batch path for non-interactive work.
• Pope et al., “Efficiently Scaling Transformer Inference” (2022): the prefill/decode split and why decode is memory-bandwidth-bound.
• Kwon et al., “Efficient Memory Management for LLM Serving with PagedAttention” (2023): how the KV cache dominates serving cost, and how batching shares the weight read but not the KV read.

Next: The Agent Design Interview Playbook.