Thinking Models and the Token Budget

A thinking model is the same model allowed to write its working-out before the answer, in the same stream, billed at the same output rate; everything else follows from that.

The mechanism

• A forward pass is one run through the model; it produces exactly one next token from the whole sequence so far. Fixed compute per pass, same for 2+2 as for a policy question.
• Decode is the serial one-token-at-a-time loop; token 51 can’t start before token 50 exists.
• The model can’t think harder in one pass, only use more passes. Thinking writes intermediate conclusions into the token stream as plain text; each later pass reads them instead of re-deriving.
• Thinking buys serial depth, not smarter passes.

ordinary:  question ─► answer(200)
thinking:  question ─► thinking(1,800) ─► answer(200)
                        ▲ answer's first token is now pass #1,801

Request and response

• Turn on with thinking={"type": "adaptive"} — model decides how much to think per request; you set no number. On Opus 5, omitting thinking still runs adaptive.
• Response content is a list of typed blocks (ThinkingBlock, TextBlock). Filter by type; never content[0].text.
• output_tokens is thinking + answer combined (e.g. 2,000 = 1,800 + 200). One meter, one bill.
• thinking.display defaults to "omitted" — you pay for thinking but get empty text. "summarized" returns a summary only; raw chain is never returned. Read spend from usage.output_tokens, not text length.
• signature is a cryptographic stamp; resend thinking blocks verbatim in tool loops.
• budget_tokens is removed on current models (Opus 5/4.8/4.7, Sonnet 5) — sending it is a 400.

max_tokens and the truncation trap

• max_tokens covers thinking + answer together, thinking first. A ceiling sized for the answer truncates mid-sentence once thinking eats it.
• Failure is a 200 OK with stop_reason: "max_tokens" and a half-answer — not an exception. Check stop_reason on every response; treat "max_tokens" as failure.
• Unused ceiling is free (billed on tokens generated, not headroom). Set it to answer + generous thinking room.

When thinking helps: dependency depth

• Predictor is dependency depth (a conclusion needed before another), not difficulty and not volume.

Cost

• Thinking tokens billed as output tokens; no discount, no separate rate. Output = 5× input on every Claude model (1:5 ratio).
• Multiplier = (I + 5·T) / (I + 5·t), where I = input, T/t = output with/without thinking. Model-independent; only input size moves it.

• Prompt caching makes it worse: cached read = 0.1× input, so input shrinks and thinking’s share grows (6,000 → 600 gives 6.63×). Cache-write requests pay 1.25× (5-min) or 2× (1-hour), diluting thinking so the multiplier falls that one request. Optimize caching first, then re-measure thinking.

Latency

• Decode is serial: time = tokens ÷ decode rate. 10× tokens = 10× wait (200 tok ≈ 4s, 2,000 tok ≈ 40s at 50 tok/s). No dilution from input size.
• TTFT stays flat — first token still arrives fast, it’s just a thinking token. Instrument time-to-first-text-block instead.
• Interactive turns need a visible thinking state. Thinking is near-incompatible with sub-few-second SLOs; move it async.

Effort and disabling

• effort (low | medium | high | xhigh | max) replaces budget_tokens. It’s a disposition, not an amount — goes inside output_config, not top level. Getting it wrong is a 400.
• No direct thinking cap anymore; max_tokens is the only hard bound, and it truncates rather than shortens.
• Disabling is model-specific: thinking={"type": "disabled"} works on Opus 4.8/4.7 and Sonnet 5; on Opus 5 only at effort high or below; on Fable 5 it’s a 400 (omit the field, thinking still runs).

Tool-loop rule and gotchas

• Send every thinking block back unmodified, with its signature, in the next request of the same turn. Reasoning lives in the token stream; the API keeps no state.
• Dropping/editing/summarizing thinking blocks isn’t a crash — it’s an agent re-calling tools, contradicting itself, or quitting. Looks like a quality regression, is a plumbing bug.
• Thinking blocks persist as input on later requests in the turn, so agent cost exceeds single-turn arithmetic and grows with loop length.

Decision procedure

1. Count dependent steps. One deep → stop.
2. Measure error rate both ways on your own data (~100 examples).
3. Price a wrong answer — the number that actually decides it.
4. Compute the multiplier for your input size, after caching.
5. Check the latency budget.

Usual answer is routed: cheap path by default, escalate to thinking only on cases that earn the 2.29×. Routing is a one-step classification, so the router is cheap.