Thinking Models and the Token Budget

A thinking model differs from an ordinary one in a single mechanical way, and that difference has predictable consequences for accuracy, token cost, and latency. In this lesson, we’ll pin down that mechanism, work out the cost multiplier so you can compute it for your own workload, and settle when thinking is worth paying for and when it is not. By the end you’ll be able to predict which of your tasks benefit, size the multiplier for your own input, and defend the on/off call in an interview.

What arrives when you turn thinking on

You send an ordinary API request (the same system, tools, and messages you would send anyway) plus a flag saying thinking is allowed.

What comes back is two kinds of content in one response: a stretch of the model’s own working-out, then the actual answer. Both are generated the same way, by the same model, one token at a time, and both are billed at the same output-token rate. The only structural difference is that the working-out arrives in a separate slot you can hide from your user.

A thinking model is not a different architecture, a second model, or a planning module bolted on the side. It is the same model, allowed to write down its intermediate work before committing to an answer, in the same stream, at the same price per token. Everything in this chapter follows from that.

Four terms you need first

• A token is the unit of text a model reads and writes: roughly ¾ of an English word, about 4 characters. The model sees a sequence of these units, not letters. A common word like "the" is one token; "unbelievable" is split into several.
• A forward pass is one run of the input through the model’s stack of layers. It consumes the entire sequence so far and produces exactly one next token. Writing 100 tokens means 100 forward passes, each seeing everything written so far.
• Decode is that one-token-at-a-time generation loop. It is the slow half of serving a model: every token needs its own full pass, and passes cannot run ahead of each other. Token 51 cannot be computed until token 50 exists.
• The context is everything the model can see on a given pass: your prompt plus every token generated so far in this response. It is plain text as far as the model is concerned. Nothing else carries over between passes.

The machinery underneath these terms is derived in LLM Internals; you do not need it to follow this chapter.

What a thinking model is

A forward pass is a fixed amount of compute

Ask an ordinary model a question and it starts emitting the answer immediately, token by token. Token 1 of the answer is produced by a single forward pass over your question. Whatever “reasoning” happened, happened inside that one pass, a fixed amount of computation, the same for 2 + 2 as for a five-step policy question.

This is the constraint: a forward pass is a fixed amount of compute. The model cannot decide to think harder within one pass. The stack has however many layers it has, and every token gets exactly that many.

A thinking model works around the constraint the only way available: by using more passes. Before it writes the answer, it writes out its working. Each working-out token costs its own forward pass and lands in the context that later passes read.

The same request, two timelines

The diagram shows one request twice: once through an ordinary model, once through a thinking model. Each row runs left to right as time. The only difference is where the first answer token sits.

flowchart LR
    subgraph ordinary["Ordinary model"]
        Q1["question<br/>(6,000 tokens)"] --> A1["answer token 1"] --> A2["answer token 2"] --> A3["... 200 answer tokens"]
    end
    subgraph thinking["Thinking model"]
        Q2["question<br/>(6,000 tokens)"] --> T1["thinking token 1"] --> T2["... 1,800 thinking tokens"] --> B1["answer token 1"] --> B2["... 200 answer tokens"]
    end

Both rows start from the same 6,000-token question. In the ordinary row, that question is followed directly by 200 answer tokens: 200 forward passes, the first answer token produced after a single pass. In the thinking row, the identical question is followed by 1,800 thinking tokens before the first answer token. The answer token that used to be pass #1 is now pass #1,801. The question is unchanged; only the number of passes between question and answer differs.

Why extra passes help

Every pass sees the entire sequence generated so far: your question and every thinking token already written. When the model computes thinking token 1,801, it is not re-deriving from scratch. It reads its own conclusions from tokens 1 through 1,800 as plain input text and builds on them.

That converts a problem the model would have had to solve in one pass into one it can solve in 1,800 chained passes, where each pass conditions on the results of every pass before it. A question needing five dependent steps no longer has to fit inside one fixed-size computation; it gets five stretches of tokens, each reading the last.

Thinking buys serial depth with output tokens. It cannot make any single pass smarter. It can only give the model more passes, chained so each reads the one before. Every cost, latency, and quality property below follows from that.

The request and the response

What you send

You add one field to a request you would have sent anyway:

client.messages.create(
    model="claude-opus-5",
    max_tokens=4000,
    thinking={"type": "adaptive"},
    messages=[{"role": "user", "content": "Can we refund order #4021?"}],
)

thinking={"type": "adaptive"} means the model decides how much to think, per request. A trivial question gets almost no thinking; a hard one gets a lot. You are not setting a number.

What you get back

An ordinary response has one string of text. A thinking response comes back as a list of content blocks: each block one typed chunk of the response ({"type": "thinking", ...}, {"type": "text", ...}, and so on). The working-out and the answer are separate blocks in the same list:

Message(
    content=[
        ThinkingBlock(thinking="The order is 40 days old...", signature="ErUBCkYIAx..."),
        TextBlock(text="Yes — but it needs manager approval. Here's why: ..."),
    ],
    usage=Usage(input_tokens=6042, output_tokens=2000),
    stop_reason="end_turn",
)

stop_reason is the API’s own explanation of why generation stopped. "end_turn" means the model finished on its own; the other value that matters, "max_tokens", appears in Watching the budget drain below.

Four things in that response that cause bugs in production

1. content is a list, and the first block is not your answer. Code that does response.content[0].text worked before thinking and breaks the moment you enable it: the first block is a ThinkingBlock, which has no .text. Filter by block type; never index by position.

2. output_tokens is the total: thinking plus answer. It reads 2,000 here (1,800 thinking + 200 answer), not 200. There is no separate meter, and this single number is what you are billed on.

3. By default there is no readable thinking text, and you pay for it regardless. On current models thinking.display defaults to "omitted": the ThinkingBlock still arrives, but its thinking field is empty. To see anything, opt in with thinking={"type": "adaptive", "display": "summarized"}. Even then you get a summary, because the raw chain of thought is never returned. So the string you can print is empty or a few hundred tokens, while output_tokens reflects the couple of thousand actually generated. Do not estimate thinking spend from the length of the text you received; read usage.output_tokens.

4. signature is not decoration. It is a cryptographic stamp proving the block came from the model unmodified. In a tool-calling loop you must send thinking blocks back exactly as received, signature intact (see Thinking in a tool-calling loop below).

One field that is now an error

Older models took thinking={"type": "enabled", "budget_tokens": 2000}, letting you set the thinking allowance yourself as a token count. On current models (Opus 5, Opus 4.8, Opus 4.7, Sonnet 5) budget_tokens has been removed and sending it returns a 400. Thinking is adaptive; on Opus 5, omitting the thinking field entirely still runs adaptive thinking. To influence how much the model thinks, use effort (covered under Effort below), not a token count.

Watching the budget drain

To see where the token budget goes, follow one request decode step by decode step, including the failure mode it exposes.

A question with real dependent steps

Thinking has something to do only when the answer requires steps that build on each other. This one does:

Policy. Refunds are allowed within 30 days of purchase. Gold-tier customers get 60 days. Any refund over $150 needs manager approval, unless the customer has no prior refunds.

The case. Order #4021, placed 40 days ago, $180, customer is Gold tier with 1 prior refund. Can we refund it?

Answering needs four steps, and step 4 depends on steps 2 and 3.

Reading the decode trace

Below is that request decoded one pass at a time, with max_tokens=1900. The remaining column is one counter covering thinking and answer together. The key moment is pass 1,801, where the block changes from thinking to answer with only 99 tokens left.

 pass    block      token emitted           remaining of max_tokens=1900
------  ---------  ----------------------  ----------------------------
    1   thinking   "The"                                          1,899
    5   thinking   "40"                                           1,895
  310   thinking   " Gold"                                        1,590     <- step 1 settled: window is 60 days
  902   thinking   " within"                                        998     <- step 2 settled: 40 <= 60, in window
 1,404   thinking   " exceeds"                                      496     <- step 3 settled: 180 > 150
 1,800   thinking   " approval"                                     100     <- step 4 settled, depends on 2 and 3
 1,801   answer     "Yes"                                             99
 1,900   answer     " manager"                                         0     <- CEILING HIT, mid-sentence

Every token, of either kind, decrements one counter. The model spent 1,800 tokens working out that the order is inside the Gold window, that $180 is over the approval threshold, and that 1 prior refund disqualifies the exemption. That left 100 tokens to say so, against the 200 the full answer needs. It ran out at token 1,900 with the sentence unfinished.

The failure is a 200 OK, not an exception

The response comes back with stop_reason: "max_tokens" and a truncated answer. Not an exception, not an error code: a 200 OK with a half-sentence in it. Code that reads .text and moves on ships “Yes, but it needs manager” to a customer.

max_tokens is not the answer’s budget. It is the thinking-plus-answer budget, and thinking spends it first. A ceiling sized generously for a 200-token answer becomes a truncation once thinking is on, because thinking consumes most of it before the answer starts.

The same request with more headroom

Re-run the identical request with max_tokens=4000 and nothing else changed. Thinking still costs 1,800 tokens, leaving 2,200; the answer gets its full 200 and finishes at pass 2,000 with stop_reason: "end_turn", leaving 2,000 tokens unspent.

Those 2,000 unused tokens cost nothing. You are billed for tokens generated, never for headroom. max_tokens is a ceiling, not a reservation. So set it to the answer you want plus generous room for thinking, treat an unused ceiling as free (its only downside is that a runaway response can take longer to stop), and check stop_reason on every response, treating "max_tokens" as the failure it is.

Guarding it in code

That rule, as a function. answer_of either returns the answer text or raises. Content blocks are plain dictionaries here so the example runs anywhere; a real SDK gives you objects with the same type field.

def answer_of(response_content, stop_reason, max_tokens):
    """Return the answer text, or raise if thinking ate the budget."""
    if stop_reason == "max_tokens":
        raise ValueError(
            f"truncated at max_tokens={max_tokens}: thinking consumed the budget "
            f"before the answer finished — raise the ceiling and retry"
        )
    text = "".join(b["text"] for b in response_content if b["type"] == "text")
    if not text:
        raise ValueError("no text block: the model thought but never answered")
    return text

The second guard (if not text) catches the rarer case: a response that is all thinking and no answer, which is what truncation looks like when the ceiling is low enough that the model never starts answering.

Why thinking makes the model better

Accuracy improves for a specific, mechanical reason. Knowing it lets you predict which of your own tasks will benefit before testing.

Not “trying harder” — surviving intermediate results

The model is not “trying harder.” Every forward pass does the same fixed work. What changes is how many passes the answer gets to depend on, and whether intermediate results survive long enough to be used.

Take the refund question. Answering it requires holding four facts and combining them in order:

1. Gold tier ⟹ the window is 60 days, not 30.
2. 40 days ≤ 60 ⟹ inside the window.
3. $180 > $150 ⟹ approval threshold is crossed.
4. 1 prior refund ≠ 0 ⟹ the exemption does not apply, so approval is genuinely required.

Step 4 cannot be evaluated until steps 2 and 3 are known.

Without thinking, all four have to be resolved inside one forward pass, and the intermediate conclusions exist only as internal activations that vanish when the pass ends. Squeezing the chain into that fixed budget is where models produce confidently wrong answers: they latch onto the salient fact (40 > 30, so it’s too late) and miss the tier rule that overrides it. With thinking, each conclusion is written into the token stream as ordinary input text that every later pass reads. Step 4’s pass does not re-derive steps 2 and 3; it reads them.

The predictor is dependency depth, not difficulty

The predictor for whether thinking helps is not difficulty. It is dependency depth. Ask: does answering require a conclusion that must be reached before another conclusion can be attempted?

The two “no” rows are the money-savers. Extraction and classification are wide, not deep: many items, each resolved independently in a single step. Turning thinking on for a high-volume classifier multiplies the bill and buys close to nothing, because there is no chain for the extra passes to build on.

What thinking costs in money

The pricing rule, in one sentence

Thinking tokens are billed as output tokens. There is no discount and no separate rate. Everything below is arithmetic on top of that.

One turn, two ways

Take the refund turn on claude-opus-5, priced at $5 per million input tokens and $25 per million output tokens. The input is the same 6,000 tokens either way; only the output count changes.

Output tokens went up 10×. The bill went up only 2.29× ($0.080 ÷ $0.035). Here is why.

Where the multiplier comes from

Write p for the input price per token. Output is priced at 5p on every current Claude model: $5/$25 for Opus 5, $3/$15 for Sonnet 5, $1/$5 for Haiku 4.5 are all the same 1:5 ratio. With I input tokens and T output tokens, the bill for a turn is I·p + T·5p, so the multiplier is one bill over the other:

cost_with_thinking     I·p + 2000·5p     I + 10,000     6,000 + 10,000     16,000
------------------  =  --------------  =  ----------  =  --------------  =  ------  =  2.29
cost_without           I·p +  200·5p     I +  1,000     6,000 +  1,000      7,000

The p cancels. The multiplier does not depend on which model you picked (Opus, Sonnet, and Haiku all give 2.29× here) because they share the 1:5 input-to-output ratio. It depends only on how big your input is relative to your thinking: a large input dilutes thinking’s contribution; a small one exposes it.

Input size is the only dial that moves it

Every row below uses the same 1,800 thinking tokens and 200-token answer. Only the input size changes, and the multiplier swings from 1.09× to 8.5×. Each row is the same (I + 10,000) / (I + 1,000) with a different I.

Prompt caching makes this worse

Prompt caching means reusing the server’s work on a prefix you send repeatedly (a long system prompt, a policy document) instead of re-processing it every request. It has two rates:

• Reading a cached prefix costs 0.1× the normal input rate.
• Writing it the first time costs 1.25× (or 2× for the 1-hour lifetime instead of the default five minutes).

Once cached, those 6,000 input tokens bill as if they were 600. Put 600 into the same formula:

600 + 10,000     10,600
------------  =  ------  =  6.63
600 +  1,000      1,600

Caching pushed the thinking multiplier from 2.29× to 6.63×. Caching did not get more expensive. It made input nearly free and removed the term diluting thinking’s share. Both bills fell in absolute terms; thinking’s share of what remains went up.

The cache-write request runs the other way

That 6.63× is the steady state, not the whole story. On the request that writes the cache you pay the premium instead of the discount, which makes input larger and dilutes thinking more than no caching at all. The multiplier falls on the write rows and jumps on the read row; it is not monotonic.

With a five-minute lifetime the cache pays for itself from the second request; with the one-hour lifetime it takes three. If your workload reads the cache many times per write (the only reason to cache) 6.63× is the number you will live at, and the write row is a one-request transient.

The consequence is a sequencing rule: optimize caching first, then re-measure whether thinking still pays. A thinking-on decision that looked cheap at 2.29× is the dominant line item at 6.63× once caching lands, and teams that tune the two independently reach the wrong conclusion about both.

At 100,000 turns a month, the plain turn is $3,500 and the thinking turn is $8,000, an extra $4,500. Whether that is worth it is not a token question; it is When thinking pays, below.

What thinking costs in latency

Thinking hurts perceived speed more than the token count suggests, and the metric most teams watch will not show it.

Ten times the tokens is ten times the wait

Decode is serial, so wall-clock time for the generated part is tokens ÷ decode rate. Measure your own rate; it varies by model, load, and streaming. At a measured 50 tokens per second:

Unlike cost, latency gets no dilution from input size. Reading the 6,000-token prompt happens in one batched pass and is fast. It is the 2,000 serial decode steps you wait for.

Why your latency dashboard will not show it

Time to first token (TTFT), the delay between sending the request and the first token appearing, is the standard streaming latency metric, and it barely moves. The first token still arrives at about the same time; it is a thinking token now. The user watching a “thinking…” indicator is waiting for the first answer token, which arrives only after all 1,800 thinking tokens.

                       TTFT       time to first ANSWER token
Without thinking       ~0.3 s     ~0.3 s
With thinking          ~0.3 s     ~36.3 s          (1,800 / 50 = 36.0 s of decode, plus ~0.3 s to first token)

Your TTFT graph looks unchanged while your users wait 36 seconds. If you monitor streaming latency, add a metric that starts the clock at the request and stops at the first text block. TTFT alone is blind to the entire cost of thinking.

Two architectural consequences

• Interactive turns need a visible thinking state. A spinner that normally resolves in under a second and now sits for half a minute reads as a crash. Stream the thinking, or show a labelled progress state. The worst option is an unchanged UI that silently takes 100× longer.
• Thinking and tight latency budgets are close to incompatible. If your SLO (the latency you have promised) is “under 2 seconds,” 1,800 thinking tokens will not fit at any realistic decode rate. That is arithmetic, not a tuning problem. Either the budget moves or the thinking does, into an asynchronous path where nobody is watching the spinner.

Effort: the dial that replaced budget_tokens

Money and seconds both scale with how much the model thinks, so the question is what control you have over that amount. One dial is left, and it is not the token count you may remember.

An amount became a disposition

Older models let you write budget_tokens: 2000 and get roughly that much thinking. That contract is gone: on Opus 5, Opus 4.8, Opus 4.7, and Sonnet 5, budget_tokens is rejected with a 400. What replaced it is effort. budget_tokens was an amount; effort is a disposition. You tell the model how thorough to be, and it decides per request how many tokens that takes. A trivial question at max still gets little thinking, because there is nothing to think about.

client.messages.create(
    model="claude-opus-5",
    max_tokens=4000,
    output_config={"effort": "high"},   # low | medium | high | xhigh | max
    thinking={"type": "adaptive"},
    messages=[...],
)

effort goes inside output_config, not at the top level. Getting this wrong is a rejected request, not a silently ignored setting.

What you lost

You have lost the ability to cap thinking spend directly. With budget_tokens you could guarantee no more than 2,000 thinking tokens per request. With effort you cannot; you can only lower the disposition and observe what happens. If you need a hard bound, max_tokens is the only one you have. Watching the budget drain showed what enforcing it looks like: a truncated answer, not a shorter thought. That is a worse failure mode, which is why the guard function above matters.

Turning thinking off is model-specific

Disabling thinking is not one flag that works everywhere, and getting it wrong is a 400, not a silent fallback. Check your model’s row before you write the call.

The default here inverts the old one: on Opus 5 and Sonnet 5, omitting thinking entirely runs adaptive thinking. Silence is opt-in now. (On Opus 4.8 and 4.7 it still means no thinking, which is why code moved between those models changes behaviour without changing a line.) Code written against an older model that never mentioned thinking is a thinking workload today, at the multiplier above, and the first sign is usually the bill.

Thinking in a tool-calling loop

One rule makes thinking work across multiple turns of tool use, and breaking it produces a specific, recognizable failure.

The rule

An agent (a model calling tools in a loop until a task is done) runs many requests per task. Thinking interacts with that loop through a single rule:

Send every thinking block back, unmodified, with its signature, in the next request of the same turn.

The reason is the mechanism from What a thinking model is, applied across requests. The model’s reasoning lives in the token stream, not in server state; the API keeps nothing between calls. Drop the thinking blocks from the conversation you send back and the model’s working-out is gone from its input. It resumes after the tool result with no record of why it called that tool.

One turn, two requests

sequenceDiagram
    participant A as Your app
    participant M as Model
    A->>M: request 1 (question + tools)
    M->>A: thinking block + tool_use(get_order 4021)
    Note over A: run the tool yourself
    A->>M: request 2 = question + THINKING BLOCK + tool_use + tool_result
    M->>A: thinking block 2 + final answer

In request 1 your app sends the question and the tool schemas. The model replies with a thinking block and a tool_use block: a content block naming a tool and its arguments, here get_order 4021. Your app runs that tool itself; the model never does, it only asks.

Request 2 is the part that matters. It resends the original question, the thinking block from request 1 verbatim, the tool_use block, and the tool_result you produced (the tool’s output, tagged with the id of the call it answers). The model then thinks again on top of its own earlier reasoning and answers.

Strip the thinking block from request 2 and the model sees a tool result it has no recorded reason for having requested. The symptom is not a crash. It is an agent that re-calls tools it already called, contradicts its own earlier steps, or gives up mid-task. It looks like a quality regression and it is a plumbing bug.

Two ways to break it by accident

1. Do not edit, truncate, re-encode, or pretty-print thinking blocks. The signature covers the exact content. A logging layer that trims long strings for readability will invalidate it and the API will reject the request.
2. Do not summarize the thinking to save tokens. It defeats the mechanism: the model needs its actual prior tokens to condition on, not a paraphrase.

The cost consequence nobody budgets for

Because thinking blocks stay in the conversation, they are resent as input on every subsequent request in that turn. A 1,800-token thought is 1,800 output tokens once, then 1,800 input tokens on every later request of the same turn. On a five-request tool loop that is one 1,800-token generation plus roughly four re-reads. So thinking’s true cost in an agent is higher than the single-turn arithmetic suggests. The re-reads are cheap per token (input is one-fifth of output, and caching can take it to a fiftieth) but they are not free, and they accumulate with loop length.

When thinking pays

Mechanism, money, and latency feed one decision: should this workload think? Build it from numbers.

The three costs in one place

Cost is a 2.29× problem and latency is a 10× problem; they are not the same size.

The number you have to supply yourself

One number no API can give you: what a wrong answer costs. Work it through for the refund case, supposing thinking takes the error rate from 8% to 2%:

• errors avoided per 100,000 turns: (8% − 2%) × 100,000 = 6,000
• cost of a wrong decision (a support contact, a reversal, an apology credit), say $10
• value bought: 6,000 × $10 = $60,000, against $4,500 of thinking

It is not close. Now change one number and the answer flips. Make it a ticket classifier where a wrong label costs $0.20 to fix downstream: 6,000 × $0.20 = $1,200 of value for $4,500 of thinking. Now thinking loses, and it loses twice over: classification is one step deep, so it would not have helped much anyway, and the error rate would not have moved from 8% to 2% in the first place.

The decision procedure, in order

1. Count the dependent steps. One step deep? Stop. Thinking will not help enough to matter, whatever it costs.
2. Measure the error rate both ways on your own data. Do not assume the improvement. A hundred labelled examples is enough to see whether the gap is real.
3. Price a wrong answer. This is the number that actually decides it, and the one nobody has written down.
4. Compute the multiplier for your input size, and if you cache, compute it after caching, where it will be several times larger.
5. Check the latency budget. If your SLO is under a few seconds, the arithmetic has already decided for you.

The answer is usually “both, routed”

The most common real answer is not on/off; it is routed. Run the cheap path by default, and escalate to thinking only on the cases that need it: the ambiguous ticket, the refund near a policy boundary, the request where the model’s confidence is low. The routing decision is itself a one-step classification (exactly the shape that does not need thinking) so the router is cheap, and you pay the 2.29× only on the fraction of traffic that earns it.

The mechanism → rule map

Every rule in this chapter is a consequence of one mechanism. Remember the left column and you can regenerate the right.

Conclusion

A thinking model is the same model given more forward passes. It buys serial depth (later passes reading earlier conclusions from the token stream), not smarter individual passes. That single fact sets everything else:

• Thinking helps tasks with dependency depth (chained steps), not wide-but-shallow tasks like classification, extraction, and translation.
• Thinking tokens are output tokens, so the bill moves by (I + 5·T)/(I + 5·t), governed by your input size, not by the raw token ratio. Prompt caching makes thinking’s share larger, so measure the multiplier after caching.
• max_tokens covers thinking and answer together, thinking first; check stop_reason and treat "max_tokens" as a failure.
• Latency scales roughly linearly with tokens and TTFT hides it, so instrument time-to-first-text-block.
• In a tool loop, resend thinking blocks verbatim with their signature.

The whole decision usually resolves to routing: run the cheap path by default, escalate to thinking only on the cases that earn the 2.29×.

One line to remember: thinking buys serial depth, so turn it on only where the answer has dependency depth, size the multiplier after caching, and check stop_reason every time.

Further reading

• Anthropic: Extended thinking (build-with-claude documentation): the current API surface for thinking, effort, display, and the signature/tool-loop rules.
• Anthropic: Prompt caching (build-with-claude documentation): the cache read/write rates and lifetimes used in the cost section.
• Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022), arXiv:2201.11903: the original evidence that writing out intermediate steps improves reasoning accuracy.