15 — Thinking Models, and Where the Token Budget Goes

What does a thinking model actually do differently from an ordinary one? Why does that difference make it better at hard problems — and what exactly does it cost you in tokens, money, and seconds?

You will watch a single request spend its token budget one token at a time, and derive the cost multiplier from first principles instead of memorizing it.

By the end you should be able to answer the question every interviewer and every finance team eventually asks: when is thinking worth paying for, and when is it just a slower bill?

The shape of the thing, before any mechanism

You send an ordinary API request — the same system, tools, and messages you would send anyway — plus a flag saying thinking is allowed.

What comes back is two kinds of content in one response: a stretch of the model’s own working-out, and then the actual answer. Both are generated the same way, by the same model, one token at a time. Both are billed at the same output-token rate.

The only structural difference is that the working-out arrives in a separate slot in the response, which you are free to hide from your user.

That is the whole idea. A thinking model is not a different architecture, a second model, or a planning module bolted onto the side.

It is the same model, given permission to write down its intermediate work before committing to an answer, in the same stream, at the same price per token. Everything in this chapter follows from that one sentence.

What you need to know before reading this

Nothing from other chapters. Four terms, defined here, are enough:

• A token is the unit of text a model reads and writes — roughly ¾ of an English word, so about 4 characters. The model does not see letters or words; it sees a sequence of these units. A short common word like "the" is one token; a long word like "unbelievable" is split into several.
• A forward pass is one run of the input through the model’s stack of layers. It consumes the entire sequence so far and produces exactly one next token. To write 100 tokens the model runs 100 forward passes, each one seeing everything written so far.
• Decode is that one-token-at-a-time generation loop. It is the slow half of serving a model, because every token requires its own full pass and passes cannot be run ahead of each other — token 51 cannot be computed until token 50 exists.
• The context is everything the model can see on a given pass: your prompt plus every token generated so far in this response. It is plain text as far as the model is concerned. Nothing else carries over between passes.

If you want the machinery underneath these, 00 — LLM Internals derives it. You do not need it to follow this chapter.

1. What a thinking model is

A thinking model differs from an ordinary one in exactly one mechanical way — and that difference is not cosmetic.

A forward pass is a fixed amount of compute

Ask an ordinary model a question and it starts emitting the answer immediately, token by token. Token 1 of the answer is produced by a single forward pass over your question.

Whatever “reasoning” happened, happened inside that one pass — a fixed amount of computation, the same for 2 + 2 as for a five-step policy question.

That is the constraint worth internalizing: a forward pass is a fixed amount of compute. The model cannot decide to think harder within one pass. The stack has however many layers it has, and every token gets exactly that many, no more.

A thinking model gets around the constraint the only way that is available: by using more passes. Before it writes the answer, it writes out its working — and each of those working-out tokens costs its own forward pass, and, crucially, lands in the context that later passes read.

The same request, two timelines

The diagram below shows one request twice: once through an ordinary model, once through a thinking model. Each row runs left to right as time, and the only thing that moves between them is where the first answer token sits — everything else about the request is identical.

flowchart LR
    subgraph ordinary["Ordinary model"]
        Q1["question<br/>(6,000 tokens)"] --> A1["answer token 1"] --> A2["answer token 2"] --> A3["... 200 answer tokens"]
    end
    subgraph thinking["Thinking model"]
        Q2["question<br/>(6,000 tokens)"] --> T1["thinking token 1"] --> T2["... 1,800 thinking tokens"] --> B1["answer token 1"] --> B2["... 200 answer tokens"]
    end
    style ordinary fill:#1d3557,color:#fff
    style thinking fill:#2d6a4f,color:#fff

Both rows start from the same box — a question of 6,000 tokens of policy and case detail.

In the ordinary model row, that question is followed directly by 200 answer tokens. That is 200 forward passes total, and the first answer token is produced after a single pass over the question.

In the thinking model row, the identical question is followed by 1,800 thinking tokens before the first answer token. The answer token that used to be pass #1 is now pass #1,801.

The question is unchanged. Only the number of passes between question and answer differs.

Why extra passes actually help

Why does putting the working-out in the context help, rather than just costing more? Because of how each forward pass reads its input.

Every pass sees the entire sequence generated so far — your question and every thinking token already written. So when the model computes thinking token 1,801, it is not re-deriving from scratch; it is reading its own conclusions from tokens 1 through 1,800 as plain input text and building on them.

That converts a problem the model would have had to solve in one pass into a problem it can solve in 1,800 chained passes, where each pass gets to condition on the results of every pass before it. A question needing five dependent steps no longer has to fit inside one fixed-size computation. It gets five stretches of tokens, each reading the last.

This is the trade in one line: thinking buys serial depth with output tokens. It cannot make any single pass smarter. It can only give the model more passes, arranged in a chain, each able to read the one before. Every cost, latency, and quality property in the rest of this chapter is a consequence of that.

2. The request and the response, concretely

All of that mechanism rides on two small, concrete shapes on the wire: the request you send and the response you get back.

What you send

Here is the whole surface. You add one field to a request you would have sent anyway:

client.messages.create(
    model="claude-opus-5",
    max_tokens=4000,
    thinking={"type": "adaptive"},
    messages=[{"role": "user", "content": "Can we refund order #4021?"}],
)

thinking={"type": "adaptive"} means the model decides how much to think, per request. A trivial question gets almost no thinking; a hard one gets a lot. You are not setting a number.

What you get back

An ordinary response has one string of text in it. A thinking response does not — it comes back as a list of content blocks. A content block is one typed chunk of the response: {"type": "thinking", ...}, {"type": "text", ...}, and so on. The working-out and the answer are separate blocks in the same list.

Here is a response to the request above, with the fields that matter filled in:

Message(
    content=[
        ThinkingBlock(thinking="The order is 40 days old...", signature="ErUBCkYIAx..."),
        TextBlock(text="Yes — but it needs manager approval. Here's why: ..."),
    ],
    usage=Usage(input_tokens=6042, output_tokens=2000),
    stop_reason="end_turn",
)

stop_reason is the API’s own explanation of why generation stopped. "end_turn" means the model finished on its own. You will meet the other value that matters, "max_tokens", in Token by token watching the budget drain.

Four things in that response that bite people in production

1. content is a list, and the first block is not your answer. Code that does response.content[0].text worked fine before thinking and breaks the moment you enable it — the first block is a ThinkingBlock, which has no .text at all. Filter by block type; never index by position.

2. output_tokens is the total: thinking plus answer. It reads 2,000 here (1,800 thinking + 200 answer), not 200. There is no separate meter, and this single number is what you are billed on. Section What thinking costs in money does the arithmetic.

3. By default there is no readable thinking text at all — and you pay for it regardless. On current models thinking.display defaults to "omitted": the ThinkingBlock still arrives, but its thinking field is an empty string. To see anything you must opt in with thinking={"type": "adaptive", "display": "summarized"} — and even then what you get is a summary, because the raw chain of thought is never returned.

   So the string you can print is either empty or a few hundred tokens, while output_tokens reflects the couple of thousand the model actually generated. Never estimate your thinking spend by measuring the length of the text you received — at the default it would tell you thinking was free. Read usage.output_tokens.

4. signature is not decoration. It is a cryptographic stamp proving the block came from the model unmodified. In a tool-calling loop you must send thinking blocks back exactly as received, signature intact — Thinking in a tool calling loop explains what breaks if you don’t.

One field you probably remember that is now an error

Older models took thinking={"type": "enabled", "budget_tokens": 2000}, letting you set the thinking allowance yourself as a token count.

On current models — Opus 5, Opus 4.8, Opus 4.7, Sonnet 5 — budget_tokens has been removed and sending it returns a 400. Thinking is adaptive: on Opus 5, omitting the thinking field entirely runs adaptive thinking anyway.

If you want to influence how much the model thinks, you use effort (Effort the dial that replaced budget_tokens), not a token count.

The whole surface in one runnable request

Everything this section described, as a script you can run against the live API. The load-bearing lines: display: "summarized" controls only what you can see — thinking happens and is billed identically under every display setting; output_config={"effort": "high"} is the dial from Effort the dial that replaced budget_tokens; and the final lines apply Thinking in a tool calling loop’s multi-turn rule — the assistant turn you send back is resp.content itself, thinking blocks and signatures untouched.

import anthropic

client = anthropic.Anthropic()

question = (
    "Refunds are allowed within 30 days of purchase; Gold-tier customers "
    "get 60 days. Any refund over $150 needs manager approval, unless the "
    "customer has no prior refunds. Order #4021: placed 40 days ago, $180, "
    "Gold tier, 1 prior refund. Can we refund it?"
)

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=4000,
    thinking={"type": "adaptive", "display": "summarized"},
    output_config={"effort": "high"},
    messages=[{"role": "user", "content": question}],
)

for block in resp.content:
    if block.type == "thinking":
        print("--- thinking (summary) ---")
        print(block.thinking)
    elif block.type == "text":
        print("--- answer ---")
        print(block.text)

print(resp.stop_reason)               # must be "end_turn", not "max_tokens"
print(resp.usage.output_tokens)       # thinking + answer, the billed total

# Continuing this conversation? Append the content list back unchanged.
history = [{"role": "user", "content": question}]
history.append({"role": "assistant", "content": resp.content})

3. Token by token: watching the budget drain

The cleanest way to see where the token budget goes is to watch one real request drain it, decode step by decode step — including the failure mode that catches everyone the first time.

A question with real dependent steps

Thinking only has something to do when the answer requires steps that build on each other. Here is a question that does:

Policy. Refunds are allowed within 30 days of purchase. Gold-tier customers get 60 days. Any refund over $150 needs manager approval, unless the customer has no prior refunds.

The case. Order #4021, placed 40 days ago, $180, customer is Gold tier with 1 prior refund. Can we refund it?

Answering needs four steps, and step 4 depends on steps 2 and 3.

Reading the decode trace

Below is that request decoded one pass at a time, with max_tokens=1900. Three columns:

• pass — which forward pass this is. Pass n produces token n.
• block — whether the token landed in the thinking block or the answer.
• remaining — how much of max_tokens is left. This is one counter covering thinking and answer together.

The interesting moment is pass 1,801, where the block changes from thinking to answer — and the remaining column shows only 99 tokens left.

 pass    block      token emitted           remaining of max_tokens=1900
------  ---------  ----------------------  ----------------------------
    1   thinking   "The"                                          1,899
    2   thinking   " order"                                       1,898
    3   thinking   " is"                                          1,897
    4   thinking   " "                                            1,896
    5   thinking   "40"                                           1,895
    6   thinking   " days"                                        1,894
    7   thinking   " old"                                         1,893
    8   thinking   "."                                            1,892
  ...
  310   thinking   " Gold"                                        1,590     <- step 1 settled: window is 60 days
  ...
  902   thinking   " within"                                        998     <- step 2 settled: 40 <= 60, in window
  ...
 1,404   thinking   " exceeds"                                      496     <- step 3 settled: 180 > 150
  ...
 1,800   thinking   " approval"                                     100     <- step 4 settled, depends on 2 and 3
 1,801   answer     "Yes"                                             99
 1,802   answer     ","                                               98
 1,803   answer     " but"                                            97
  ...
 1,900   answer     " manager"                                         0     <- CEILING HIT, mid-sentence

Read down the remaining column and the mechanism is visible. Every token, of either kind, decrements one counter.

The model spent 1,800 tokens working out that the order is inside the Gold window, that $180 is over the approval threshold, and that 1 prior refund disqualifies the exemption. That left 1,900 − 1,800 = 100 tokens in which to say so, against the 200 the full answer needs. It ran out at token 1,900 with the sentence unfinished.

The failure is a 200 OK, not an exception

The response comes back with stop_reason: "max_tokens" and a truncated answer. Not an exception, not an error code — a 200 OK with a half-sentence in it. Code that reads .text and moves on will happily ship “Yes, but it needs manager” to a customer.

The lesson, stated plainly: max_tokens is not the answer’s budget. It is the thinking-plus-answer budget, and thinking gets to spend it first. A max_tokens that was generously sized for a 200-token answer is a truncation waiting to happen once thinking is switched on, because thinking will consume nine-tenths of it before the answer starts.

The same request with more headroom

Now re-run the identical request with max_tokens=4000 and nothing else changed. Here is the tail of the trace — compare the remaining column at pass 1,800 with the run above:

 1,800   thinking   " approval"                                   2,200
 1,801   answer     "Yes"                                         2,199
  ...
 2,000   answer     "."                                           2,000     <- stop_reason: "end_turn"

The thinking cost the same 1,800 tokens: 4,000 − 1,800 = 2,200 remaining when the answer starts. The answer got its full 200 and finished at pass 2,000, leaving 2,000 unspent.

Those 2,000 unused tokens cost nothing — you are billed for tokens generated, never for headroom. max_tokens is a ceiling, not a reservation.

So the rule is: set max_tokens to the answer you want plus generous room for thinking, and treat a wasted-looking ceiling as free. The only real cost of a high ceiling is that a runaway response can take longer before it stops. Then check stop_reason on every response and treat "max_tokens" as a failure, because it is one.

Guarding it in code

The code below is that rule as a function rather than a comment. answer_of takes the pieces of a response and either returns the answer text or raises.

Content blocks are written as plain dictionaries — {"type": "thinking", ...} and {"type": "text", ...} — so the example runs anywhere; a real SDK gives you objects with the same type field. The two runs from the traces above are then fed through it, one truncated and one complete.

def answer_of(response_content, stop_reason, max_tokens):
    """Return the answer text, or raise if thinking ate the budget."""
    if stop_reason == "max_tokens":
        raise ValueError(
            f"truncated at max_tokens={max_tokens}: thinking consumed the budget "
            f"before the answer finished — raise the ceiling and retry"
        )
    text = "".join(b["text"] for b in response_content if b["type"] == "text")
    if not text:
        raise ValueError("no text block: the model thought but never answered")
    return text


# The truncated run from the trace above: 200 OK, but the answer is a fragment.
truncated = [
    {"type": "thinking", "thinking": "The order is 40 days old..."},
    {"type": "text", "text": "Yes, but it needs manager"},
]
try:
    answer_of(truncated, stop_reason="max_tokens", max_tokens=1900)
    raise AssertionError("a truncated response must not be accepted")
except ValueError as e:
    assert "truncated" in str(e)

# The 4,000-token run: same thinking, complete answer.
ok = [
    {"type": "thinking", "thinking": "The order is 40 days old..."},
    {"type": "text", "text": "Yes, but it needs manager approval."},
]
assert answer_of(ok, stop_reason="end_turn", max_tokens=4000).endswith("approval.")

# The block that bites everyone: index 0 is not the answer.
assert ok[0]["type"] == "thinking"
assert "text" not in ok[0]

That second guard — if not text — catches the rarer, nastier case: a response that is all thinking and no answer, which is what truncation looks like when the ceiling is low enough that the model never gets to start.

4. Why thinking makes the model better

Accuracy improves for a specific, mechanical reason — and knowing it lets you predict which of your own tasks will benefit before you spend anything testing.

Not “trying harder” — surviving intermediate results

It is tempting to believe the model is “trying harder.” It isn’t — every forward pass does the same fixed work. What changes is how many passes the answer gets to depend on, and whether intermediate results survive long enough to be used.

Consider the refund question again. Answering it requires holding four facts and combining them in order:

1. Gold tier ⟹ the window is 60 days, not 30.
2. 40 days ≤ 60 ⟹ inside the window.
3. $180 > $150 ⟹ approval threshold is crossed.
4. 1 prior refund ≠ 0 ⟹ the exemption in the policy does not apply, so approval is genuinely required.

Step 4 cannot be evaluated until steps 2 and 3 are known.

Without thinking, all four have to be resolved inside one forward pass, and the intermediate conclusions exist only as internal activations that vanish when the pass ends. Squeezing the chain into that fixed budget is exactly where models produce confidently wrong answers: they latch onto the salient fact (40 > 30, so it’s too late) and miss the tier rule that overrides it.

With thinking, each conclusion is written into the token stream, where it becomes ordinary input text that every later pass reads. Step 4’s forward pass doesn’t have to re-derive steps 2 and 3 — it reads them.

The predictor is dependency depth, not difficulty

So the predictor for whether thinking will help your task is not difficulty. It is dependency depth. Ask: does answering require a conclusion that must be reached before another conclusion can even be attempted?

The table applies that test to six common tasks. The middle column counts dependent steps — steps that cannot start until an earlier one has finished — and the “yes” rows are exactly the rows with a chain in that column, whatever the task names suggest.

The two “no” rows are the money-savers. Extraction and classification are wide, not deep — lots of items, but each resolved independently in a single step.

Turning thinking on for a high-volume classifier multiplies the bill (see What thinking costs in money) and buys close to nothing, because there is no chain for the extra passes to build.

5. What thinking costs in money

The cost multiplier is not a figure to memorize. It falls out of first principles, and once you have derived it you can compute it for your own workload instead of trusting a rule of thumb.

The pricing rule, in one sentence

Thinking tokens are billed as output tokens. There is no discount and no separate rate. That is the entire pricing rule; everything below is arithmetic on top of it.

One turn, two ways

Take the refund turn on claude-opus-5, priced at $5 per million input tokens and $25 per million output tokens. The input is the same 6,000 tokens either way; only the output count changes.

Doing the multiplication for the thinking row:

• input: 6,000 × $5 / 1,000,000 = $0.030
• output: 2,000 × $25 / 1,000,000 = $0.050
• total: $0.080

Both rows, side by side:

Output tokens went up 10×. The bill went up 2.29× ($0.080 ÷ $0.035). That gap is the whole point, and it is worth seeing why rather than remembering it.

Where the multiplier comes from

Write p for the input price per token. Output is priced at 5p on every current Claude model — $5/$25 for Opus 5, $3/$15 for Sonnet 5, $1/$5 for Haiku 4.5 are all the same 1:5 ratio.

With I input tokens and T output tokens, the bill for a turn is I·p + T·5p. So the multiplier is one bill over the other, with the actual numbers substituted in three steps: fill in the token counts, cancel p, then divide.

cost_with_thinking     I·p + 2000·5p     I + 10,000     6,000 + 10,000     16,000
------------------  =  --------------  =  ----------  =  --------------  =  ------  =  2.29
cost_without           I·p +  200·5p     I +  1,000     6,000 +  1,000      7,000

Step by step:

• 2000·5p is 10,000p, and 200·5p is 1,000p. So the two bills are I·p + 10,000p and I·p + 1,000p.
• Every term on both sides has a p in it, so p divides out. That leaves (I + 10,000) / (I + 1,000).
• Substitute I = 6,000: 16,000 / 7,000 = 2.2857…, which is 2.29 to two places.

p cancels. The multiplier does not depend on which model you picked — Opus, Sonnet and Haiku all give 2.29× here — because they share the 1:5 input-to-output price ratio.

It depends only on how big your input is relative to your thinking. A large input dilutes thinking’s contribution; a small one exposes it.

Input size is the only dial that moves it

That relationship is worth tabulating, because it is where intuition fails. Every row below uses the same 1,800 thinking tokens and the same 200-token answer. Only the input size changes, and the multiplier swings from 1.09× to 8.5×.

Each row is the same (I + 10,000) / (I + 1,000) with a different I. The 1,000-token row, for instance, is 11,000 / 2,000 = 5.5.

Now the trap: prompt caching makes this worse

Prompt caching means reusing the server’s work on a prefix you send repeatedly — a long system prompt, a policy document — instead of re-processing it every request. It interacts with the multiplier badly, and in the direction nobody expects.

Caching has two rates, and the difference matters here:

• Reading a cached prefix costs 0.1× the normal input rate.
• Writing it the first time costs 1.25× (or 2× if you ask for the 1-hour lifetime instead of the default five minutes).

Once the prefix is cached, those 6,000 input tokens bill as if they were 600 (6,000 × 0.1). Put 600 into the same formula:

600 + 10,000     10,600
------------  =  ------  =  6.63
600 +  1,000      1,600

Caching pushed the thinking multiplier from 2.29× to 6.63×.

Caching didn’t get more expensive — it made input nearly free, and removed the term that was diluting thinking’s share. Both bills fell in absolute terms; thinking’s share of what remains went up.

The cache-write request runs the other way

That 6.63× is the steady state, not the whole story, and reading it as the whole story gets the first request exactly backwards.

On the request that writes the cache you pay the premium instead of the discount, which makes input larger — so it dilutes thinking more than no caching at all.

The table below prices the same turn under all four conditions. The multiplier falls on the write rows and then jumps on the read row — it is not monotonic.

To check one row by hand: on the 5-minute cache write, input bills at 6,000 × $5/M × 1.25 = $0.0375, so the plain turn is $0.0375 + $0.005 = $0.0425 and the thinking turn is $0.0375 + $0.050 = $0.0875. Dividing gives 2.06×.

So the honest statement is: caching raises thinking’s share of the bill once the cache is being read, and lowers it on the request that writes it.

With a five-minute lifetime the cache pays for itself from the second request; with the one-hour lifetime it takes three. If your workload reads the cache many times per write — which is the only reason to cache — 6.63× is the number you will live at, and the write row is a one-request transient.

The practical consequence is a sequencing rule for cost work: optimize caching first, then re-measure whether thinking still pays. A thinking-on decision that looked cheap at 2.29× is the dominant line item at 6.63× once caching lands, and teams that tune the two independently reach the wrong conclusion about both.

Verify the whole section

The block below rebuilds every number above from the two prices and the token counts. turn_cost is the pricing rule from the top of this section; each assert is one claim from the tables.

OPUS_IN, OPUS_OUT = 5 / 1e6, 25 / 1e6          # $ per token
CACHE_READ = 0.1                                # cached input bills at 0.1x

def turn_cost(in_tok, out_tok, cached=False, p_in=OPUS_IN, p_out=OPUS_OUT):
    return in_tok * p_in * (CACHE_READ if cached else 1.0) + out_tok * p_out

plain    = turn_cost(6_000,   200)
thinking = turn_cost(6_000, 2_000)
assert round(plain, 4) == 0.0350
assert round(thinking, 4) == 0.0800
assert round(thinking / plain, 2) == 2.29        # 10x the output tokens, 2.29x the bill

# The multiplier is price-independent: Sonnet 5 and Haiku 4.5 share the 1:5 ratio.
for p_in, p_out in [(3 / 1e6, 15 / 1e6), (1 / 1e6, 5 / 1e6)]:
    ratio = turn_cost(6_000, 2_000, p_in=p_in, p_out=p_out) / \
            turn_cost(6_000,   200, p_in=p_in, p_out=p_out)
    assert round(ratio, 2) == 2.29

# Input size is what actually moves it.
assert round(turn_cost(100_000, 2_000) / turn_cost(100_000, 200), 2) == 1.09
assert round(turn_cost(  1_000, 2_000) / turn_cost(  1_000, 200), 2) == 5.50

# Caching makes thinking's share LARGER, not smaller.
cached_ratio = turn_cost(6_000, 2_000, cached=True) / turn_cost(6_000, 200, cached=True)
assert abs(cached_ratio - 6.625) < 1e-9          # 6.63 to two places
assert turn_cost(6_000, 2_000, cached=True) < thinking      # both bills still fell

# What it means at volume, per 100,000 turns:
assert round(plain * 100_000) == 3_500
assert round(thinking * 100_000) == 8_000       # +$4,500 for the thinking

At 100,000 turns a month that is $3,500 versus $8,000. Whether the extra $4,500 is worth it is not a token question — it is a question about what a wrong refund decision costs you, which is When thinking pays.

6. What thinking costs in latency

Thinking hurts perceived speed more than the token count suggests — and the metric most teams watch will not show it.

Ten times the tokens is ten times the wait

Decode is serial, so wall-clock time for the generated part is simply tokens ÷ decode rate.

Measure your own rate — it varies by model, load, and whether you stream — but suppose you measure 50 tokens per second. The table then follows by division: 200 ÷ 50 and 2,000 ÷ 50.

Unlike cost, latency gets no dilution from input size — reading the 6,000-token prompt happens in one batched pass and is fast; it is the 2,000 serial decode steps that you wait for.

Why your latency dashboard will not show it

Here is the part that catches teams with dashboards. Time to first token (TTFT) — the standard streaming latency metric, the delay between sending the request and the first token appearing — barely moves.

The first token still arrives in about the same time; it is simply a thinking token now. The user watching a “thinking…” indicator is waiting for the first answer token, which arrives only after all 1,800 thinking tokens.

The two columns below are the same request measured two ways. TTFT is unchanged; the metric nobody instruments is 100× worse.

                       TTFT       time to first ANSWER token
Without thinking       ~0.3 s     ~0.3 s
With thinking          ~0.3 s     ~36.3 s          (1,800 / 50 = 36.0 s of decode, plus the ~0.3 s to first token)

Your TTFT graph will look unchanged while your users wait 36 seconds. If you monitor streaming latency, add a metric that starts the clock at the request and stops at the first text block — TTFT alone is blind to the entire cost of thinking.

Two architectural consequences

• Interactive turns need a visible thinking state. A spinner that normally resolves in under a second and now sits for half a minute reads as a crash. Stream the thinking, or show a labelled progress state — the worst option is an unchanged UI that silently takes 100× longer.
• Thinking and tight latency budgets are close to incompatible. If your SLO — the service-level objective, the latency you have promised — is “under 2 seconds,” 1,800 thinking tokens will not fit at any realistic decode rate. That is not a tuning problem; it is arithmetic. Either the budget moves or the thinking does, into an asynchronous path where nobody is watching the spinner.

7. Effort: the dial that replaced budget_tokens

Money and seconds both scale with how much the model thinks, so the obvious next question is what control you have over that amount. One dial is left, and it is not the token count you may remember.

An amount became a disposition

Older models let you write budget_tokens: 2000 and get roughly that much thinking. It was a clean contract and it is gone: on Opus 5, Opus 4.8, Opus 4.7 and Sonnet 5, budget_tokens is rejected with a 400.

What replaced it is effort, and the difference is not cosmetic. budget_tokens was an amount; effort is a disposition.

You are telling the model how thorough to be, and it decides per request how many tokens that takes — a trivial question at max still gets little thinking, because there is nothing to think about:

client.messages.create(
    model="claude-opus-5",
    max_tokens=4000,
    output_config={"effort": "high"},   # low | medium | high | xhigh | max
    thinking={"type": "adaptive"},
    messages=[...],
)

Note where effort goes: inside output_config, not at the top level. It is a common enough slip to be worth stating, because the failure is a rejected request rather than a silently ignored setting.

What you lost

The practical difference is that you have lost the ability to cap thinking spend directly.

With budget_tokens you could guarantee no more than 2,000 thinking tokens per request. With effort you cannot — you can only lower the disposition and observe what happens.

If you need a hard bound, max_tokens is now the only one you have, and Token by token watching the budget drain showed what enforcing it looks like: a truncated answer rather than a shorter thought. That is a worse failure mode, which is why the guard in §3 matters.

Turning thinking off is model-specific

Disabling thinking is not one flag that works everywhere, and getting it wrong is a 400 rather than a silent fallback. Check your model’s row before you write the call:

And note the default, because it inverts the old one: on Opus 5 and Sonnet 5, omitting thinking entirely runs adaptive thinking. Silence is opt-in now. (On Opus 4.8 and 4.7 it still means no thinking, which is why code moved between those models changes behaviour without changing a line.)

Code written against an older model that simply never mentioned thinking is a thinking workload today, at the multiplier from What thinking costs in money, and the first sign is usually the bill.

8. Thinking in a tool-calling loop

One rule makes thinking work across multiple turns of tool use — and breaking it produces a specific, recognizable failure.

The rule

An agent — a model calling tools in a loop until a task is done — runs many requests per task. Thinking interacts with that loop through a single rule:

Send every thinking block back, unmodified, with its signature, in the next request of the same turn.

The reason is What a thinking model is’s mechanism applied across requests. The model’s reasoning lives in the token stream, not in server state — the API keeps nothing between calls.

Drop the thinking blocks from the conversation you send back and the model’s own working-out is gone from its input. It resumes after the tool result with no memory of why it called that tool.

One turn, two requests

The diagram traces a single tool-calling turn as the two API requests it actually takes; the whole rule is the one thing request 2 carries that request 1 did not.

sequenceDiagram
    participant A as Your app
    participant M as Model
    A->>M: request 1 (question + tools)
    M->>A: thinking block + tool_use(get_order 4021)
    Note over A: run the tool yourself
    A->>M: request 2 = question + THINKING BLOCK + tool_use + tool_result
    M->>A: thinking block 2 + final answer

In request 1 your app sends the question and the tool schemas. The model replies with a thinking block and a tool_use block — a content block naming a tool and its arguments, here get_order 4021, meaning “look up order 4021 for me.”

Your app runs that tool itself. The model never does; it only asks.

Request 2 is the part that matters. It resends the original question, the thinking block from request 1 verbatim, the tool_use block, and the tool_result you produced (the tool’s output, tagged with the id of the call it answers). The model then thinks again on top of its own earlier reasoning and answers.

Strip the thinking block from request 2 and the model sees a tool result it has no recorded reason for having requested. The symptom is not a crash — it is an agent that re-calls tools it already called, contradicts its own earlier steps, or gives up mid-task. It looks like a quality regression and it is a plumbing bug.

Two ways to break it by accident

1. Do not edit, truncate, re-encode, or pretty-print thinking blocks. The signature covers the exact content. A logging layer that trims long strings for readability will invalidate it and the API will reject the request.
2. Do not summarize the thinking to save tokens. It reads like an obvious optimization and it defeats the mechanism — the model needs its actual prior tokens to condition on, not a paraphrase.

The cost consequence nobody budgets for

Because thinking blocks stay in the conversation, they are resent as input on every subsequent request in that turn.

A 1,800-token thought is 1,800 output tokens once, and then 1,800 input tokens on every later request of the same turn. On a five-request tool loop that is one 1,800-token generation plus roughly four re-reads.

So thinking’s true cost in an agent is higher than the single-turn arithmetic in What thinking costs in money suggests. The re-reads are cheap per token (input is priced at one-fifth of output, and caching can take it to a fiftieth) but they are not free, and they accumulate with loop length.

9. When thinking pays

Mechanism, money, latency — all of it feeds one decision: should this workload think? That decision has to survive a design review, so build it from numbers.

The three costs in one place

The table collects what §5 and §6 computed for the refund turn. Cost is a 2.29× problem and latency is a 10× problem — they are not the same size.

The number you have to supply yourself

Against that sits one number no API can give you: what a wrong answer costs.

Work it through for the refund case. Suppose thinking takes the error rate from 8% to 2%:

• errors avoided per 100,000 turns: (8% − 2%) × 100,000 = 6,000
• cost of a wrong decision — a support contact, a reversal, an apology credit — say $10
• value bought: 6,000 × $10 = $60,000, against $4,500 of thinking

It is not close.

Now change one number and the answer flips. Make it a ticket classifier where a wrong label costs $0.20 to fix downstream: 6,000 × $0.20 = $1,200 of value for $4,500 of thinking.

Now thinking loses, and it loses twice over — because classification is one step deep, Why thinking makes the model better predicted it would not help much, so the error rate would not have moved from 8% to 2% in the first place.

The decision procedure, in order

1. Count the dependent steps (Why thinking makes the model better). One step deep? Stop — thinking will not help enough to matter, whatever it costs.
2. Measure the error rate both ways on your own data. Do not assume the improvement. A hundred labelled examples is enough to see whether the gap is real.
3. Price a wrong answer. This is the number that actually decides it, and it is the one nobody has written down.
4. Compute the multiplier for your input size (What thinking costs in money) — and if you cache, compute it after caching, where it will be several times larger.
5. Check the latency budget (What thinking costs in latency). If your SLO is under a few seconds, the arithmetic has already decided for you.

The answer is usually “both, routed”

The most common real answer is not on/off — it is routed.

Run the cheap path by default, and escalate to thinking only on the cases that need it: the ambiguous ticket, the refund near a policy boundary, the request where the model’s own confidence is low.

The routing decision is itself a one-step classification, which is exactly the shape that does not need thinking. So the router is cheap, and you pay the 2.29× only on the fraction of traffic that earns it.

10. Deriving the numbers

Every figure in this chapter can be rebuilt from two prices and two token counts — worth doing once, so that under interview pressure you derive rather than recall.

Left column is the number as it appeared; right column is the arithmetic that produces it. If you can reproduce the right column you never have to memorize the left.

And the same table as code, so you can run it rather than trust it. Every value is rebuilt from the two prices and the two token counts at the top:

# Every headline number, rebuilt from the two prices and the two token counts.
IN_TOK, THINK_TOK, ANSWER_TOK = 6_000, 1_800, 200
P_IN, P_OUT = 5 / 1e6, 25 / 1e6

out_thinking = THINK_TOK + ANSWER_TOK
assert out_thinking == 2_000

plain    = IN_TOK * P_IN + ANSWER_TOK   * P_OUT
thinking = IN_TOK * P_IN + out_thinking * P_OUT
assert (round(plain, 3), round(thinking, 3)) == (0.035, 0.080)

# The multiplier, from the closed form -- note p_in never appears.
def multiplier(in_tok, out_with, out_without, cache=1.0):
    return (in_tok * cache + 5 * out_with) / (in_tok * cache + 5 * out_without)

assert round(multiplier(IN_TOK, 2_000, 200), 2) == 2.29
assert abs(multiplier(IN_TOK, 2_000, 200, cache=0.1) - 6.625) < 1e-9
assert round(multiplier(100_000, 2_000, 200), 2) == 1.09
assert round(multiplier(  1_000, 2_000, 200), 2) == 5.50

# Latency: decode is serial, so it is a straight division.
RATE = 50                                        # tokens/second, measure your own
assert out_thinking / RATE == 40.0
assert ANSWER_TOK / RATE == 4.0
assert THINK_TOK / RATE == 36.0                  # the wait TTFT does not show

# The decision, at volume.
assert round((thinking - plain) * 100_000) == 4_500
avoided = (0.08 - 0.02) * 100_000
assert round(avoided) == 6_000 and round(avoided * 10) == 60_000

The mechanism → rule map

Every rule in this chapter is a consequence of one mechanism. If you remember the left column you can regenerate the right, which is what you want under interview pressure.

What interviewers probe

• “Why does thinking improve accuracy? Be specific.” The wrong answer is “it tries harder.” The right one: a forward pass is fixed compute, and thinking tokens land in the context, so later passes read earlier conclusions as input. It converts one fixed computation into a chain of them.
• “Your output tokens went up 10× — what happened to the bill?” Not 10×. (I + 5·T)/(I + 5·t). Ask for the input size before answering, because that is the term that decides it.
• “You enabled prompt caching and your thinking multiplier got worse. Explain.” Caching cut input to a tenth, removing the term that diluted thinking’s share. Both bills fell; thinking’s fraction rose.
• “Your p50 TTFT is unchanged but users say it’s slow. Where would you look?” TTFT measures the first token, which is now a thinking token. Instrument time-to-first-text-block.
• “When would you turn thinking off?” One-step-deep tasks — classification, extraction, translation — and any path inside a tight latency SLO. Then: route, don’t choose globally.
• “Your agent started re-calling tools it already called after you enabled thinking.” Thinking blocks are being dropped from the conversation on subsequent requests. The model lost the record of why it acted.
• “Set budget_tokens to 1,000 for me.” The trap. It is removed on current models and returns a 400; effort is the replacement, and it is a disposition, not a cap.