04 — Memory & Context

This chapter is about two decisions your agent makes on every turn: what to send the model on this request, and what to keep after the process exits.

By the end you should be able to:

• name the four kinds of memory an agent can hold, and say what writes to each one
• lay out a request so that prompt caching keeps paying. Caching is the server storing the result of reading a repeated opening chunk of your request, so that the next request starting with the same bytes is billed at a fraction of the normal rate — The context window budget has the mechanism
• choose between the four ways to stop a conversation from growing without bound
• decide what a long-term store should save, overwrite, and delete

Those four skills are what sit behind three common interview questions: price a memory design before you build it, explain why an agent that behaved at turn 3 is confused at turn 40, and debug a bill that tripled after a one-line change.

The shape of one request

Before any mechanism, fix the shape of the whole thing: what goes in, and what comes out.

What goes in

The application programming interface (API) you call is stateless. That means the server keeps nothing between requests. Everything the model knows on this request is in this request.

What you send is one object with three parts that carry text:

• tools — the schemas for every function the model is allowed to call. A schema is the JSON description of a function’s name, purpose, and parameters.
• system — standing instructions.
• messages — the conversation.

That third part is a plain list, and it is what people mean by “context”:

messages = [
    {"role": "user",      "content": "What's the capital of France?"},
    {"role": "assistant", "content": "Paris."},
    {"role": "user",      "content": "And its population?"},
]

That list is the entire memory of the conversation. There is no session on the server that also remembers “Paris” — delete that middle entry and the model cannot answer the third one.

A turn is one user message plus everything the agent does in response to it. So the list above holds one complete turn and the opening of a second.

What comes out

The response is an object carrying the reply plus a usage record that tells you what you were billed for.

In the block below, -> is not Python. It is shorthand for “if you print this expression, here is what you see.”

resp = client.messages.create(
    model="claude-opus-5", max_tokens=1024,
    system=SYSTEM, tools=TOOLS, messages=messages,
)

resp.content  ->  [TextBlock(text="About 2.1 million.")]
resp.usage    ->  Usage(input_tokens=41, output_tokens=7,
                        cache_creation_input_tokens=0, cache_read_input_tokens=0)

Two things to read off that block.

resp.content is a list of content blocks, not a string. Here it holds a single TextBlock. A request that triggers a tool call comes back with a differently typed block in the same list.

resp.usage is where every cost claim in this chapter is actually measured. input_tokens is what you paid to have your request read. The two cache_ counters are zero here only because nothing was cached yet.

Why the array only grows

Your code appends that reply to messages and sends the whole list again on the next request. So the array only ever grows, and you pay for all of it, every turn.

A token is the unit the model and the bill are both denominated in — roughly four characters of English prose.

Two different ceilings act on that array, and only one of them is enforced by the server.

The context window is the hard cap on how many tokens may appear in one request, prompt and generated output together — on the order of 200,000 tokens for current models. It is a limit, not a budget: nothing bills differently at 199k than at 20k.

The budget is the thing you impose on yourself, because every token in the array is paid for again on every remaining turn.

Memory is the discipline of deciding what “everything” means — which text earns its place in that array this turn, and which facts are worth writing somewhere durable so a future array can be rebuilt without them. It is a cost strategy before it is a capability.

1. Two axes: context engineering and memory

Two distinct problems both get called “memory”: curating one request, and persisting facts after the process exits. Interviewers conflate them constantly, and separating them is the first signal.

The diagram below puts the two side by side. The dotted arrows connecting them are the only moving parts.

flowchart TD
    subgraph WITHIN["Within a session — Context engineering"]
        A["What goes in the window this turn,<br/>and in what order"]
    end
    subgraph ACROSS["Across sessions — Memory"]
        B["What survives after<br/>the process exits"]
    end
    A -.->|write durable facts| B
    B -.->|retrieve relevant| A

    style WITHIN fill:#40916c,color:#fff
    style ACROSS fill:#7209b7,color:#fff

The two colours in that diagram are the two lifespans, and they recur throughout the chapter. Green is the within-session half, which lives and dies inside one process. Purple is the across-session half, which gets written to storage.

Context engineering is the green box: curating one prompt. It decides what goes into the window this turn, and in what order.

Memory is the purple box: persistence. It decides what survives after the process exits.

The first is a list in your process’s memory that dies in seconds. The second is a database row that outlives the process.

The two arrows, and when each fires

The two dotted arrows are the entire loop, and they fire at different moments.

The write durable facts arrow (A→B) fires at the end of a turn. The trigger is a sentence in the transcript that will still be true next week. A durable fact is one that outlives the task that produced it — a preference, a constraint, a correction. “I’m in Berlin” is durable; “what’s the weather there” is not. What to remember what to forget gives the full save/don’t-save rule; the working test is would I want to be told this again three days from now?

The retrieve relevant arrow (B→A) fires at the start of a turn, before the first model call. It takes the incoming user message as the query, ranks the store against it, and returns the top few facts as text that the caller splices into the array. “Relevant” is measured against that incoming message, not against the whole conversation — which is why a store holding four hundred facts can cost you forty tokens this turn.

The loop runs once per turn, not once per session, and that asymmetry is what gets probed. Write only at session end and a crash at minute 39 loses every fact the user stated. Retrieve only at session start and you never pick up a fact a parallel session wrote two turns ago.

Why a 1M-token window removes neither need

The obvious objection is that windows keep growing, so eventually you just put everything in. That fails for two independent reasons, both from chapter 00.

Reason 1 — cost scales with tokens, and the history is resent every turn

Total input tokens over n turns is:

total  ≈  n·P  +  a·n²/2

Two symbols, both from Deriving the numbers:

• P is the fixed prefix — the system prompt plus the tool schemas. It is paid on every request and it never grows.
• a is the number of tokens each turn adds to the array.

The first term is linear: you pay P once per request, n times. The second term is quadratic — it grows with the square of the turn count, so doubling the conversation length roughly quadruples that part of the bill.

Substitute real values. Take P = 3,000, a = 500, and 40 turns:

prefix term    n·P     =  40 × 3,000          =  120,000
history term   a·n²/2  =  500 × 40² / 2
                       =  500 × 1,600 / 2     =  400,000
                                                 ───────
total                                            520,000 input tokens

400,000 of those 520,000 tokens — 77% of the bill — is re-reading what you already sent.

The ≈ in the formula is load-bearing. The exact sum is n·P + a·n(n+1)/2 = 120,000 + 500 × (40 × 41 / 2) = 120,000 + 410,000 = 530,000, so the printed shorthand understates it by 1.9%. That gap is harmless here and is not always harmless — Managing growth hits a case where it is worth 600,000 tokens.

Filling a big window doesn’t cost you once. It costs you on every remaining turn.

Reason 2 — attention dilutes, and position matters

Attention is the mechanism by which the model, generating each new token, weighs every token already in the window to decide which ones matter here.

Those weights sum to a fixed total, so they behave like a fixed budget of notice being shared out: every token you add shrinks every other token’s share. That is dilution, and it is why a fact does not get easier to find just because it is present.

Dilution is only half of it. How reliably the model finds a fact also depends on where that fact sits in the window. Accuracy is U-shaped across position: high at the start, high at the end, worst in the middle.

So content in the middle of a full window is the least likely to be used, and a bigger window simply makes the middle bigger. Why quality degrades in long contexts derives both effects.

So the window is a limit you must stay under, and the budget is a discipline you impose well before it.

2. The four memory types

The dotted arrows in Two axes context engineering and memory’s diagram need somewhere to point — a store to write durable facts into and read them back from. The standard taxonomy of agent memory names four: working, episodic, semantic, and procedural.

The taxonomy answers: where does a given piece of information live, and how long does it survive? There are four stores because there are four different answers to “how long” — one turn, forever, until the fact changes, until the workflow changes. Everything else about a store — what it costs, how you query it, what breaks when it’s missing — follows from its lifespan.

In plain words, before the diagram:

• Working memory is what the agent is holding right now: the messages list from the introduction. It lives for one request.
• Episodic memory is what happened — a record of past runs and how they turned out, the way you remember last Tuesday’s deploy.
• Semantic memory is what is true — facts and preferences, held without any memory of when you learned them.
• Procedural memory is how to do things — the workflow, the steps, the house style.

The last two are the ones people confuse, so pin them with an example: “the user prefers metric” is semantic, and “this team ships with make ship” is procedural. One is a fact about the world; the other is a method.

The diagram adds the two things that list leaves out — each store’s lifespan, and where it physically lives. The bottom row is where both of those appear.

flowchart TD
    M[Agent memory] --> W["Working<br/>current window"]
    M --> E["Episodic<br/>what happened"]
    M --> S["Semantic<br/>durable facts"]
    M --> P["Procedural<br/>learned workflows"]

    W --> W1["Messages, tool results<br/>Lifespan: one turn<br/>Store: the array"]
    E --> E1["Past runs, outcomes<br/>Lifespan: forever<br/>Store: log + vector DB"]
    S --> S1["Facts, preferences<br/>Lifespan: until changed<br/>Store: KV or docs"]
    P --> P1["Skills, learned lessons<br/>Lifespan: versioned<br/>Store: files"]

    style W fill:#40916c,color:#fff
    style E fill:#7209b7,color:#fff
    style S fill:#7209b7,color:#fff
    style P fill:#7209b7,color:#fff

The colours carry the same code as the §1 diagram: Working is green because it is the within-session half; the other three are purple because they are the across-session half. That one visual split is the whole taxonomy — one store lives in your process, three live in storage.

The bottom row of that diagram is the part readers skip, and it is the part that decides your architecture.

• Working — the current window. Store: the array. That is literal: working memory is the messages list you hand to the API this request, and nothing else. It has no persistence layer because it was never anywhere but your process’s memory.
• Episodic — what happened. Store: log + vector DB. Past runs and their outcomes, appended as records and retrieved by similarity. A vector database (vector DB) stores each record alongside a list of numbers representing its meaning, so you can ask for “records that mean something like this” rather than “the record with this ID” — which is what you need, because “have I deployed this repo before?” is a fuzzy question, not a key lookup. Chapter 05 is the full treatment of that retrieval path.
• Semantic — durable facts. Store: KV or docs. A key–value store (KV) maps a name to a value, like a dictionary: user.timezone → "Europe/Berlin". It is keyed precisely so a new value can overwrite the old one instead of sitting beside it. Lifespan is until changed, which is a property of the key, not of an expiry timer.
• Procedural — learned workflows. Store: files. Skills and learned lessons as versioned text on disk: the same /srv/memories tree the memory tool in Long term memory reads and writes. Because it is files, two versions can be compared line by line and a change can go through code review — which is why versioned, not forever, is the right word for its lifespan.

The table below adds the column that matters in an interview — not what each store is, but what writes to it:

The four write triggers fire at four different times, and only one of them is periodic. Working memory is written every turn by construction. Episodic is written once, at the end. Semantic and procedural are event-driven — they fire on something the user said, not on a clock — which is why they are the two that get skipped in a first implementation and the two an interviewer will ask about.

Short-term = working. Long-term = the other three. That is the answer to “what are the types of agent memory,” and adding the write-trigger column is what makes it sound like you have built one.

One session, traced through all four

The write triggers in that last column are recitable but not yet usable — nobody can implement “on a stated fact” from the phrase alone. So here is one run of a release bot, with every write and every read-back named at the moment it happens.

Tuesday, 14:02 — session A.

Row 2 mentions CI, short for continuous integration — the service that runs checks on every push and rejected this one because the version tag was not cryptographically signed. The detail does not matter; what matters is that it is a failure the agent should not repeat.

Friday, 09:00 — session B, new process. The user types six words. Without memory, this is the whole request:

messages = [
    {"role": "user", "content": "Schedule the next release for the same time."},
]

The model cannot resolve “the same time” and does not know how this team ships. It asks two questions the user already answered on Tuesday — which is exactly the failure the table’s third and fourth rows predict.

With memory, the retrieve relevant arrow fires before this array is built, keyed on those six words.

The call below is written in the API of LangGraph, an open-source framework that models an agent as a graph of functions sharing state. Long term memory builds the whole store in it; only two pieces of the syntax matter here.

• The tuple ("memories", "u1") is a namespace — a folder path into the store. "memories" names the collection, and "u1" is this user’s identifier. That second element is what keeps one user’s facts out of another user’s results.
• limit=5 caps how many facts come back. Retrieval must return a handful, not the whole store.

As before, -> marks what the call returns. The returned values are flattened here to just the text of each hit, which is not the shape you actually get back. A real hit is an object: metadata plus a value dictionary, so the first line below is really hits[0].value["text"]. Long term memory uses exactly that expression. The strings are shown bare here because only their content matters to the point.

store.search(("memories", "u1"), query="Schedule the next release…", limit=5)
 -> "user.timezone = Europe/Berlin"                                   [semantic]
    "deploy: use `make ship`, never `git push --tags` (unsigned tags rejected)"
                                                                      [procedural]
    "run_412: unsigned tag rejected by CI; make ship worked"           [episodic]

Those three hits are spliced into the user turn, so the array the model actually sees becomes:

messages = [
    {"role": "user", "content":
        "[known about you]\n"
        "- timezone: Europe/Berlin\n"
        "- deploys with `make ship`, never `git push --tags`\n"
        "- run_412: unsigned tag was rejected by CI\n\n"
        "Schedule the next release for the same time."},
]

Three details in that array are the whole lesson.

The facts go in the user turn, not the system prompt. The system prompt is the part that gets cached and reused across requests. Retrieval changes per turn, so putting the facts there would throw away the cache on every request (The context window budget derives this).

Episodic came back too, unprompted, and it is what stops the agent from re-deriving Tuesday’s failure.

Nothing about the Tuesday transcript came back. The transcript died with the process. The three lines above are what was distilled from it.

Reads are not symmetric with writes

Semantic and procedural facts are read on every turn of every future session. They are small and almost always relevant.

Episodic memory is read on demand, because a log of every past run injected wholesale would swamp the window. That is the Store: log + vector DB label doing its job.

Working memory is never “read.” It is the request.

What that retrieved block costs

The block is neither free nor a one-off. It enters the array every turn, so it is part of the per-turn increment a from Two axes context engineering and memory. A block of m tokens over an n-turn session costs m·n, not m.

Measure m, never estimate it. client.messages.count_tokens(...) is a free endpoint: hand it the request you were about to send and it returns the token count. For a three-fact block like the one above, that is tens of tokens.

Put numbers on it. At m = 45 over a 20-turn session:

45 tokens × 20 turns  =  900 tokens

900 tokens is about a third of a single 1080p screenshot (Multi modal context) — and it buys you never asking the timezone question again.

3. The context window budget

Most of what you resend every turn never changes — the tool schemas, the system prompt, the settled early turns — yet the arithmetic in Two axes context engineering and memory billed every one of those tokens at full price, every time. It does not have to be billed that way.

The mechanism you are protecting

Before the model can generate anything, it has to read your entire request once. That read-through is called prefill, and it is what you are billed for as input tokens.

Prompt caching lets the server keep the result of prefilling a prefix of your request and reuse it next time instead of prefilling it again. A prefix here is literal: tokens 0 through j, in order — the leading run of the request, with nothing skipped.

The two multipliers, which every cost claim in this chapter uses:

• Reads bill at roughly 0.1× the normal input rate — a token served from cache costs a tenth of a token prefilled fresh.
• Writes cost 1.25×, once, on the request that populates the entry.

Why a prefix match and not a general one: the model reads left to right, so a token’s stored computation depends on itself and everything before it, and on nothing after it (Prompt caching derived). Change a token at position 30 and every stored value from 30 onward is wrong.

That single fact turns ordering into a cost decision. The diagram below is the request laid out top to bottom in render order, with the most stable content first.

flowchart TD
    CTX["Context window"] --> A["Tool schemas<br/>renders FIRST · stable"]
    A --> B["System prompt<br/>stable"]
    B --> BP{{cache breakpoint}}
    BP --> C["Retrieved docs<br/>semi-stable"]
    C --> D["Conversation history<br/>grows"]
    D --> E["Current turn<br/>volatile"]

    style A fill:#2d6a4f,color:#fff
    style B fill:#2d6a4f,color:#fff
    style BP fill:#2d6a4f,color:#fff
    style D fill:#bc6c25,color:#fff
    style E fill:#9d0208,color:#fff

Order is not cosmetic — it is your cache hit rate. Cache hit rate is the fraction of your input tokens that get served from cache instead of recomputed.

The render order is fixed by the API: tools → system → messages. So tool schemas sit at the top of the diagram because they render first and rarely change. The current turn sits at the bottom because it is different on every request. The conversation history sits between them and grows by a tokens a turn. The ordering is by volatility, cheapest-to-keep first.

The colours encode that volatility, read top to bottom:

• Dark green is stable — bytes that are identical on every request. That is exactly what can sit inside the cached prefix.
• The uncoloured Retrieved docs box is semi-stable — identical within a turn, different between turns.
• Orange is growing — the history, which gains a tokens a turn.
• Red is volatile — the current turn, which is new text by definition.

The cache breakpoint is green because it belongs at the last stable boundary. Everything below it is a colour that changes.

The cache breakpoint node

That node is a real API object, not a diagram convention: a cache_control marker you attach to one content block, meaning cache everything from position 0 through this block. Three properties fix where it goes.

• It belongs on the last block whose bytes never change. Push it later and you cache volatile content, which writes a fresh entry every request and reads none — you pay the write premium forever and never collect.
• ephemeral is a lifetime, not a storage class. It sets a time to live (TTL) — how long the entry survives unused — of roughly five minutes, refreshed on each hit. A longer TTL exists and costs more to write, so it pays off only across a bigger gap between requests.
• You get at most four breakpoints per request. They are a budget. Spend them on stability boundaries — after tools plus system, after retrieved documents, after the last completed turn — not on every block that looks important.

The fourth property, which raises no error

A prefix below the model’s minimum cacheable length silently does not cache.

Every model refuses to cache a prefix shorter than some floor, because the bookkeeping would cost more than the saving. That floor is model-specific. The values in circulation are 512, 1,024, 2,048 and 4,096 tokens.

The trap is that the floor does not only move in one direction as models get newer. The newest models sit at the low end, around 512, while models further up the range are still in service. So a prompt that cached fine yesterday can stop caching after a model swap: no code change, no warning, and every token that used to bill at 0.1× now billing at 1×.

Do not memorize the mapping. It changes, and the API will tell you. You never have to look the floor up, because a prefix under it is indistinguishable from a prefix that was never cached at all. Send the same request twice and read cache_creation_input_tokens and cache_read_input_tokens on the second response. Both zero means the prefix never cached, and being under the minimum is one of the two reasons why. The three-step diagnosis below writes that check out.

Why Retrieved docs sits below the breakpoint

That placement is deliberate, and it is the one worth arguing about in an interview.

Retrieved passages are stable within a turn and change between turns. Put them before the breakpoint and they are cheap whenever the query repeats and catastrophic whenever it does not, because a changed passage invalidates everything after it. Put them after the breakpoint and they cost full price every time — but they cost it once, instead of taking the whole prefix down with them.

Long term memory shows the split that gets both: stable facts inside the cached prefix, volatile retrieved ones after it.

The classic bug, and exactly why it’s catastrophic

One line accounts for most broken caches in the wild, and it is worth tracing all the way through, because the damage is wildly out of proportion to the change.

# ✗ Kills caching for the whole conversation, forever.
system = f"You are an assistant. Current time: {datetime.now()}"

Walk the mechanism one step at a time.

1. The timestamp is about 8 tokens, sitting near position 30 of the request.
2. Those 8 tokens differ on every request, so the stored computation for token 30 differs.
3. Each token’s stored computation depends on everything before it. So every token from position 30 onward now has a different preceding context and cannot be reused.
4. Your 3,000-token system prompt and your 40,000-token history are all downstream of position 30.

You changed 8 tokens and invalidated 43,000.

# ✓ Stable prefix; volatile fact injected after the cached span.
system = [{"type": "text", "text": STABLE_PROMPT,
           "cache_control": {"type": "ephemeral"}}]
messages = [..., {"role": "user", "content": f"[time: {now}] {user_msg}"}]

Two lines there do the work, and neither is obvious.

system becomes a list of blocks rather than a string. That is the only shape that can carry a cache_control marker at all, so the type change is the fix, not tidying up around it.

The timestamp moves into the user message. Not because user messages are special — because messages render last. Anything placed there is already past the breakpoint, so it may differ on every request for free.

Generalize it as “volatile content goes after the last breakpoint” and the specific fix stops being something you have to remember.

The audit list

There are six common ways to get a differing byte into the prefix of two otherwise identical requests. They are worth memorizing as a set, because the interview question is “how would you debug it,” and naming the population of suspects is most of the answer.

Rows 1–3 are accidents; rows 4–6 are decisions.

The accidents are cheap to fix once you can see them.

The decisions need a fact this chapter has been assuming without stating it: a cache entry is shared by every request that shares the prefix. It is not scoped to one conversation or one user.

That is what makes row 4 expensive. A tool list built per user gives every user a private prefix starting at position 0, so a thousand users pay a thousand cache writes for a prompt that could have paid one.

Diagnosing it, in three steps

The symptom of a broken cache is never specific — the bill simply did not drop — so the diagnosis has to be a procedure rather than a hunch. Do these three in order, because step 3 is expensive and steps 1 and 2 tell you whether it is even the right hunt.

1. Send the same request twice. Request 1 always reports a zero read — there was nothing there to read yet. The measurement starts at request 2, and skipping this is how people conclude caching is broken on their very first call.

2. On request 2, read both counters, not one.

print(resp.usage.cache_creation_input_tokens)   # tokens written to the cache
print(resp.usage.cache_read_input_tokens)       # tokens served from the cache

The pair is what separates two completely different zeros:

3. Only then, diff the bytes. The SDK will not hand you the rendered prompt, so serialize the request body yourself — the same dictionary you pass to messages.create() — for both requests, and compare the two files:

import json, pathlib

def dump(body: dict, path: str) -> None:
    """Serialize deterministically, or the diff reports noise the API never saw."""
    pathlib.Path(path).write_text(json.dumps(body, sort_keys=True, indent=2))

The first differing byte is your bug.

sort_keys=True is not cosmetic here. It is the same fix as row 2 of the audit table, applied to your diagnostic instead of your prompt. Without it, two dictionaries holding identical data can serialize in different key orders, and the diff points at a difference the API never saw.

This is a five-minute fix, and on a cache-heavy workload the payback is large. Treat the 3–10× you will see quoted as a practitioner estimate rather than a measurement; your own number is whatever fraction of your prefix stopped hitting, which the two counters above hand you directly.

Editing the system prompt mid-session

The last row of the audit table has a fix worth knowing. Instead of editing the top-level system parameter, append a {"role": "system", "content": "..."} entry to messages[]. It sits after the cached prefix, so the history survives, and it still carries operator authority rather than reading as user text.

That is a third role, and it contradicts the array you met at the top of this chapter — deliberately. The opening example held only user and assistant, and on most models those really are the only two roles messages[] will accept. Standing instructions go in the separate top-level system parameter, outside the array.

A system role inside the array is a newer, model-gated addition. It exists precisely for this case: an instruction that arrives mid-conversation and must not be retro-fitted into the prefix. Two constraints come with it — it has to follow a user turn rather than open the array, and the model has to support it.

Checking support is an experiment, not a lookup, because the answer moves with the model list. Send one throwaway request on the exact model string you deploy: a user turn, then a one-line system message. A model that does not support the role returns a 400 — the HTTP status for a malformed request — with a message naming the role.

Do that in a startup check rather than in production, so an unsupported model fails on deploy instead of on a user’s turn 30.

Context rot, and what to do about it

“Context rot” is the name for quality falling as the window fills. It is two effects, not one — attention dilution and U-shaped positional recall, the same pair Two axes context engineering and memory used against the big-window argument, both derived in Why quality degrades in long contexts.

Three practical consequences, each a direct read-off from that U-shaped curve:

• Restate the task near the end of each turn, not only in the system prompt. The end is a high-recall position; the system prompt becomes the start of an increasingly long window, and everything you added since is between them.
• Long tool outputs push instructions into the middle — the worst position. This is why truncating tool output improves quality, not just cost.
• “It worked at turn 3, it’s confused at turn 40” is positional, not a model regression. Do not respond by switching models.

4. Managing growth

Caching makes the resend cheap, but it does not make the array smaller — and both the quadratic bill and context rot come from size. When the history gets too big, there are four levers, cheapest first.

flowchart LR
    G[History too big] --> T["1. Trim<br/>drop the oldest turns"]
    G --> C["2. Clear<br/>prune stale tool results"]
    G --> S["3. Compact<br/>summary block replaces prefix"]
    G --> O["4. Offload<br/>pointer replaces payload"]

    T --> T1["Cheap · loses detail<br/>rewrites the prefix"]
    C --> C1["Cheap · keeps structure<br/>rewrites the prefix"]
    S --> S1["1 extra call · lossy<br/>rewrites the prefix"]
    O --> O1["Best · needs a read tool<br/>shrinks per-turn growth 4000x"]

    style O fill:#2d6a4f,color:#fff

Each right-hand box makes three claims: what the lever does, what it costs, and whether it touches the prefix. Only 4. Offload is coloured, and it is the same dark green as the cached, stable blocks in The context window budget’s diagram. That is the point being made in colour: offloading is the only one of the four that leaves the cached prefix intact.

Here is each one in words:

• Trim drops the oldest turns off the front of the array. Cheap, and it loses detail you cannot get back.
• Clear prunes stale tool results. It keeps structure — the turn boundaries survive, only the payloads go.
• Compact replaces a span of old turns with a short summary of them. It buys one extra call to the model and gives you a summary block in place of the turns it covers, which is lossy by definition.
• Offload writes the payload to disk, so that in the array a pointer replaces payload. A pointer here is a one-line reference such as a file path. This is the only one of the four that needs a read tool, because the agent has to be able to fetch back what you moved.

The right-hand column is the ranking, and the first three rows say the same thing there. Trim, clear, and compact all rewrite the prefix, so all three cost you the cache from the edit point onward. That is one fact stated three times, not a coincidence. Only offloading appends, and appending is the single edit that leaves the prefix untouched.

Why offloading is the one to name

The others reduce a growing number. Offloading shrinks the number that gets multiplied by the turn count.

without offload:  context(n) = P + n·a          cumulative cost O(n²)
with offload:     context(n) = P + n·ε          cumulative cost O(n²)  ← same, on purpose:
                                                 same shape, constant 4000x smaller
                                                 (ε is a one-line pointer)

Two pieces of notation in that block. O(n²) is big-O notation: it names how a total grows as the number of turns n grows, ignoring constants, so O(n²) means “with the square of the turn count.” And ε (epsilon) is the conventional symbol for a very small quantity — here, the token size of the one-line pointer that replaces the payload.

Both sides are still quadratic — say so out loud, because the sloppy version of this claim gets caught. P + n·ε is linear in n, not flat, and summing it over n turns is still O(n²). What offloading changes is the coefficient a, and it changes it by a factor that makes the exponent almost irrelevant.

The worked example

Set up the run:

• 20 turns.
• Each turn makes a tool call that returns a 60,000-token file dump — picture a data-analysis agent working through a table one query at a time. That per-turn premise is what the arithmetic bills, so it has to be the premise: one dump total would be a different and much smaller calculation.
• With offloading, each dump is written to /tmp/analysis.json and the array keeps only the one-line result "wrote 4,812 rows to /tmp/analysis.json" — about 15 tokens.
• The fixed prefix is P = 3,000.

Now count the dumps. Turn 1 sends its own dump. Turn 2 sends turn 1’s and its own. Turn t sends t of them. So the total number of dump-payloads billed across the run is:

Σ t  for t = 1..20   =   1 + 2 + 3 + … + 20   =   20·21/2   =   210

(Σ, capital sigma, is the summation sign: “add up this expression for every value of t in the stated range.”)

Note the switch of formula. Two axes context engineering and memory printed the shorthand a·n²/2; this is the exact form n(n+1)/2 that the shorthand approximates. They count the same thing, and the exact form charges a payload on turn 1 as well as on the deltas after it, which is the convention Deriving the numbers sets out.

The gap between the two is worth naming here only because the payload is huge:

shorthand   n²/2       =  20²/2      =  200 payloads
exact       n(n+1)/2   =  20·21/2    =  210 payloads
difference                              10 payloads × 60,000 tok = 600,000 tokens

When the per-turn increment is large, use the exact form.

Now the two bills, each n·P plus the payload total:

without offload:  20·3,000 + 60,000·210  =  60,000 + 12,600,000  =  12,660,000 input tokens
with offload:     20·3,000 +     15·210  =  60,000 +      3,150  =      63,150 input tokens
                                                                    ───────────────────────
                              12,660,000 / 63,150  ≈  200x cumulative
                                  60,000 / 15      =  4000x per turn at the margin

200× on the cumulative bill, and 4000× on the per-turn increment, from a change that adds one line of code and one tool.

That is a constant-factor win: the shape of the growth curve is unchanged, every number on it is just 4000× smaller. An asymptotic win would change the shape itself. This constant-factor win is worth more than most asymptotic ones, and it is why long-running coding agents work at all.

There is a second benefit people miss. Trim, clear, and compact all damage the cache — each rewrites the prefix, invalidating everything after the edit point. Offloading does not: you are appending a short result instead of a long one, so the prefix stays intact and keeps hitting. The 200× above is computed before counting the cache reads you get to keep.

Compaction

Compaction is replacing a long span of old turns with a short summary of them, in place, so the conversation continues with the same meaning and a fraction of the tokens.

Settle one thing before the diagram, because the whole section hinges on it: the summarizing happens on the server. That is why the arrows below run to the model and back. You do not write a summarizer prompt, and you do not choose when it runs.

As the window fills past a threshold — on the order of a hundred thousand tokens — the API summarizes the older turns itself and hands the summary block back to you inside the normal response. That threshold is not your max_tokens, which caps only the length of this reply.

Your only job is to persist the block. It is state. Drop it and the next request re-sends the uncompacted history as though nothing had happened.

The sequence diagram below is that exchange: what you send, what comes back, and what your code does with it.

sequenceDiagram
    participant A as Agent
    participant M as Model
    A->>M: turns 1..40 (approaching limit)
    M-->>A: compaction block (summary of 1..30)
    Note over A: history := [summary, turns 31..40]
    A->>M: next turn — window back to normal

The 1..30 / 31..40 split in that diagram is the shape of every compaction: an old span becomes one block, a recent tail stays verbatim. Recent turns are kept whole because they are what the agent is currently acting on, and summarizing them would cost accuracy on the only turns where accuracy is still live.

So read the next table as two things at once: what the managed version is trying to preserve for you, and — if you roll your own summarizer, which you will the moment you need domain-specific retention — the specification you write against.

The rule: keep what you’d need to hand this to another engineer. The most expensive omission is “we already tried X and it broke Y” — drop that and the agent immediately retries X.

Here is the managed feature on the wire. Two things to watch: the feature is named twice, in two different spellings, and the response has to be appended whole rather than unwrapped.

resp = client.beta.messages.create(
    betas=["compact-2026-01-12"],
    model="claude-opus-5",
    max_tokens=16000,
    messages=messages,
    context_management={"edits": [{"type": "compact_20260112"}]},
)
# CRITICAL: append the full content, not just the text.
# The compaction block IS the state — extracting .text silently drops it,
# and the next request re-sends uncompacted history with no error.
messages.append({"role": "assistant", "content": resp.content})

# Reading it back out: response content is a list of typed blocks, and the
# compaction block sits alongside the text one rather than replacing it.
for block in resp.content:
    if block.type == "compaction":
        print(block.content)          # what the server chose to keep

The feature is named twice in that call because the two names do different jobs.

betas=[...] is the opt-in. A beta is a pre-release feature you must ask for by name, and this one says “this account is willing to receive a response shape that did not exist before.” Without it, the server refuses the request.

context_management={"edits": [...]} is the configuration. edits is a list of strategies to apply to the context, and compaction is one entry in it. That is why it is a list at all — the same parameter carries the sibling feature described below.

The two spellings are not interchangeable, and this is the exact thing people mistype:

• compact-2026-01-12 — hyphens, full ISO date — is a beta header value and belongs only in betas.
• compact_20260112 — underscores, compact date — is a strategy type and belongs only in context_management.

They name the same feature in two different namespaces with two different spelling conventions. Swapping them does not “still work”: you get a rejected request, and the error names the field rather than the typo.

Context editing, the sibling feature

Context editing is a separate beta with its own header and its own strategy types. It is not a mode of compaction.

Editing clears stale tool results rather than summarizing them: the turns stay, the payloads go. A second strategy clears old thinking blocks — the model’s own recorded reasoning, which comes back as its own kind of content block.

Use editing when old outputs are simply irrelevant, and compaction when the narrative matters. Both rewrite the prefix, so both cost you the cache from the edit point onward. Budget for one expensive uncached turn after each.

5. Long-term memory

Every lever so far has managed the green, within-session half of Two axes context engineering and memory’s diagram. The purple half — the store behind the write durable facts and retrieve relevant arrows — still has to be built.

The three tiers below are that store, the same design written three ways, from most abstract to most concrete.

• Tier 1 is the algorithm with no library at all.
• Tier 2 is the same thing in LangGraph, the framework behind The four memory types’s retrieval call.
• Tier 3 is the same thing against Anthropic’s software development kit (SDK) — the client library you call the API through — where you implement the storage backend yourself.

Tier 1 — Pseudocode

Strip every library away and the whole design is two functions, one fired at the end of a turn and one at the start of the next.

on_turn_end(session):
    facts = extract_durable_facts(session)
    for f in facts:
        if contradicts(f, store): store.update(f)     # replace, don't append
        else: store.add(f)

on_turn_start(user_msg):
    relevant = store.search(user_msg, k=5)
    context.inject_after_breakpoint(relevant)   # NOT prepend — see below

Read it as the two arrows from Two axes context engineering and memory, now with bodies. At the end of a turn, pull out the durable facts and either replace a conflicting one or add a new one. At the start of a turn, search the store with the incoming message and splice the top k results — here five — into the request.

extract_durable_facts is a model call

That is the one box left unopened, and nothing but a model can do it. Deciding that “I’m in Berlin” is worth keeping while “what’s the weather there” is not requires reading the sentence.

So in practice you make a second, cheap model call at the end of each turn. You hand it the turn’s transcript and ask for a short list of durable facts. What to remember what to forget’s save/don’t-save table is the rubric you put in that call’s prompt.

Budget for it. It is one extra call per turn, which is why teams run it on a small model, and often only on turns where the user actually said something new.

inject_after_breakpoint, and why prepend is wrong

The obvious version of that last line is prepend, and it breaks both of The context window budget’s rules at once:

1. It puts per-turn-varying content at the front of the prefix, invalidating the cache on every request.
2. It parks the facts in the low-recall middle of the window once the history grows past them.

Inject after the cache breakpoint and late in the array instead. One session traced through all four does exactly this, splicing the retrieved block into the user turn.

Resolving contradictions

Three parts of this design are hard, and they are exactly what gets probed: what’s worth saving (What to remember what to forget answers it), how contradictions resolve, and when to forget (What to remember what to forget again).

Contradictions are the one that gets flagged and then skipped, so have the answer ready. contradicts(f, store) in the pseudocode is not free — you have to detect the conflict before you can replace anything, and there are exactly three ways to detect it. The table compares them on what each catches and what each costs.

Two terms in that table need unpacking.

An embedding is a list of numbers — typically several hundred — that a separate small model produces from a piece of text, arranged so that texts with similar meanings get similar numbers. “Embedding similarity” is therefore a cheap numeric test for “do these two sentences mean roughly the same thing.” Its weakness is that negation barely moves the numbers, which is why “likes Python” and “dislikes Python” look nearly identical to it.

An LLM judge — LLM being large language model, the thing you are already calling — is an ordinary model call whose job is to answer a yes/no question about text. Here the question is “does this new fact contradict any of these?” It is accurate because it actually reads the sentences, and slow and expensive for the same reason.

The usual production answer is key-first, judge on collision: keys handle the 90% you designed for, and the judge only runs when the cheap layer is ambiguous.

Tier 2 — LangGraph

The framework gives you two scopes, and getting the split between them right is most of a memory design: one scope for the conversation, one for what outlives it. Read the block below as scaffolding, not as a template to copy — two of its lines are marked as wrong on purpose and both are corrected before the section ends.

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.store.memory import InMemoryStore

# ✗ Development only: a dictionary in this process. It dies with the process,
#   which is the one thing long-term memory must not do. Swap in a persistent
#   store (e.g. PostgresStore) before this is real. See the note below.
store = InMemoryStore()

def node(state, *, store):
    uid = state["user_id"]
    hits = store.search(("memories", uid), query=state["messages"][-1].content, limit=5)
    facts = "\n".join(h.value["text"] for h in hits)
    # ✗ Retrieved facts in the *system* position — this breaks caching.
    #   Corrected at the end of this section.
    reply = llm.invoke([("system", f"Known about user:\n{facts}"), *state["messages"]])
    return {"messages": [reply]}

app = graph.compile(checkpointer=PostgresSaver(conn), store=store)
app.invoke({...}, config={"configurable": {"thread_id": "t1", "user_id": "u1"}})

The two scopes map onto two objects:

• thread_id scopes the conversation. It is handled by the checkpointer, which saves the graph’s state to a database after each step so a run can be resumed.
• user_id scopes what outlives the conversation. That is the store.

Notice the asymmetry between those two, because it is the first thing to fix. The checkpointer gets PostgresSaver — a real database — while the store gets InMemoryStore(), which is a Python dictionary living in this process.

That inverts the durability the two scopes need. The checkpointer holds conversation state, which by this chapter’s own definition is the half that may die when the process does. The store holds the facts that are supposed to outlive the process, and a dictionary cannot do that.

InMemoryStore is the right choice for a test or a demo and a defect in production. Replace it with LangGraph’s persistent store backend against the same Postgres instance and the rest of the code above is unchanged, because both back ends implement the same search/put interface.

Two more notes on that block. It is a fragment, not a runnable file — conn, graph, and llm come from your application.

And one line that looks like a typo is not. The keyword-only store parameter in def node(state, *, store) shadows the module-level store on purpose — “shadows” meaning the parameter deliberately hides the outer variable of the same name. That is how LangGraph injects the store you passed to compile(), so the node never reaches for a global and can be tested by handing it a fake.

h.value["text"] is the only place in this chapter you see a stored record’s shape: a hit is an object with metadata plus a value dictionary, not a bare string.

The caching cost of this pattern

Injecting retrieved facts into the system position means the prefix changes whenever retrieval changes. That is the same prepend bug as Tier 1, wearing a framework.

The fix is a split, not a move — keep the stable half where it was and send the volatile half down past the breakpoint:

# ✗ One system string, rebuilt per turn. Retrieval changes → prefix changes →
#   tools + system + every prior turn re-prefill at full price.
llm.invoke([("system", f"{STABLE_PROMPT}\nKnown about user:\n{facts}"), *msgs])

# ✓ Stable half stays byte-identical and cached; volatile half rides the user
#   turn, after the breakpoint, where changing every request is free.
llm.invoke([
    ("system", STABLE_PROMPT),                      # cached prefix
    *msgs[:-1],
    ("user", f"[known about you]\n{facts}\n\n{msgs[-1].content}"),
])

Tier 3 — Anthropic SDK

At this tier there is no store object at all. Instead the model gets a memory tool: a small filesystem it can read from and write to, with commands like view, create, str_replace and delete. The model decides what to save and where; you implement the backend that executes those commands. That is the procedural store from The four memory types — files on disk — made concrete.

The handler below is that backend. Read it as two pieces: safe(), which is the whole security story, and handle_memory(), which dispatches one command per branch.

import os, pathlib, anthropic

client = anthropic.Anthropic()
ROOT = pathlib.Path("/srv/memories").resolve()

def safe(path: str) -> pathlib.Path:
    """Model-supplied paths are untrusted input. Confine to ROOT."""
    p = (ROOT / path.lstrip("/")).resolve()      # resolve() BEFORE the check
    if not p.is_relative_to(ROOT):               # catches .., symlinks, absolute
        raise ValueError(f"path escapes memory root: {path}")
    return p

def handle_memory(cmd: dict) -> str:
    op = cmd["command"]
    if op == "view":
        p = safe(cmd["path"])
        return "\n".join(sorted(x.name for x in p.iterdir())) if p.is_dir() \
               else p.read_text()
    if op == "create":
        p = safe(cmd["path"]); p.parent.mkdir(parents=True, exist_ok=True)
        p.write_text(cmd["file_text"]); return f"wrote {cmd['path']}"
    if op == "str_replace":
        p = safe(cmd["path"]); t = p.read_text()
        if t.count(cmd["old_str"]) != 1:
            raise ValueError("old_str must match exactly once")
        p.write_text(t.replace(cmd["old_str"], cmd["new_str"]))
        return "ok"
    if op == "delete":
        safe(cmd["path"]).unlink(); return "deleted"
    raise ValueError(op)

That handler is half of a tool. The other half is what connects it to anything.

The model only reaches the handler if the tool is declared in the request. And the handler’s return value only reaches the model if you send it back as a tool_result — a message entry that carries a tool’s output back into the conversation.

One thing to read off the declaration below: memory_20250818 is a dated variant of the memory tool, not a version number. The provider ships several variants at once and you pick the behaviour you want, the way you pick a model. The date labels when that variant’s behaviour was fixed; a later date does not mean “newer, use this one.” (Sandboxed code execution works through a case where two dated variants of the same tool are live simultaneously.)

TOOLS = [{"type": "memory_20250818", "name": "memory"}]

resp = client.messages.create(model="claude-opus-5", max_tokens=4096,
                              tools=TOOLS, messages=messages)

results = []
for block in resp.content:
    if block.type == "tool_use" and block.name == "memory":
        # block.input IS the cmd dict handle_memory() expects: {"command": ...}
        try:
            out, failed = handle_memory(block.input), False
        except ValueError as e:
            # The error must reach the MODEL, not just your logs. is_error=True
            # is how a tool_result says "this failed" instead of returning the
            # words 'path escapes memory root' as if they were the file.
            out, failed = str(e), True
        results.append({"type": "tool_result", "tool_use_id": block.id,
                        "content": out, "is_error": failed})
if results:
    messages += [{"role": "assistant", "content": resp.content},
                 {"role": "user", "content": results}]

Errors have to reach the model

Note what handle_memory covers: four commands, and the set is not exhaustive. Insert-at-line and rename also exist, and the raise ValueError(op) at the bottom is what you will hit the first time the model reaches for one.

Raising is the right default, because an unknown command should be loud rather than silent. But a raise that escapes the loop kills the run, so the try in the block above converts it into a tool_result carrying is_error: True.

That field is the whole point. Without it, the exception text is delivered as ordinary tool output, and the model reads “path escapes memory root” as the contents of the file it asked for. With it, the model sees a failure and can stop asking.

safe() is the entire security story

The ordering inside it matters. resolve() must run before the containment check.

resolve() collapses .. path segments and follows symbolic links, so a path like ../../etc/passwd — and a symlink pointing outside ROOT — both come out as their real destination, and both then fail is_relative_to(ROOT).

Doing the string check first and resolving later is the classic bypass: the string looks contained, and the real path is not.

And never store secrets in memory. Memories are replayed verbatim into every future session, so a key written once leaks into every later context — including sessions belonging to other users, if the per-user separation of your storage keys is wrong.

6. What to remember, what to forget

A store you only ever write to gets worse, not better, with age — so no memory design is finished until it can refuse a fact and delete one. Start with what earns a place in the store at all.

PII is personally identifiable information — names, addresses, account numbers, anything that identifies a specific human. It appears in the “don’t save” column not because it is never useful but because storing it needs a retention and deletion policy first, and a memory store built without one is a compliance problem that ships silently.

Forgetting is a feature, not an omission. Every memory is a permanent context tax on every future run, and an unbounded store degrades quality by diluting attention across irrelevant facts (Why quality degrades in long contexts). So you need a way to remove things. Eviction is the general name for that: deleting a stored item to make room, or because it has stopped being worth its space.

Three mechanisms, and they are not interchangeable.

1. TTL — delete a fact N days after it was last read, not last written.

Measuring from the write is the bug. A preference stated once in January and used on every session since would expire on its birthday while still being correct. Measuring from the read turns the TTL into a usage signal.

TTL is a good default for facts with a natural shelf life (project.deadline) and wrong for facts without one (user.timezone).

2. Relevance decay — multiply the retrieval score by a factor that falls with age.

A common form is:

adjusted_score  =  score × 0.5 ^ (age_days / half_life)

which halves a fact’s ranking score every half_life days. Substitute to see the shape: at half_life = 30, a fact 30 days old keeps 0.5^1 = 50% of its score, one 60 days old keeps 0.5^2 = 25%, and one 90 days old keeps 0.5^3 = 12.5%.

Nothing is deleted. Old facts just stop winning a place in the top k — the handful of highest-scoring facts that retrieval actually returns. That makes decay the right mechanism when you cannot tell whether a fact is stale or merely quiet, because it is reversible: one fresh mention restores it.

3. Size cap with LRU eviction — hard-bound the store at N facts per user.

When the store is full, evict the least recently used (LRU) fact, meaning the one that has gone longest without being retrieved. This is the only one of the three mechanisms that gives you a cost guarantee, which is why production stores usually run it underneath one of the others.

And contradicting writes must replace, not accumulate — a store holding both “prefers metric” and “prefers imperial” is worse than a store holding neither, because the model will pick one arbitrarily and sound confident. Detection is the hard half; Long term memory has the three ways to do it.

7. State in complex workflows

Facts are not the only thing worth keeping. A run long enough to fail partway through also has task state — open subtasks, artifacts written, failures — and none of it belongs in the message array.

The diagram below groups all of that into one State object and gives it a single operation: a checkpoint after every step.

flowchart LR
    subgraph State
        M["messages[]"]
        T["task queue"]
        A["artifacts<br/>file paths, not blobs"]
        B["budget ledger"]
        E["error history"]
    end
    State --> CP[(Atomic checkpoint<br/>after each step)]
    CP --> R[Resume / replay / debug]

    style CP fill:#7209b7,color:#fff

Four of those boxes are not messages[], and that is the point — each holds something the transcript either cannot express or can only express by making the model re-derive it.

• task queue — the open subtasks and their statuses. It lives outside the messages because it must be queryable (“what is still PENDING?”) without a model call, and because on resume you need the queue as data, not as prose the agent has to re-read and re-parse.
• artifacts — the outputs the run produced, held as file paths, not blobs — a blob being the raw bytes of the file inlined into the state, and a path being a short string pointing at them. Rule 3 below is why the distinction matters.
• budget ledger — tokens and dollars spent so far, per step. Deriving it from the transcript means summing usage across forty responses you may have already compacted away; keeping it in state makes “stop at $2” a cheap if.
• error history — every failure and what was tried, kept separately from the transcript because it is the first thing compaction drops (Managing growth’s Keep/Drop table calls it the most expensive omission) and the last thing you can afford to lose.

The Atomic checkpoint after each step node is where that state gets written. A checkpoint is a snapshot of the whole state, saved to disk; atomic means a reader only ever sees the complete old snapshot or the complete new one, never a half-written mixture of the two.

The Resume / replay / debug node is three payoffs from that one artifact, and they are not the same thing. Resume picks the run back up where it stopped. Replay re-runs a recorded sequence to reproduce a failure without re-paying for the model calls that led to it. Debug is reading the checkpoints as a time series — which is only possible if state is small enough to compare, which is rule 3.

Three rules:

1. Checkpoint after every step, not on a timer. A 40-minute agent that crashes at minute 39 must resume, not restart.

2. Write atomically — write to a temporary file, then rename it over the destination. A crash mid-write otherwise leaves a torn checkpoint, which is worse than none because it fails at load time, after you have already lost the run.

   import json, os, tempfile
   
   def save_atomic(state: dict, final: str) -> None:
       """rename() is atomic only WITHIN a filesystem — so the temp file must
       be created in the destination's directory, not /tmp."""
       d = os.path.dirname(final) or "."
       fd, tmp = tempfile.mkstemp(dir=d)
       with os.fdopen(fd, "w") as f:
           json.dump(state, f)
           f.flush()
           os.fsync(f.fileno())      # bytes on disk before the rename, not after
       os.replace(tmp, final)        # atomic: readers see old or new, never torn

   Two preconditions the one-liner hides. tmp and final must be on the same filesystem, because a rename across filesystems is really a copy, and a copy is not atomic — which is why the temporary file is created in the destination’s own directory rather than /tmp. And the fsync, which forces the operating system to flush its write buffers to the physical disk, must happen before the rename, or a power loss can leave a correctly-named file full of zeros.

3. Store artifacts by reference. State holds file paths and IDs; bytes stay on disk. State carrying blobs cannot be cheaply serialized, compared, or logged — and debug in the diagram is exactly “compare two checkpoints,” which stops working the moment a 60k-token dump is inline.

And the subtle one: mark a task IN_PROGRESS before executing it. On resume, an IN_PROGRESS task means “unknown outcome” — it must be verified, never blindly re-executed, or a resume double-sends the email.

8. Multi-modal context

Everything priced so far has been text. Multi-modal means the context holds more than that: images, documents, audio. All of it is converted to tokens and billed on the same meter as prose — and images deserve the arithmetic, because their price changed with the current model generation and the obvious way to save money on them goes wrong once caching is on.

The table below is the price and the trap for each. One term in it: the Files API is a separate endpoint that stores an upload server-side and hands you a file_id, so a document you reference on twenty turns is transmitted once rather than twenty times.

How an image becomes a token count

Name the vision tier when you quote those numbers, because they moved. A vision tier here is just the generation of the model’s image handling. Two generations are in circulation and they price 1080p differently.

An image is not billed by area. The model cuts it into 28×28-pixel patches and charges one token per patch:

tokens  =  ceil(width / 28) × ceil(height / 28)

(ceil is “round up” — a partial patch still costs a whole token.)

Each tier caps two things: the long edge in pixels, and the number of tokens one image may cost. An image over either cap is downscaled until it fits both.

On the current tier the long edge caps at 2,576 px and the token count at 4,784. A 1080p screenshot is 1920 × 1080, which is under both caps, so it is not downscaled:

ceil(1920 / 28)  =  ceil(68.57)  =  69 patches across
ceil(1080 / 28)  =  ceil(38.57)  =  39 patches down
                    69 × 39      =  2,691 tokens

The “~1.5k at 1080p” figure you will see quoted belongs to the older, standard tier. That tier caps the long edge at 1,568 px and the image at 1,568 tokens, which is tight enough that 1080p gets downscaled to 1456 × 819:

ceil(1456 / 28)  =  52
ceil( 819 / 28)  =  30      (29.25, rounded up)
                    52 × 30  =  1,560 tokens

So the common phrasing “~1.5k at 1080p, up to ~4.8k at max” silently mixes two generations: the first half is the standard tier, the second half is the current one. Both were true, never at the same time.

The practical consequence is that “capture at native 1080p, it gets downscaled anyway” is now wrong. On the current tier nothing is thrown away, and you pay 2,691 / 1,560 ≈ 1.7× what the same screenshot used to cost.

The resend arithmetic

That 1.7× makes resending worse than the old numbers implied. An untrimmed 20-step run resends every image on every turn — turn 1 carries 1 image, turn 2 carries 2, turn t carries t — so the image count summed over turns is Σ t for t = 1..20 = 210:

Σ (t · 2,691) for t = 1..20  =  2,691 · 210  =  565,110 image tokens

Images are the one context type you cannot compress. Text can be summarized and stay useful; a summarized screenshot is useless. So the only two levers are trim old ones and lower the resolution.

But those two levers are not equivalent, and the ranking inverts once caching is on.

Trimming means a sliding window: keep the most recent few images, drop the rest. That rewrites the prefix every turn, because each turn advances the point where the old and new requests stop matching by one image. Every surviving image is then re-processed at full rate, forever.

Lowering the resolution shrinks every image without touching the prefix at all.

Setting up the comparison

Every cell in the block below is measured against one baseline: the same 20-step run with neither lever applied — every image kept, at full resolution, for a total of the 565,110 image tokens computed above. Uncached is that baseline with no prompt caching in play. Cached is that baseline with a working cached prefix, which is the situation you are actually in.

Four more setup facts, because a ratio without its assumptions is worthless:

• The trim level is keep_last = 3 — the three most recent screenshots stay, everything older is replaced by a one-line placeholder. The placeholder is counted at zero tokens below, because ten-odd tokens against a 2,691-token image does not survive rounding. A trim ratio without its keep level means nothing, because the keep level is the amount of context you decided to pay for.
• Prices are claude-opus-5 at $5 / MTok input. MTok means one million tokens, so $5 / MTok is $0.000005 per token.
• Cache reads bill at 0.1× and cache writes at 1.25× (Prompt caching derived).
• Only image tokens are counted, because those are the tokens the two levers act on. Output bills at $25 / MTok and is identical in all four rows.

The block below prices the same two options twice, first without caching and then with it. Watch the arrow at the end of each pair — it flips from WIN to LOSS between the two halves.

images resent per turn, 20 steps, 2,691 tok/image, keep_last = 3
turn t          1   2   3   4   5  …  20     Σ images      Σ image tokens
no trim         1   2   3   4   5  …  20         210              565,110
keep_last = 3   1   2   3   3   3  …   3          57              153,387

UNCACHED — every token billed once, at 1×
  no trim      565,110 × 1.00  =  565,110 billed tok  =  $2.826
  keep 3       153,387 × 1.00  =  153,387 billed tok  =  $0.767   →  3.7× WIN

CACHED — reads at 0.1×, writes at 1.25×
  no trim      reads   511,290 × 0.10  =   51,129
               writes   53,820 × 1.25  =   67,275
                                          ────────
                                           118,404 billed tok  =  $0.592
  keep 3       reads     8,073 × 0.10  =      807
               writes  145,314 × 1.25  =  181,643
                                          ────────
                                           182,450 billed tok  =  $0.912   →  1.5× LOSS

Where the read/write split comes from is the only non-mechanical step, so check it row by row.

Untrimmed, the array is append-only. On turn t, the t−1 images already in it are byte-identical to last turn’s request and are served from cache; only the one new image is written.

reads   Σ (t−1) · 2,691  for t = 1..20  =  190 × 2,691  =  511,290
writes           20 · 2,691             =                    53,820
                                                          ─────────
                                            sums back to    565,110  ✓ the baseline

Trimmed, the first trim fires on turn 4. From then on the oldest surviving image is dropped every single turn, so this request stops matching the last one at the very first image block, and nothing after that point can be read from cache.

Only turns 2 and 3 — before any trimming has happened — get a cache read at all:

reads    turn 2: 1 image  +  turn 3: 2 images  =  3 × 2,691  =    8,073
writes   153,387 total image tokens  −  8,073  =                145,314

Those 145,314 tokens are rewritten at the 1.25× premium, every turn.

Read the sign flip in the trim row as the whole lesson.

Uncached, dropping old images removes tokens you were paying full price for. That is a 3.7× win.

Cached, those same tokens were already being served at 0.1× — a 90% discount off the normal rate — so removing them saves very little. Meanwhile the prefix rewrite the trim causes forces every surviving image to be written to the cache again at 1.25×.

Those are the cache’s two multipliers, 0.1× on reads and 1.25× on writes. The 90% is just 1 − 0.1 restated as a percentage, not a third multiplier. You give up a tenth-rate bill to buy a quarter-over-full-rate one, and the trade comes out a 1.5× loss.

So: resolution first, and reach for trimming only past ~60 steps, in batches rather than per turn.

Two honest caveats on those ratios

Counting image tokens only makes the ratios as large as they can be. It is what keeps the arithmetic anchored to this section’s single baseline, but the system prompt, the per-turn assistant text and the output are the same in all four rows — and adding a constant to both sides of a ratio pulls it toward 1.

The magnitudes move with the run length. The uncached win grows with the step count, because untrimmed is quadratic while trimmed is linear. The cached loss does not.

The signs are the durable part; the numbers belong to this baseline. Case study 01 is a second data point at a different scale — a 7-turn task on the older per-image number, also at keep_last = 3. There the same model gives a 1.4× win uncached ($0.265 untrimmed against $0.190 trimmed) and a 1.9× loss cached ($0.205 trimmed against $0.107 untrimmed). Different run, different image price, same flip.

Measure before optimizing, because the resolution-to-token curve is not linear. The patch formula rounds up on each axis, so shrinking an image by 10% may cost you nothing at all until the new width or height crosses a 28-pixel patch boundary.

The same count_tokens endpoint from The four memory types prices an image. b64 below is the image file read and base64-encoded, which is how bytes travel inside a JSON request:

n = client.messages.count_tokens(
    model="claude-opus-5",
    messages=[{"role": "user", "content": [
        {"type": "image",
         "source": {"type": "base64", "media_type": "image/png", "data": b64}},
    ]}],
).input_tokens

Run that across the resolutions you are considering before picking one. A 20% width cut that buys 36% fewer tokens is a different decision from one that buys 5%, and only the measurement tells you which you got.

And the audio row deserves a mitigation, since every other gotcha here gets one. Transcription errors are silent because a transcript is plain text, and text looks authoritative. Two things blunt that.

Keep the pointer alongside the transcript. Offload the audio (Managing growth), store the timestamped span next to the text, and let the agent re-listen to a segment when a decision turns on an exact string — an amount, a name, a command.

Carry the transcriber’s per-span confidence into the context. A low-confidence span is a place to ask the user, not a place to guess.

Cheat sheet

Each row is a symptom you can observe from outside, the mechanism underneath it, and the fix. Read it symptom-first — that is the order a real debugging session arrives in.

Next: 05 — RAG for Agents.