Memory and Context Management

An agent makes two decisions on every turn: what to send the model on this request, and what to keep after the process exits. In this lesson, we’ll work through both, pricing each choice before we commit to it.

By the end you should be able to:

• name the four kinds of memory an agent can hold, and say what writes to each
• lay out a request so prompt caching keeps paying. Caching is the server storing the result of reading a repeated opening chunk of a request, so the next request starting with the same bytes is billed at a fraction of the rate, The context window budget has the mechanism
• choose among the four ways to stop a conversation from growing without bound
• decide what a long-term store should save, overwrite, and delete

These skills sit behind three common interview questions: price a memory design before building it, explain why an agent that behaved at turn 3 is confused at turn 40, and debug a bill that tripled after a one-line change.

The shape of one request

Start with what goes in and what comes out.

What goes in

The application programming interface (API) you call is stateless: the server keeps nothing between requests. Everything the model knows on this request is in this request.

What you send is one object with three parts that carry text:

• tools: the schemas for every function the model may call. A schema is the JSON description of a function’s name, purpose, and parameters.
• system: standing instructions.
• messages: the conversation.

That third part is a plain list, and it is what people mean by “context”:

messages = [
    {"role": "user",      "content": "What's the capital of France?"},
    {"role": "assistant", "content": "Paris."},
    {"role": "user",      "content": "And its population?"},
]

That list is the entire memory of the conversation. There is no session on the server that also remembers “Paris”; delete the middle entry and the model cannot answer the third.

A turn is one user message plus everything the agent does in response. The list above holds one complete turn and the start of a second.

What comes out

The response carries the reply plus a usage record of what you were billed for. Below, -> is not Python; it means “print this expression and here is what you see.”

resp = client.messages.create(
    model="claude-opus-5", max_tokens=1024,
    system=SYSTEM, tools=TOOLS, messages=messages,
)

resp.content  ->  [TextBlock(text="About 2.1 million.")]
resp.usage    ->  Usage(input_tokens=41, output_tokens=7,
                        cache_creation_input_tokens=0, cache_read_input_tokens=0)

resp.content is a list of content blocks, not a string. Here it holds a single TextBlock; a request that triggers a tool call returns a differently typed block in the same list.

resp.usage is where every cost claim in this chapter is measured. input_tokens is what you paid to have your request read. The two cache_ counters are zero because nothing was cached yet.

Why the array only grows

Your code appends the reply to messages and sends the whole list again next request. The array only grows, and you pay for all of it every turn.

A token is the unit the model and the bill are both denominated in, roughly four characters of English.

Two ceilings act on the array. The context window is the hard cap on tokens in one request, prompt and output together, on the order of 200,000 tokens for current models. It is a limit, not a budget: nothing bills differently at 199k than at 20k. The budget is what you impose on yourself, because every token in the array is paid for again on every remaining turn.

Memory is the discipline of deciding what “everything” means: which text earns its place this turn, and which facts are worth writing somewhere durable so a future array can be rebuilt without them.

Two axes: context engineering and memory

Two distinct problems both get called “memory”: curating one request, and persisting facts after the process exits. Separating them is the first thing to get right.

flowchart TD
    subgraph WITHIN["Within a session — Context engineering"]
        A["What goes in the window this turn,<br/>and in what order"]
    end
    subgraph ACROSS["Across sessions — Memory"]
        B["What survives after<br/>the process exits"]
    end
    A -.->|write durable facts| B
    B -.->|retrieve relevant| A

    style WITHIN fill:#40916c,color:#fff
    style ACROSS fill:#7209b7,color:#fff

Context engineering curates one prompt: what goes into the window this turn, and in what order. It is a list in your process’s memory that dies in seconds.

Memory is persistence: what survives after the process exits. It is a database row that outlives the process.

The two arrows, and when each fires

The two dotted arrows are the whole loop, and they fire at different moments.

Write durable facts (A→B) fires at the end of a turn. A durable fact outlives the task that produced it: a preference, a constraint, a correction. “I’m in Berlin” is durable; “what’s the weather there” is not. The working test is would I want to be told this again three days from now?

Retrieve relevant (B→A) fires at the start of a turn, before the first model call. It takes the incoming user message as the query, ranks the store against it, and returns the top few facts as text the caller splices into the array. “Relevant” is measured against that incoming message, not the whole conversation, which is why a store of four hundred facts can cost forty tokens this turn.

The loop runs once per turn, not once per session. Write only at session end and a crash at minute 39 loses every fact the user stated. Retrieve only at session start and you miss a fact a parallel session wrote two turns ago.

Why a bigger window removes neither need

The obvious objection is that windows keep growing, so eventually you just put everything in. That fails for two independent reasons, both from the LLM internals chapter.

Cost scales with tokens, and history is resent every turn. Over n turns, total input tokens are roughly:

total  ≈  n·P  +  a·n²/2

P is the fixed prefix (system prompt plus tool schemas), paid on every request and never growing. a is the tokens each turn adds. The first term is linear; the second is quadratic, so doubling the conversation length roughly quadruples that part of the bill. With P = 3,000, a = 500, and 40 turns, that is about 120,000 + 400,000 = 520,000 input tokens, and 77% of it is re-reading what you already sent. Filling a big window doesn’t cost you once; it costs you on every remaining turn.

Attention dilutes, and position matters. Attention is how the model weighs every token already in the window when generating the next one. Those weights sum to a fixed total, so every token you add shrinks every other token’s share; a fact does not get easier to find just because it is present. And recall is U-shaped across position: high at the start, high at the end, worst in the middle. A bigger window just makes the middle bigger. Why quality degrades in long contexts derives both effects.

The four memory types

The dotted arrows need somewhere to point: a store to write durable facts into and read them back from. The standard taxonomy names four, and there are four because there are four answers to “how long does a piece of information survive”: one turn, forever, until the fact changes, until the workflow changes.

• Working memory is what the agent holds right now: the messages list. It lives for one request.
• Episodic memory is what happened: a record of past runs and how they turned out.
• Semantic memory is what is true: facts and preferences, held without any memory of when you learned them.
• Procedural memory is how to do things: the workflow, the steps, the house style.

The last two are the ones people confuse. “The user prefers metric” is semantic; “this team ships with make ship” is procedural. One is a fact about the world, the other is a method.

flowchart TD
    M[Agent memory] --> W["Working<br/>current window"]
    M --> E["Episodic<br/>what happened"]
    M --> S["Semantic<br/>durable facts"]
    M --> P["Procedural<br/>learned workflows"]

    W --> W1["Messages, tool results<br/>Lifespan: one turn<br/>Store: the array"]
    E --> E1["Past runs, outcomes<br/>Lifespan: forever<br/>Store: log + vector DB"]
    S --> S1["Facts, preferences<br/>Lifespan: until changed<br/>Store: KV or docs"]
    P --> P1["Skills, learned lessons<br/>Lifespan: versioned<br/>Store: files"]

    style W fill:#40916c,color:#fff
    style E fill:#7209b7,color:#fff
    style S fill:#7209b7,color:#fff
    style P fill:#7209b7,color:#fff

Working memory lives in your process; the other three live in storage. That split is the whole taxonomy.

• Working: the array itself. The messages list you hand to the API this request, nothing else. It has no persistence layer because it was never anywhere but your process.
• Episodic: log + vector DB. Past runs and outcomes, appended as records and retrieved by similarity. A vector database (vector DB) stores each record alongside numbers representing its meaning, so you can ask for “records that mean something like this” instead of “the record with this ID”, which is what “have I deployed this before?” needs, a fuzzy question, not a key lookup. The RAG for agents chapter covers that retrieval path.
• Semantic: KV or docs. A key–value store (KV) maps a name to a value like a dictionary (user.timezone → "Europe/Berlin"). It is keyed precisely so a new value overwrites the old one instead of sitting beside it. Lifespan is until changed, a property of the key, not an expiry timer.
• Procedural: files. Skills and learned lessons as versioned text on disk: the same /srv/memories tree the memory tool in Long-term memory reads and writes. Because it is files, two versions compare line by line and a change can go through code review, which is why versioned, not forever, is the right word.

The column that turns the taxonomy into a design is what writes to each store:

The four write triggers fire at four different times, and only one is periodic. Working is written every turn by construction. Episodic once, at run end. Semantic and procedural are event-driven (they fire on something the user said, not on a clock) which is why they are the two most often skipped in a first implementation. Short-term is working; long-term is the other three.

One session, traced through all four

Here is one run of a release bot, with every write and every read-back named at the moment it happens.

Tuesday, 14:02: session A.

CI is continuous integration, the service that runs checks on every push; it rejected this one because the tag was not cryptographically signed. What matters is that it is a failure the agent should not repeat.

Friday, 09:00: session B, new process. The user types six words. Without memory, this is the whole request:

messages = [
    {"role": "user", "content": "Schedule the next release for the same time."},
]

The model cannot resolve “the same time” and does not know how this team ships, so it asks two questions the user already answered Tuesday.

With memory, retrieve fires before the array is built, keyed on those six words. The call below uses LangGraph, an open-source framework that models an agent as a graph of functions sharing state. Two pieces of syntax matter: the tuple ("memories", "u1") is a namespace (a folder path into the store; "u1" keeps one user’s facts out of another’s results), and limit=5 caps how many facts come back.

As before, -> marks what the call returns; the values are flattened to the text of each hit. A real hit is an object (metadata plus a value dictionary, so the first line is really hits[0].value["text"]).

store.search(("memories", "u1"), query="Schedule the next release…", limit=5)
 -> "user.timezone = Europe/Berlin"                                   [semantic]
    "deploy: use `make ship`, never `git push --tags` (unsigned tags rejected)"
                                                                      [procedural]
    "run_412: unsigned tag rejected by CI; make ship worked"           [episodic]

Those hits are spliced into the user turn, so the array the model sees becomes:

messages = [
    {"role": "user", "content":
        "[known about you]\n"
        "- timezone: Europe/Berlin\n"
        "- deploys with `make ship`, never `git push --tags`\n"
        "- run_412: unsigned tag was rejected by CI\n\n"
        "Schedule the next release for the same time."},
]

Three details carry the point. The facts go in the user turn, not the system prompt. The system prompt is what gets cached and reused, and retrieval changes per turn, so putting the facts there would throw away the cache every request. Episodic came back too, unprompted, and it stops the agent from re-deriving Tuesday’s failure. And nothing of the Tuesday transcript came back; the transcript died with the process. The three lines are what was distilled from it.

Reads are not symmetric with writes

Semantic and procedural facts are read on every turn of every future session: small and almost always relevant. Episodic is read on demand, because a log of every past run injected wholesale would swamp the window. Working memory is never read; it is the request.

What the retrieved block costs

The block enters the array every turn, so it is part of the per-turn increment a: a block of m tokens over an n-turn session costs m·n, not m. Measure m with client.messages.count_tokens(...), a free endpoint that returns the token count of a request you were about to send. A three-fact block is tens of tokens; at m = 45 over 20 turns that is 900 tokens, and it buys never asking the timezone question again.

The context window budget

Most of what you resend every turn never changes, tool schemas, system prompt, settled early turns, yet the arithmetic above billed all of it at full price every time. It does not have to be.

The mechanism you are protecting

Before generating anything, the model reads your entire request once. That read-through is prefill, billed as input tokens. Prompt caching lets the server keep the result of prefilling a prefix of your request and reuse it next time. A prefix is literal: tokens 0 through j in order, the leading run of the request with nothing skipped.

Two multipliers, used by every cost claim in this chapter:

• Reads bill at roughly 0.1× the normal input rate.
• Writes cost 1.25×, once, on the request that populates the entry.

Why a prefix and not a general match: the model reads left to right, so a token’s stored computation depends on itself and everything before it, and on nothing after it (Prompt caching derived). Change a token at position 30 and every stored value from 30 on is wrong. That turns ordering into a cost decision. Lay the request out most-stable-first:

flowchart TD
    CTX["Context window"] --> A["Tool schemas<br/>renders FIRST · stable"]
    A --> B["System prompt<br/>stable"]
    B --> BP{{cache breakpoint}}
    BP --> C["Retrieved docs<br/>semi-stable"]
    C --> D["Conversation history<br/>grows"]
    D --> E["Current turn<br/>volatile"]

    style A fill:#2d6a4f,color:#fff
    style B fill:#2d6a4f,color:#fff
    style BP fill:#2d6a4f,color:#fff
    style D fill:#bc6c25,color:#fff
    style E fill:#9d0208,color:#fff

The render order is fixed by the API: tools → system → messages. Tool schemas sit first because they rarely change; the current turn sits last because it differs every request; history sits between and grows by a tokens a turn. Order by volatility, cheapest-to-keep first, order determines your cache hit rate, the fraction of input tokens served from cache instead of recomputed.

The cache breakpoint

The breakpoint is a real API object: a cache_control marker on one content block meaning cache everything from position 0 through here. Three properties fix where it goes:

• It belongs on the last block whose bytes never change. Push it later and you cache volatile content, writing a fresh entry every request and reading none, paying the write premium forever.
• ephemeral is a lifetime, not a storage class. It sets a time to live (TTL) of roughly five minutes, refreshed on each hit. A longer TTL exists, costs more to write, and pays off only across a bigger gap between requests.
• You get at most four breakpoints per request. Spend them on stability boundaries (after tools plus system, after retrieved documents, after the last completed turn), not on every block that looks important.

A fourth property raises no error: a prefix below the model’s minimum cacheable length silently does not cache. Every model refuses to cache a prefix shorter than some floor, because the bookkeeping would cost more than the saving. The floor is model-specific; values in circulation are 512, 1,024, 2,048, and 4,096 tokens. It does not only move one way as models get newer, the newest sit around 512 while older models still in service sit higher, so a prompt that cached yesterday can stop caching after a model swap, with no code change and no warning. Don’t memorize the mapping: a prefix under the floor is indistinguishable from one never cached, and the API will tell you (the three-step diagnosis below).

Retrieved docs sit below the breakpoint deliberately. They are stable within a turn and change between turns. Put them before the breakpoint and a changed passage invalidates everything after it; put them after, and they cost full price every time but only once, instead of taking the whole prefix down. Long-term memory shows the split that gets both: stable facts inside the cached prefix, volatile retrieved ones after it.

The classic bug

One line accounts for most broken caches in the wild:

# ✗ Kills caching for the whole conversation, forever.
system = f"You are an assistant. Current time: {datetime.now()}"

The timestamp is about 8 tokens near position 30, and it differs every request. Each token’s stored computation depends on everything before it, so every token from position 30 onward (your 3,000-token system prompt and 40,000-token history) now has a different preceding context and cannot be reused. You changed 8 tokens and invalidated 43,000.

# ✓ Stable prefix; volatile fact injected after the cached span.
system = [{"type": "text", "text": STABLE_PROMPT,
           "cache_control": {"type": "ephemeral"}}]
messages = [..., {"role": "user", "content": f"[time: {now}] {user_msg}"}]

Two changes do the work. system becomes a list of blocks, not a string: the only shape that can carry a cache_control marker. And the timestamp moves into the user message, which renders last and so is already past the breakpoint, where it may differ every request for free. The rule: volatile content goes after the last breakpoint.

The audit list

Six common ways to get a differing byte into the prefix of two otherwise identical requests:

Rows 1–3 are accidents, cheap to fix once you can see them. Rows 4–6 are decisions, and they need a fact assumed until now: a cache entry is shared by every request that shares the prefix, not scoped to one conversation or user. That is what makes row 4 expensive, a tool list built per user gives every user a private prefix from position 0, so a thousand users pay a thousand cache writes for a prompt that could have paid one.

Diagnosing it, in three steps

The symptom is never specific: the bill simply did not drop. Do these in order, because step 3 is expensive.

1. Send the same request twice. Request 1 always reports a zero read; the measurement starts at request 2. Skipping this is how people conclude caching is broken on their first call.

2. On request 2, read both counters, not one.

print(resp.usage.cache_creation_input_tokens)   # tokens written to the cache
print(resp.usage.cache_read_input_tokens)       # tokens served from the cache

The pair separates two completely different zeros:

3. Only then, diff the bytes. The SDK won’t hand you the rendered prompt, so serialize both request bodies yourself and compare:

import json, pathlib

def dump(body: dict, path: str) -> None:
    """Serialize deterministically, or the diff reports noise the API never saw."""
    pathlib.Path(path).write_text(json.dumps(body, sort_keys=True, indent=2))

The first differing byte is your bug. sort_keys=True is the same fix as row 2 of the audit table applied to your diagnostic: without it, identical data can serialize in different key orders and the diff points at a difference the API never saw. On a cache-heavy workload the payback is large; treat the 3–10× you’ll see quoted as a practitioner estimate, and read your own number off the two counters.

Editing the system prompt mid-session

The last audit row has a fix worth knowing. Instead of editing the top-level system parameter, append a {"role": "system", "content": "..."} entry to messages[]. It sits after the cached prefix, so the history survives, and it still carries operator authority instead of reading as user text.

That is a third role, and it contradicts the two-role array from the top of the chapter, deliberately. On most models messages[] accepts only user and assistant, with standing instructions in the separate top-level system parameter. A system role inside the array is a newer, model-gated addition for exactly this case: an instruction that arrives mid-conversation and must not be retrofitted into the prefix. It has to follow a user turn, and the model has to support it. Check support with one throwaway request on the exact model string you deploy; an unsupported model returns a 400 (the HTTP status for a malformed request) naming the role. Do that in a startup check, so an unsupported model fails on deploy, not on a user’s turn 30.

Context rot, and what to do about it

“Context rot” is quality falling as the window fills. It is two effects, not one, attention dilution and U-shaped positional recall, the same pair used above against the big-window argument. Three practical consequences:

• Restate the task near the end of each turn, not only in the system prompt. The end is a high-recall position; the system prompt becomes the start of an ever-longer window.
• Long tool outputs push instructions into the middle, the worst position. Truncating tool output improves quality, not just cost.
• “It worked at turn 3, it’s confused at turn 40” is positional, not a model regression. Don’t respond by switching models.

Managing growth

Caching makes the resend cheap but does not make the array smaller, and both the quadratic bill and context rot come from size. When the history gets too big there are four levers, cheapest first.

flowchart LR
    G[History too big] --> T["1. Trim<br/>drop the oldest turns"]
    G --> C["2. Clear<br/>prune stale tool results"]
    G --> S["3. Compact<br/>summary block replaces prefix"]
    G --> O["4. Offload<br/>pointer replaces payload"]

    T --> T1["Cheap · loses detail<br/>rewrites the prefix"]
    C --> C1["Cheap · keeps structure<br/>rewrites the prefix"]
    S --> S1["1 extra call · lossy<br/>rewrites the prefix"]
    O --> O1["Best · needs a read tool<br/>shrinks per-turn growth 4000x"]

    style O fill:#2d6a4f,color:#fff

• Trim drops the oldest turns off the front. Cheap, and it loses detail you can’t get back.
• Clear prunes stale tool results. It keeps structure: the turn boundaries survive, only the payloads go.
• Compact replaces a span of old turns with a short summary. It buys one extra model call and is lossy by definition.
• Offload writes the payload to disk and leaves a one-line pointer (a file path) in the array. It is the only one that needs a read tool, so the agent can fetch back what you moved.

Trim, clear, and compact all rewrite the prefix, so all three cost you the cache from the edit point onward. Only offloading appends, and appending is the one edit that leaves the prefix untouched.

Why offloading is the one to name

The others reduce a growing number; offloading shrinks the number that gets multiplied by the turn count. Take a 20-turn run where each turn makes a tool call returning a 60,000-token file dump (picture a data-analysis agent working through a table one query at a time). Because the whole array is resent every turn, the dumps billed across the run total 1 + 2 + … + 20 = 210 payloads. With offloading each dump is written to /tmp/analysis.json and the array keeps only a ~15-token pointer.

The two bills, each n·P (with P = 3,000) plus the payloads:

without offload:  20·3,000 + 60,000·210  ≈  12.7M input tokens
with offload:     20·3,000 +     15·210  ≈  63k  input tokens
                  ≈ 200x cumulative, 4000x per turn at the margin

That is a constant-factor win: the growth curve keeps its shape, every number on it is just 4000× smaller. This is why long-running coding agents work at all. There is a second benefit: trim, clear, and compact all rewrite the prefix and damage the cache, while offloading appends and leaves the prefix hitting. The 200× is computed before counting the cache reads you get to keep.

Compaction

Compaction replaces a long span of old turns with a short summary of them, in place, so the conversation continues with the same meaning and a fraction of the tokens. The point the whole section hinges on: the summarizing happens on the server. As the window fills past a threshold (on the order of a hundred thousand tokens), the API summarizes the older turns itself and hands the summary block back inside the normal response. That threshold is not your max_tokens, which caps only this reply’s length. Your only job is to persist the block: it is state; drop it and the next request re-sends the uncompacted history.

sequenceDiagram
    participant A as Agent
    participant M as Model
    A->>M: turns 1..40 (approaching limit)
    M-->>A: compaction block (summary of 1..30)
    Note over A: history := [summary, turns 31..40]
    A->>M: next turn — window back to normal

Every compaction keeps a recent tail verbatim and turns the older span into one block. Recent turns stay whole because they are what the agent is currently acting on. If you write your own summarizer for domain-specific retention, this is the spec:

Keep what you’d need to hand this to another engineer. The most expensive omission is “we already tried X and it broke Y”, drop that and the agent retries X.

The managed feature on the wire:

resp = client.beta.messages.create(
    betas=["compact-2026-01-12"],
    model="claude-opus-5",
    max_tokens=16000,
    messages=messages,
    context_management={"edits": [{"type": "compact_20260112"}]},
)
# CRITICAL: append the full content, not just the text.
# The compaction block IS the state — extracting .text silently drops it,
# and the next request re-sends uncompacted history with no error.
messages.append({"role": "assistant", "content": resp.content})

# Reading it back out: response content is a list of typed blocks, and the
# compaction block sits alongside the text one rather than replacing it.
for block in resp.content:
    if block.type == "compaction":
        print(block.content)          # what the server chose to keep

Two names do different jobs. betas=[...] is the opt-in: a beta is a pre-release feature you ask for by name, and this one says the account will accept a response shape that did not exist before; without it, the server refuses. context_management={"edits": [...]} is the configuration, a list of strategies, with compaction one entry.

The two spellings are not interchangeable and are the exact thing people mistype:

• compact-2026-01-12 (hyphens, full ISO date) is a beta header value and belongs only in betas.
• compact_20260112 (underscores) is a strategy type and belongs only in context_management.

Swapping them gets a rejected request whose error names the field, not the typo.

Context editing, the sibling feature

Context editing is a separate beta with its own header and strategy types, not a mode of compaction. It clears stale tool results instead of summarizing them: the turns stay, the payloads go. A second strategy clears old thinking blocks (the model’s own recorded reasoning). Use editing when old outputs are simply irrelevant, compaction when the narrative matters. Both rewrite the prefix, so both cost the cache from the edit point onward; budget for one expensive uncached turn after each.

Long-term memory

Every lever so far has managed the within-session half. The store behind the write durable facts and retrieve relevant arrows still has to be built. Below is the same design three ways, most abstract to most concrete: pseudocode, then LangGraph, then Anthropic’s software development kit (SDK), the client library you call the API through, where you implement the storage backend yourself.

Tier 1 — Pseudocode

Strip every library away and the design is two functions, one fired at the end of a turn and one at the start of the next:

on_turn_end(session):
    facts = extract_durable_facts(session)
    for f in facts:
        if contradicts(f, store): store.update(f)     # replace, don't append
        else: store.add(f)

on_turn_start(user_msg):
    relevant = store.search(user_msg, k=5)
    context.inject_after_breakpoint(relevant)   # NOT prepend — see below

At turn end, pull out the durable facts and either replace a conflicting one or add a new one. At turn start, search the store with the incoming message and splice the top k results (here five) into the request.

extract_durable_facts is a model call

Nothing but a model can decide that “I’m in Berlin” is worth keeping while “what’s the weather there” is not. It requires reading the sentence. So in practice you make a second, cheap model call at the end of each turn: hand it the transcript, ask for a short list of durable facts, using the save/don’t-save rule from What to remember, what to forget as the rubric. Budget for it: one extra call per turn, so teams run it on a small model and often only on turns where the user said something new.

inject_after_breakpoint, not prepend

The obvious version of that last line is prepend, which breaks both caching rules at once: it puts per-turn-varying content at the front of the prefix (invalidating the cache every request), and it parks the facts in the low-recall middle once the history grows past them. Inject after the cache breakpoint and late in the array instead, as One session, traced through all four does.

Resolving contradictions

contradicts(f, store) is not free: you have to detect the conflict before you can replace anything, and there are three ways:

An embedding is a list of numbers (typically several hundred) a small model produces from text, arranged so similar meanings get similar numbers; “embedding similarity” is a cheap numeric test for “do these mean roughly the same thing,” but negation barely moves the numbers, so “likes Python” and “dislikes Python” look nearly identical. An LLM judge is an ordinary model call answering a yes/no question about text: accurate because it reads the sentences, slow and expensive for the same reason.

The usual production answer is key-first, judge on collision: keys handle the 90% you designed for, and the judge runs only when the cheap layer is ambiguous.

Tier 2 — LangGraph

The framework gives two scopes, and getting the split right is most of a memory design: one for the conversation, one for what outlives it. Read the block as scaffolding, not a template, two lines are marked wrong on purpose and corrected before the section ends.

from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.store.memory import InMemoryStore

# ✗ Development only: a dictionary in this process. It dies with the process,
# which is the one thing long-term memory must not do. Swap in a persistent
# store (e.g. PostgresStore) before this is real. See the note below.
store = InMemoryStore()

def node(state, *, store):
    uid = state["user_id"]
    hits = store.search(("memories", uid), query=state["messages"][-1].content, limit=5)
    facts = "\n".join(h.value["text"] for h in hits)
    # ✗ Retrieved facts in the *system* position — this breaks caching.
    #   Corrected at the end of this section.
    reply = llm.invoke([("system", f"Known about user:\n{facts}"), *state["messages"]])
    return {"messages": [reply]}

app = graph.compile(checkpointer=PostgresSaver(conn), store=store)
app.invoke({...}, config={"configurable": {"thread_id": "t1", "user_id": "u1"}})

The two scopes map to two objects. thread_id scopes the conversation, handled by the checkpointer, which saves the graph’s state to a database after each step so a run can resume. user_id scopes what outlives the conversation: the store.

The asymmetry is the first thing to fix. The checkpointer gets PostgresSaver, a real database, while the store gets InMemoryStore(), a Python dictionary in this process. That inverts the durability the two scopes need: the checkpointer holds conversation state (the half that may die with the process), while the store holds facts meant to outlive the process, and a dictionary cannot. InMemoryStore is right for a test and a defect in production; swap in a persistent store backend against the same Postgres and the rest is unchanged, because both back ends implement the same search/put interface.

Two more notes. The block is a fragment, conn, graph, and llm come from your application. And def node(state, *, store) is not a typo: the keyword-only store parameter deliberately shadows the module-level store, which is how LangGraph injects the store you passed to compile() so the node never reaches for a global and can be tested with a fake. h.value["text"] is the stored record’s shape: a hit is an object with metadata plus a value dictionary, not a bare string.

Injecting retrieved facts into the system position means the prefix changes whenever retrieval changes, the same prepend bug wearing a framework. The fix is a split, not a move: keep the stable half where it was and send the volatile half past the breakpoint.

# ✗ One system string, rebuilt per turn. Retrieval changes → prefix changes →
# tools + system + every prior turn re-prefill at full price.
llm.invoke([("system", f"{STABLE_PROMPT}\nKnown about user:\n{facts}"), *msgs])

# ✓ Stable half stays byte-identical and cached; volatile half rides the user
# turn, after the breakpoint, where changing every request is free.
llm.invoke([
    ("system", STABLE_PROMPT),                      # cached prefix
    *msgs[:-1],
    ("user", f"[known about you]\n{facts}\n\n{msgs[-1].content}"),
])

Tier 3 — Anthropic SDK

At this tier there is no store object. The model gets a memory tool: a small filesystem it can read from and write to, with commands like view, create, str_replace, and delete. The model decides what to save and where; you implement the backend that executes those commands. That is the procedural store, files on disk, made concrete.

import os, pathlib, anthropic

client = anthropic.Anthropic()
ROOT = pathlib.Path("/srv/memories").resolve()

def safe(path: str) -> pathlib.Path:
    """Model-supplied paths are untrusted input. Confine to ROOT."""
    p = (ROOT / path.lstrip("/")).resolve()      # resolve() BEFORE the check
    if not p.is_relative_to(ROOT):               # catches .., symlinks, absolute
        raise ValueError(f"path escapes memory root: {path}")
    return p

def handle_memory(cmd: dict) -> str:
    op = cmd["command"]
    if op == "view":
        p = safe(cmd["path"])
        return "\n".join(sorted(x.name for x in p.iterdir())) if p.is_dir() \
               else p.read_text()
    if op == "create":
        p = safe(cmd["path"]); p.parent.mkdir(parents=True, exist_ok=True)
        p.write_text(cmd["file_text"]); return f"wrote {cmd['path']}"
    if op == "str_replace":
        p = safe(cmd["path"]); t = p.read_text()
        if t.count(cmd["old_str"]) != 1:
            raise ValueError("old_str must match exactly once")
        p.write_text(t.replace(cmd["old_str"], cmd["new_str"]))
        return "ok"
    if op == "delete":
        safe(cmd["path"]).unlink(); return "deleted"
    raise ValueError(op)

That handler is half of a tool. The model only reaches it if the tool is declared in the request, and its return value only reaches the model if you send it back as a tool_result (a message entry carrying a tool’s output back into the conversation). One thing to read off the declaration: memory_20250818 is a dated variant of the memory tool, not a version number. The provider ships several variants at once and you pick the behaviour you want, the way you pick a model; a later date does not mean “newer, use this one.” (Sandboxed code execution works through a case where two dated variants are live at once.)

TOOLS = [{"type": "memory_20250818", "name": "memory"}]

resp = client.messages.create(model="claude-opus-5", max_tokens=4096,
                              tools=TOOLS, messages=messages)

results = []
for block in resp.content:
    if block.type == "tool_use" and block.name == "memory":
        # block.input IS the cmd dict handle_memory() expects: {"command": ...}
        try:
            out, failed = handle_memory(block.input), False
        except ValueError as e:
            # The error must reach the MODEL, not just your logs. is_error=True
            # is how a tool_result says "this failed" instead of returning the
            # words 'path escapes memory root' as if they were the file.
            out, failed = str(e), True
        results.append({"type": "tool_result", "tool_use_id": block.id,
                        "content": out, "is_error": failed})
if results:
    messages += [{"role": "assistant", "content": resp.content},
                 {"role": "user", "content": results}]

Errors have to reach the model

handle_memory covers four commands, and the set is not exhaustive, insert-at-line and rename also exist, and raise ValueError(op) is what you hit the first time the model reaches for one. Raising is the right default (an unknown command should be loud), but a raise that escapes the loop kills the run, so the try converts it into a tool_result carrying is_error: True. Without that field, the exception text is delivered as ordinary tool output and the model reads “path escapes memory root” as the contents of the file it asked for. With it, the model sees a failure and can stop asking.

safe() is the entire security story

The ordering matters: resolve() must run before the containment check. resolve() collapses .. segments and follows symbolic links, so ../../etc/passwd, and a symlink pointing outside ROOT, both come out as their real destination and both fail is_relative_to(ROOT). Checking the string first and resolving later is the classic bypass: the string looks contained, the real path is not.

And never store secrets in memory. Memories are replayed verbatim into every future session, so a key written once leaks into every later context, including other users’ sessions, if your per-user key separation is wrong.

What to remember, what to forget

A store you only ever write to gets worse with age, so no memory design is finished until it can refuse a fact and delete one. Start with what earns a place at all:

PII is personally identifiable information, names, addresses, account numbers, anything identifying a specific human. It is in the “don’t save” column not because it is never useful but because storing it needs a retention and deletion policy first, and a store built without one is a compliance problem that ships silently.

Forgetting is a feature, not an omission. Every memory is a permanent context tax on every future run, and an unbounded store degrades quality by diluting attention across irrelevant facts (Why quality degrades in long contexts). Eviction is the general name for removing a stored item. Three mechanisms, not interchangeable:

1. TTL: delete a fact N days after it was last read, not last written. Measuring from the write is the bug: a preference stated once in January and used every session since would expire on its birthday while still correct. Measuring from the read turns the TTL into a usage signal. Good for facts with a natural shelf life (project.deadline), wrong for facts without one (user.timezone).

2. Relevance decay: multiply the retrieval score by a factor that falls with age.

adjusted_score  =  score × 0.5 ^ (age_days / half_life)

which halves a fact’s ranking score every half_life days: at half_life = 30, a 30-day-old fact keeps 50%, a 60-day-old 25%, a 90-day-old 12.5%. Nothing is deleted; old facts just stop winning a place in the top k. That makes decay right when you can’t tell whether a fact is stale or merely quiet, because it is reversible: one fresh mention restores it.

3. Size cap with LRU eviction: hard-bound the store at N facts per user. When full, evict the least recently used (LRU) fact, the one longest without being retrieved. This is the only mechanism that gives a cost guarantee, so production stores usually run it under one of the others.

And contradicting writes must replace, not accumulate: a store holding both “prefers metric” and “prefers imperial” is worse than one holding neither, because the model picks one arbitrarily and sounds confident. Detection is the hard half (Long-term memory has the three ways).

State in complex workflows

Facts are not the only thing worth keeping. A run long enough to fail partway through also has task state, open subtasks, artifacts written, failures, and none of it belongs in the message array.

flowchart LR
    subgraph State
        M["messages[]"]
        T["task queue"]
        A["artifacts<br/>file paths, not blobs"]
        B["budget ledger"]
        E["error history"]
    end
    State --> CP[(Atomic checkpoint<br/>after each step)]
    CP --> R[Resume / replay / debug]

    style CP fill:#7209b7,color:#fff

Each box other than messages[] holds something the transcript either cannot express or can only express by making the model re-derive it:

• task queue: the open subtasks and their statuses. It lives outside the messages because it must be queryable (“what is still PENDING?”) without a model call, and on resume you need it as data, not prose to re-parse.
• artifacts: the run’s outputs, held as file paths, not blobs (a blob being the raw bytes inlined into state; rule 3 is why the distinction matters).
• budget ledger: tokens and dollars spent per step. Deriving it from the transcript means summing usage across forty responses you may have compacted away; keeping it in state makes “stop at $2” a cheap if.
• error history: every failure and what was tried, kept separately because it is the first thing compaction drops and the last thing you can afford to lose.

The checkpoint node is where state gets written. A checkpoint is a snapshot of the whole state saved to disk; atomic means a reader only ever sees the complete old snapshot or the complete new one, never a half-written mixture. It buys three things: resume picks the run back up where it stopped, replay re-runs a recorded sequence to reproduce a failure without re-paying for the model calls, and debug reads the checkpoints as a time series (only possible if state is small enough to compare, which is rule 3).

Three rules:

1. Checkpoint after every step, not on a timer. A 40-minute agent that crashes at minute 39 must resume, not restart.

2. Write atomically: write to a temp file, then rename it over the destination. A crash mid-write otherwise leaves a torn checkpoint, worse than none because it fails at load time, after you have lost the run.

   import json, os, tempfile
   
   def save_atomic(state: dict, final: str) -> None:
       """rename() is atomic only WITHIN a filesystem — so the temp file must
       be created in the destination's directory, not /tmp."""
       d = os.path.dirname(final) or "."
       fd, tmp = tempfile.mkstemp(dir=d)
       with os.fdopen(fd, "w") as f:
           json.dump(state, f)
           f.flush()
           os.fsync(f.fileno())      # bytes on disk before the rename, not after
       os.replace(tmp, final)        # atomic: readers see old or new, never torn

   Two preconditions the one-liner hides. tmp and final must be on the same filesystem, because a rename across filesystems is really a copy, and a copy is not atomic, which is why the temp file goes in the destination’s own directory, not /tmp. And the fsync, which forces the OS to flush write buffers to physical disk, must happen before the rename, or a power loss can leave a correctly named file full of zeros.

3. Store artifacts by reference. State holds file paths and IDs; bytes stay on disk. State carrying blobs cannot be cheaply serialized, compared, or logged, and debug is exactly “compare two checkpoints,” which stops working the moment a 60k-token dump is inline.

And the subtle one: mark a task IN_PROGRESS before executing it. On resume, IN_PROGRESS means “unknown outcome”, verify it, never blindly re-execute, or a resume double-sends the email.

Multi-modal context

Everything priced so far has been text. Multi-modal means the context holds images, documents, or audio too. All of it is converted to tokens and billed on the same meter as prose. Images deserve the arithmetic, because their price changed with the current model generation and the obvious way to save money on them goes wrong once caching is on.

The Files API is a separate endpoint that stores an upload server-side and hands you a file_id, so a document referenced on twenty turns is transmitted once.

How an image becomes a token count

An image is not billed by area. The model cuts it into 28×28-pixel patches and charges one token per patch:

tokens  =  ceil(width / 28) × ceil(height / 28)

(ceil is “round up”; a partial patch still costs a whole token.) Each vision tier, the generation of the model’s image handling, caps two things: the long edge in pixels and the tokens one image may cost. An image over either cap is downscaled to fit both.

Two tiers are in circulation and they price 1080p differently. On the current tier (long edge capped at 2,576 px, tokens at 4,784) a 1080p screenshot (1920 × 1080) is under both caps and is not downscaled: 69 × 39 patches = 2,691 tokens. On the older standard tier (both caps at 1,568) the same screenshot is downscaled and costs about 1,560 tokens. So the common phrasing “~1.5k at 1080p, up to ~4.8k at max” mixes two generations: the first half is the standard tier, the second the current one.

The practical consequence: “capture at native 1080p, it gets downscaled anyway” is now wrong. On the current tier nothing is thrown away, and you pay about 1.7× what the same screenshot used to cost.

The resend arithmetic

An untrimmed 20-step run resends every image every turn, turn t carries t images, so the image count summed over turns is 1 + 2 + … + 20 = 210, or 210 × 2,691 ≈ 565,000 image tokens. Images are the one context type you cannot compress: a summarized screenshot is useless. So the only two levers are trim old ones and lower the resolution, and they are not equivalent once caching is on.

Trimming means a sliding window: keep the most recent few, drop the rest. Dropping the oldest surviving image every turn moves the point where this request stops matching the last one, so it rewrites the prefix every turn and every surviving image is re-processed at full rate. Lowering the resolution shrinks every image without touching the prefix.

Price the two levers against the same untrimmed baseline, with keep_last = 3 (keep the three most recent, replace the rest with a one-line placeholder):

The sign flip in the trim row is the result. Uncached, dropping old images removes tokens you paid full price for: a 3.7× win. Cached, those same tokens were already served at 0.1×, so removing them saves little, while the prefix rewrite forces every surviving image to be written to cache again at 1.25×. You give up a tenth-rate bill to buy a quarter-over-full-rate one, and it comes out a 1.5× loss. Resolution, by contrast, wins in both worlds, because it shrinks each image without moving the prefix.

So: resolution first, and reach for trimming only past ~60 steps, in batches, not per turn. Two honest caveats. Counting image tokens only makes the ratios as large as they can be; the system prompt, per-turn text, and output are the same in all cases, and adding a constant to both sides of a ratio pulls it toward 1. And the magnitudes move with run length (the uncached win grows with step count; the cached loss does not), the signs are the durable part, the numbers belong to this baseline. The computer-use agent case study is a second data point at a different scale, with the same flip: a 1.4× win uncached and a 1.9× loss cached.

Measure before optimizing, because the resolution-to-token curve is not linear: the patch formula rounds up on each axis, so shrinking an image by 10% may cost nothing until the new width or height crosses a 28-pixel boundary. The same count_tokens endpoint prices an image (b64 is the file read and base64-encoded, how bytes travel inside a JSON request):

n = client.messages.count_tokens(
    model="claude-opus-5",
    messages=[{"role": "user", "content": [
        {"type": "image",
         "source": {"type": "base64", "media_type": "image/png", "data": b64}},
    ]}],
).input_tokens

Run it across the resolutions you’re considering before picking one.

And the audio row deserves a mitigation, since every other gotcha here gets one. Transcription errors are silent because a transcript is plain text and text looks authoritative. Two things blunt that: keep the pointer alongside the transcript (offload the audio, store the timestamped span next to the text, and let the agent re-listen when a decision turns on an exact string, an amount, a name, a command), and carry the transcriber’s per-span confidence into the context, so a low-confidence span becomes a place to ask the user, not a place to guess.

Cheat sheet

Each row is a symptom you can observe from outside, the mechanism underneath, and the fix.

Conclusion

• Two problems wear the name “memory”: context engineering (curating one request, which dies with the process) and persistence (facts that outlive it). Keep them separate.
• The messages array only grows and is resent every turn, so cost is quadratic. A bigger window fixes neither the cost nor the U-shaped positional recall, so a budget is still a discipline you impose well before the limit.
• Order the request most-stable-first and set the cache breakpoint at the last unchanging block. Keep volatile content (timestamps, retrieved facts) after it.
• Of the four growth levers, offloading is the one to reach for first: it appends instead of rewriting the prefix, so it keeps the cache and shrinks per-turn growth by a large constant factor.
• A long-term store needs three parts to stay useful: writing durable facts, resolving contradictions by replacing not accumulating, and evicting. A write-only store decays.
• Once caching is on, trimming images can cost more than keeping them, because cached tokens are already cheap and the trim rewrites the prefix. Lower the resolution instead.

One line to remember: the array only grows and you pay for all of it every turn, so every memory decision is really one question, is this worth resending on every remaining turn, and if not, where does it live instead?

Further reading

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023), the empirical basis for U-shaped positional recall.
• Anthropic, Prompt caching, developer documentation.
• Anthropic, Building effective agents.
• LangGraph, Persistence and memory documentation.

Next: 05 — RAG for Agents.