Memory and Context Management

Every turn the agent decides what to send now and what to keep after the process exits; the messages array only grows and you pay for all of it every turn.

Two problems, one name

• Context engineering: curate one request (what goes in the window, in what order). Dies with the process.
• Memory: persistence, facts that survive after the process exits. A database row.
• Two arrows: write durable facts fires at turn end; retrieve relevant fires at turn start, keyed on the incoming message. Loop runs once per turn.
• Durable fact test: would I want to be told this again three days from now? (“I’m in Berlin” yes; “weather there” no.)

Cost model

• Token ≈ 4 chars English. Context window ≈ 200k tokens, a hard limit, not a budget; nothing bills differently at 20k vs 199k.
• History resent every turn → total input ≈ n·P + a·n²/2. P = fixed prefix (system + tools); a = tokens each turn adds. Second term is quadratic.
• Attention dilutes (weights sum to a fixed total); recall is U-shaped by position (best at start/end, worst middle). A bigger window just enlarges the middle.

Four memory types

• Semantic vs procedural: “prefers metric” is a fact; “ship with make ship” is a method.
• Semantic/procedural are event-driven (fire on what the user said), so most often skipped first. Working = short-term; other three = long-term.
• Reads aren’t symmetric with writes: semantic/procedural read every turn; episodic on demand; working never read (it is the request).

Prompt caching

• Prefill: model reads the whole request once, billed as input. Prompt caching reuses the prefill of a prefix (tokens 0..j, nothing skipped).
• Reads bill ≈ 0.1×; writes cost 1.25× once. Change one token at position j and everything from j on is invalid → order most-stable-first.
• Render order fixed by API: tools → system → messages.
• Breakpoint (cache_control on a block = “cache 0 through here”): put on the last block whose bytes never change. ephemeral = ~5-min TTL, refreshed per hit. Max 4 per request. Prefix under the model’s floor (512/1024/2048/4096, model-specific) silently doesn’t cache.
• Classic bug: datetime.now() in system — 8 volatile tokens near pos 30 invalidate everything after. Fix: system as a list of blocks with cache_control; put the timestamp in the user message (renders last, past the breakpoint).
• Prefix breakers: datetime.now()/uuid4() in system; json.dumps without sort_keys=True; iterating a set; per-user tools (position 0 → no sharing); switching models mid-conversation (caches are model-scoped); editing system mid-session.
• Cache entries are shared across every request sharing the prefix, not per-user.

Diagnose a dead cache (in order, step 3 is expensive)

1. Send the same request twice — request 1 always reports zero read.
2. On request 2 read both counters:

3. Serialize both bodies with sort_keys=True, diff; first differing byte is the bug.

Four growth levers (cheapest first)

• Trim: drop oldest turns. Cheap, loses detail. Rewrites prefix.
• Clear: prune stale tool payloads, keep structure. Rewrites prefix.
• Compact: replace a span with a summary. One extra call, lossy. Rewrites prefix.
• Offload: write payload to disk, leave a ~15-token pointer; needs a read tool. Appends, keeps cache.
• Only offload appends. It shrinks the number multiplied by turn count → constant-factor win (~200× cumulative, ~4000×/turn at the margin in the file-dump example).
• Compaction: server summarizes older turns past a threshold (~100k, not max_tokens) and returns a compaction block. Append the full resp.content, not .text — the block is state. betas=["compact-2026-01-12"] (header) vs compact_20260112 (strategy type); not interchangeable.
• Context editing: sibling beta that clears stale tool results / thinking blocks. Editing = irrelevant outputs; compaction = narrative matters. Both rewrite the prefix.

Long-term store: build and forget

• Two functions: on_turn_end extracts durable facts (a model call — only a model can judge worth-keeping), replaces contradictions or adds; on_turn_start searches and injects after the breakpoint, never prepend.
• Contradiction detection: same-key overwrite (free, misses paraphrases) → embedding similarity (catches paraphrases, false-positives on negation, one embedding/write) → LLM judge (semantic, slow, one call/write). Production: key-first, judge on collision.
• Anthropic SDK: model gets a memory tool (view/create/str_replace/delete); you implement the backend. resolve() before the containment check (collapses .. and symlinks). Errors must return is_error: True so the model sees a failure, not file contents. Never store secrets — memories replay verbatim into future sessions.
• Save: stable prefs, corrections and why, invisible constraints, confirmed approaches. Don’t: anything derivable from code/git, verbatim transcripts, secrets/PII without policy, “be more careful” (unfalsifiable).
• Eviction: TTL from last read not write; relevance decay score × 0.5^(age/half_life) (reversible, not deleted); size cap + LRU (only cost guarantee). Contradictions must replace, not accumulate.

Workflow state and multi-modal

• Keep outside messages[]: task queue (queryable), artifacts (file paths, not blobs), budget ledger, error history (first thing compaction drops).
• Checkpoint after every step (not a timer); write atomically (temp file in destination dir → fsync → os.replace, same filesystem); store artifacts by reference. Mark tasks IN_PROGRESS before executing; verify on resume, never blind re-execute.
• Images = ceil(w/28) × ceil(h/28) tokens, one per 28×28 patch. Current tier: 1080p = 2,691 tok (not downscaled, ~1.7× the old 1,560). Re-sent every turn.
• Once caching is on, trimming images can lose (1.5× loss): cached tokens are already ~0.1×, and the trim rewrites the prefix. Lower the resolution instead (no prefix touch). Use count_tokens before choosing; patch formula rounds up so shrinking may cost nothing until it crosses a 28px boundary.

stable tools → system │BREAKPOINT│ retrieved docs → history → current turn
  cache these once     │          │      volatile — full price, but no invalidation