Autonomous Agent

In this lesson, we’ll design an agent that works unattended for eight hours with no human in the loop, and produces a report you can trust in the morning.

An agent, here, is a program that loops. It sends text to a large language model (an LLM: a model that predicts the next chunk of text and can ask for tools to be run on its behalf), runs the tools the model asks for, feeds the results back, and then repeats.

The loop itself is straightforward. Everything that makes an unsupervised eight-hour run survivable sits outside the loop, in five mechanisms:

We’ll build each of those five, price roughly what a night costs (about $22), and mark when not to build this at all. By the end you’ll be able to decide whether a task is safe to run unattended and defend each mechanism when an interviewer pushes on it.

The problem, as input and output

Fix exactly what the agent receives and what it must produce. Every later decision follows from those two things.

What goes in is three things.

1. A goal, written in plain English.
2. A budget with two dimensions: money, and wall-clock time. Wall-clock time means real elapsed time on a clock on the wall: eight hours is eight hours whether the agent worked hard or sat waiting on a slow test suite.
3. A success predicate. This one decides whether the project is viable at all. A predicate is a function that returns true or false; a success predicate answers “is the goal met?” by running code, not by asking for an opinion.

What comes out is three things.

1. The work itself: a branch of code changes, files written, data cleaned.
2. A structured report: which of the seven possible endings the run reached, what it spent, every task it completed and verified, every task it gave up on and why, and how far it drifted from the goal over time.
3. An exit code (the small integer a program returns when it finishes), so a script that wakes up at 9am can tell success from failure without reading prose.

Here is all of that filled in for one overnight goal. The IN half is what you hand the harness at submit time; the OUT half is what the harness produces by morning.

IN   goal       "Reduce p95 latency on /api/search to under 200ms."
     budget     $28 hard cap, 8 hours of wall clock
     predicate  a Python function the HARNESS runs:  p95_ms("/api/search") < 200

OUT  status     one of GOAL_MET, BUDGET_EXHAUSTED, TIME_LIMIT, DRIFT_DETECTED,
                NO_PROGRESS, QUEUE_EMPTY_GOAL_UNMET, UNRECOVERABLE_ERROR
     exit code  0 success, 2 partial, 3 failure, 4 unrecoverable
     work       a git branch, one commit per verified task
     report     verified tasks, parked tasks with reasons, spend, drift history

Two terms in that block need definitions before the rest of the lesson.

p95 latency is the response time that 95% of requests come in under. The slowest 5% are worse than this. It measures the bad-but-not-freak case, not the average, which is why teams set targets on it instead of on the mean.

The harness is the ordinary, non-model program that owns the loop. It holds the goal, keeps the money ledger, writes the checkpoints, and runs the success predicate. When this lesson says “the harness does X, not the agent”, it means X is Python the model cannot reach through any tool. That distinction (harness code versus anything the model can influence) drives every design decision below.

In, on, and out of the loop

“No human in the loop” needs a precise definition.

Human in the loop means a person sits inside the agent’s control flow. The run pauses, a human approves or corrects, and only then does it continue.

No human in the loop means nobody is inside the control flow at all. Every mechanism here exists to replace one of the things that person would have done.

Human on the loop is a third, weaker thing, and the names are close enough that people mix them up. A person watches the run and can intervene, but the run never waits for them. The supervised hybrid in Alternatives is human-on-the-loop with a review queue, and it is the design to reach for whenever the constraint permits.

Why this is hard: nobody is watching for eight hours. With no human correcting it, small errors compound, and the agent drifts toward tasks that feel productive but no longer serve the goal.

Fully autonomous operation is usually the wrong choice. It is survivable only when three things are true: actions are reversible, success is machine-checkable, and there is a hard budget ceiling. If any is missing, build a supervised agent instead. The rest of this lesson designs the fully autonomous case anyway, because the design is what makes those three preconditions concrete.

The three preconditions, and what each one replaces

Each precondition substitutes for a specific thing a human supervisor would otherwise provide.

The middle row decides most cases. Machine-checkable means a program can answer “done?”: a test suite that goes green, a query that returns under 200ms, a checksum that matches. If the only available judge is a person’s taste, the agent is evaluating its own output, and a model asked “did you succeed?” is not consulting evidence; it is producing the text a successful assistant would produce.

If you cannot write the success predicate as code, the task is not a candidate for autonomy. That single test disqualifies most tasks people want to run overnight.

Architecture

One lap of the loop, walked end to end, gives every later mechanism (termination, drift, checkpointing, budgets) a place to attach.

Following the diagram top to bottom traces one lap: a task comes off the queue, runs, is verified, is checkpointed, and then passes three gates (drift, budget, goal) before the next lap starts. The two purple boxes mark the two points where state reaches disk.

flowchart TD
    G([Goal + budget]) --> INIT[Seed task queue]
    INIT --> Q[[Queue]]
    Q --> PICK[Pop highest-value task]
    PICK --> MARK[Mark IN_PROGRESS<br/>+ checkpoint]
    MARK --> EX[Execute]
    EX --> VER{Machine-checkable<br/>verification}
    VER -->|pass| CP[Checkpoint state]
    VER -->|fail| RETRY{Retries left?}
    RETRY -->|yes| EX
    RETRY -->|no| PARK[Park task + record why]
    PARK --> CP
    CP --> DRIFT{Drift check<br/>vs ORIGINAL goal}
    DRIFT -->|drifted| HALT([Halt + alert])
    DRIFT -->|ok| BUD{Budget}
    BUD -->|over| STOP([Halt + report])
    BUD -->|ok| DONE{Goal predicate<br/>run by the HARNESS}
    DONE -->|yes| FIN([Done + report])
    DONE -->|no, queue empty| FAIL([FAILURE + report])
    DONE -->|no, queue non-empty| REPLAN[Update queue] --> Q

    style DRIFT fill:#bc6c25,color:#fff
    style HALT fill:#9d0208,color:#fff
    style STOP fill:#bc6c25,color:#fff
    style FIN fill:#2d6a4f,color:#fff
    style FAIL fill:#9d0208,color:#fff
    style CP fill:#7209b7,color:#fff
    style MARK fill:#7209b7,color:#fff

One lap, box by box

The goal and budget arrive, and the harness turns the goal into a starting task queue: a plain list of concrete units of work, held by the harness and refilled as the run learns things.

Each lap pops the task with the highest value to the goal. Before executing anything, the Mark IN_PROGRESS + checkpoint box writes down which task is about to run, so a crash mid-task leaves a record of exactly what was in flight. The matching Checkpoint state box writes the result afterwards. Two disk writes, one before and one after.

The task then executes, and its result goes to a verification step that is machine-checkable, not a self-assessment: a test, a query, a file that either exists or does not. A failure is retried while retries remain. When they run out, park task + record why sets the task aside with a written reason instead of looping on it forever.

Whether the task passed or was parked, the state is checkpointed. Then two guards fire before the next lap:

• The drift check compares the queue against the original goal, and halts if the run has wandered.
• The budget check halts if the money is gone.

Only after both decline does the harness (never the model) run the goal predicate. Three outcomes:

• It passes: the run is done.
• It fails and the queue is empty: the run ran out of tasks before it met the goal. That is a failure, not a finish; the empty-queue trap below explains why.
• It fails with the queue non-empty: the replan step (Update queue) adds what was learned, and the loop takes another lap.

What makes this more than a ReAct loop

ReAct is reason and act, the standard pattern where a model alternates between thinking a step and calling a tool (the ReAct pattern). The inner loop above is exactly that. Five additions distinguish this design from a bare ReAct loop:

1. Verification after every task: machine-checkable, not the model’s own opinion.
2. A checkpoint before and after every task: the two purple boxes.
3. A drift check against the immutable original goal.
4. Dual budgets: one the model can see, one it cannot.
5. A termination set in which every exit is reported, including the failures.

Termination

Every way the run is allowed to end has a name. Treating those endings as a formal set, not as scattered if statements, is what makes the morning report honest.

The seven endings are the whole set:

STOP = { GOAL_MET, QUEUE_EMPTY_GOAL_UNMET, BUDGET_EXHAUSTED,
         TIME_LIMIT, DRIFT_DETECTED, NO_PROGRESS, UNRECOVERABLE_ERROR }

The set needs two properties. Each is a constraint on how the loop is written, not a comment above it.

Property 1: the set is exhaustive

Exhaustive means every path out of the loop matches exactly one member of that set. No exit that is not on the list.

Writing the loop as while True with a single unconditional stop check at the top gets you most of the way. No break buried in a branch, no loop condition that can quietly end the run.

It does not get you all the way, and this is the part usually missed: an exception raised inside the loop body is also a path out. On an eight-hour run it is the likeliest one, because the loop body is where the tools and the model calls live. while True alone leaves the agent one uncaught BudgetExceeded away from an ending that has an exit code in the table and a traceback in reality. Exhaustive means the exception path is classified too, which is what the try in the code below is for.

Property 2: the set is prioritized, and therefore disjoint

Several members can be true at the same instant. A run can exhaust its budget on the same lap that it meets its goal. So the order of the checks decides which one gets reported.

GOAL_MET is checked first, so a run that reaches the goal on its last dollar reports success, not BUDGET_EXHAUSTED. Checking in a different order would report a different ending from the same facts, which is why the order is code, not convention.

That order is exactly what the next diagram shows: six checks in sequence, each one falling through to the next.

flowchart TD
    T([Top of loop]) --> C1{goal predicate<br/>passes?}
    C1 -->|yes| S1["GOAL_MET<br/>SUCCESS · exit 0"]
    C1 -->|no| C2{ledger refused, or<br/>spend >= hard cap?}
    C2 -->|yes| S2["BUDGET_EXHAUSTED<br/>PARTIAL · exit 2 · alert"]
    C2 -->|no| C3{elapsed > wall clock?}
    C3 -->|yes| S3["TIME_LIMIT<br/>PARTIAL · exit 2 · alert"]
    C3 -->|no| C4{drift > 0.5?}
    C4 -->|yes| S4["DRIFT_DETECTED<br/>FAILURE · exit 3 · page"]
    C4 -->|no| C5{5 tasks, no change?}
    C5 -->|yes| S5["NO_PROGRESS<br/>FAILURE · exit 3 · alert"]
    C5 -->|no| C6{queue empty?}
    C6 -->|yes| S6["QUEUE_EMPTY_GOAL_UNMET<br/>FAILURE · exit 3 · alert"]
    C6 -->|no| RUN[Run next task]

    style S1 fill:#2d6a4f,color:#fff
    style S2 fill:#bc6c25,color:#fff
    style S3 fill:#bc6c25,color:#fff
    style S4 fill:#9d0208,color:#fff
    style S5 fill:#9d0208,color:#fff
    style S6 fill:#9d0208,color:#fff

Each ending carries three attachments, aimed at three different audiences:

• A class (SUCCESS, PARTIAL or FAILURE) which says how bad it is, for a human skimming the report.
• An exit code, for whatever script reads the outcome.
• An alerting action. Alert means a message to a channel someone reads in the morning. Page means waking someone up now. Only drift gets a page, because drift is the one ending where the run was actively spending money on the wrong problem.

The table is the same seven endings with those attachments, plus the one thing each report must contain.

Some shorthand in that table:

• The ledger is the harness’s running tally of money spent, built in Budget ceilings.
• The spend curve is that tally plotted over time. It tells you whether the money went on steady work or on one runaway task.
• The diff is the complete set of changes the run made, as a line-by-line comparison against the state it started from.
• K, in the NO_PROGRESS row, is the streak length: how many consecutive tasks may change nothing verifiable before the run is declared stuck. It is 5 in the code below.

One row has no if anywhere in should_stop, and that is deliberate: UNRECOVERABLE_ERROR is the exception ending. Nothing returns it. It is produced by the except clause in run below, which catches what Goal.assert_intact and both refusal branches of Checkpoint.load raise. Write the loop without that clause and the row advertises an exit code that no code path can reach, which is what the module-level set-equality check exists to make impossible.

The loop in code

The priority order from the diagram is literally the body of should_stop, read top to bottom. Two names to notice: state.budget.blocked is a flag the ledger sets when it refuses a call (the Budget ceilings section explains why the flag has to exist), and state.harness_goal_predicate is the success test, held by the harness, with no tool pointing at it.

from typing import Optional

STOP = frozenset({"GOAL_MET", "QUEUE_EMPTY_GOAL_UNMET", "BUDGET_EXHAUSTED",
                  "TIME_LIMIT", "DRIFT_DETECTED", "NO_PROGRESS",
                  "UNRECOVERABLE_ERROR"})

DRIFT_EVERY = 5                                 # tasks between drift checks

class BudgetExceeded(Exception):
    """Raised by Ledger.reserve BEFORE a call that would cross the hard cap."""

def should_stop(state) -> tuple[bool, str]:
    """Exhaustive and prioritized. Order is load-bearing."""
    if state.harness_goal_predicate():          # the HARNESS runs it, not the agent
        return True, "GOAL_MET"
    if state.budget.blocked or state.budget.spent >= state.budget.hard_cap:
        return True, "BUDGET_EXHAUSTED"         # `blocked`: reserve() already refused
    if state.elapsed > state.wall_clock_limit:
        return True, "TIME_LIMIT"
    if state.drift_score > 0.5:
        return True, "DRIFT_DETECTED"
    if state.no_progress_streak >= 5:
        return True, "NO_PROGRESS"
    if not state.queue:
        return True, "QUEUE_EMPTY_GOAL_UNMET"   # NOT success — report it as such
    return False, ""

STOP_CLASS = {
    "GOAL_MET": "SUCCESS",
    "BUDGET_EXHAUSTED": "PARTIAL",
    "TIME_LIMIT": "PARTIAL",
    "DRIFT_DETECTED": "FAILURE",
    "NO_PROGRESS": "FAILURE",
    "QUEUE_EMPTY_GOAL_UNMET": "FAILURE",
    "UNRECOVERABLE_ERROR": "FAILURE",
}

# At import, and in BOTH directions. A reason with no class and a class with no
# producer are different bugs, and only the second one catches a stop reason
# that is advertised in the table and returned by nothing.
assert STOP == set(STOP_CLASS), STOP ^ set(STOP_CLASS)

def run(state, drift_check=None):
    while True:
        try:
            stop, reason = should_stop(state)
            if stop:
                assert reason in STOP_CLASS, f"unclassified stop: {reason}"
                return report(state, reason, STOP_CLASS[reason])
            execute_next(state)
            if drift_check and state.step % DRIFT_EVERY == 0:
                drift_gate(state, drift_check(state.goal, state.queue))
        except BudgetExceeded:              # reserve() refused the NEXT call. Do not
            state.budget.blocked = True     # classify it here: let the top of the
            continue                        # loop report BUDGET_EXHAUSTED, exit 2
        except AssertionError:
            raise                           # the tripwire above stays loud
        except Exception as e:              # EVERY other way out, and classified
            return report(state, "UNRECOVERABLE_ERROR",
                          STOP_CLASS["UNRECOVERABLE_ERROR"], detail=repr(e))

Three things in that listing are easy to under-read:

• The try is what makes the set exhaustive. Without it, while True is not the only exit: any exception out of execute_next is a second one, and it is the common one, because execute_next is where the tools and the model calls are. UNRECOVERABLE_ERROR has no other producer, so this except is the only thing that turns a raised RuntimeError into a classified, reported ending instead of a traceback in a log nobody is reading at 3am.

• BudgetExceeded gets its own branch above the catch-all, because it is not unrecoverable. The ledger refused the next call before making it, so nothing is broken: the run is a PARTIAL with a full report, not a crash. That branch sets blocked and continues, letting the top of the loop classify the stop in the one place stops get classified.

• The module-level assert STOP == set(STOP_CLASS) is the check that does the work, not the one inside run. The inner assert only fires on the branch the run happens to take, at 3am, in production. The set equality fires at import, on every branch at once, in both directions, which is how you notice a class that has an exit code, a report contract and no code path that returns it.

Exercised across all six paths, the loop reports GOAL_MET when the predicate passes, QUEUE_EMPTY_GOAL_UNMET when the queue drains unmet, BUDGET_EXHAUSTED when reserve() refuses the next call (spend stops at $27.75 of a $28.00 cap and never reaches it, which is why the blocked flag exists), and UNRECOVERABLE_ERROR for any other exception out of the loop body.

The empty-queue trap

The most common trap here is that the usual loop shape cannot report its most common failure:

def run_naive(state):
    while state.queue:             # <-- the bug is here, not in the reporting
        execute_next(state)
    return "done"

The trouble is that while queue: makes “empty queue, goal unmet” the normal exit. There is no branch to report it from, because the condition that should be a failure is the loop’s own termination condition. The bug is structural: you cannot report a state your control flow treats as success.

The cost shows up in a trace of the last two minutes of a run. Everything from 03:52:41 onward happens in the same second.

03:41:12  task t_33  "add index on search_events(created_at)"     PASS
03:52:40  task t_34  "verify p95 < 200ms on /api/search"          FAIL  (p95 = 410ms)
03:52:41  replan: no further optimizations identified
03:52:41  park t_34  reason="verification failed, no remaining ideas"
03:52:41  queue length = 0
03:52:41  loop exits
03:52:41  run.status = "completed"
03:52:41  report: "Completed 33 of 34 tasks. See diff."

morning:  p95 is 410ms, unchanged from 24 hours ago.
          The one task that measured the actual goal is the one that failed,
          and it is a footnote in a report headed "completed".

Thirty-three tasks passed and one failed, and the one that failed is the only one that measured the goal. The loop drained its queue, so it exited the way it always exits, and the status field says completed because that is the only thing a drained queue can mean in this shape.

An empty queue with an unmet goal is a failure, and an agent that reports it as success is worse than one that crashes, because a crash gets investigated and a false success does not.

Two further rules follow from the same idea. Both keep the verdict away from the thing being judged.

1. The harness runs the goal predicate, never the agent. state.harness_goal_predicate is a Python callable defined at submit time, and the agent has no tool that can influence it. If the agent could declare success, the entire termination set collapses to whatever it feels like saying.
2. Parked tasks are part of the report, not a log line. The most informative artifact of a failed run is the list of tasks it gave up on, each with its reason. That list is what tells you in the morning whether to retry, re-scope, or abandon.

Drift detection

A characteristic failure of autonomous agents is that the run slowly stops working on the thing you asked for, and only one kind of check catches it.

Drift is what happens when each self-generated task looks locally reasonable but the chain of them does not. Twenty steps in, the agent is optimizing something nobody asked for, having never taken a step you could point at as wrong.

The chain below is six tasks the agent generated for itself, left to right, starting from a goal of reducing p95 latency. It shifts from on-goal (✓) to off-goal (✗), and no single arrow is obviously wrong.

flowchart LR
    G["Original goal:<br/>reduce p95 latency"] --> T1["Profile endpoints ✓"]
    T1 --> T2["Add caching ✓"]
    T2 --> T3["Refactor cache layer ~"]
    T3 --> T4["Add cache metrics ~"]
    T4 --> T5["Build a metrics dashboard ✗"]
    T5 --> T6["Add dashboard auth ✗✗"]

    style T1 fill:#2d6a4f,color:#fff
    style T2 fill:#2d6a4f,color:#fff
    style T3 fill:#bc6c25,color:#fff
    style T4 fill:#bc6c25,color:#fff
    style T5 fill:#9d0208,color:#fff
    style T6 fill:#9d0208,color:#fff

Follow the chain one link at a time and try to name the wrong step:

• Profile endpoints: exactly right. You cannot reduce latency you have not measured.
• Add caching: follows directly from what the profile showed.
• Refactor the cache layer: defensible cleanup of the thing just added.
• Add cache metrics: how you would confirm the cache is actually working.
• Build a metrics dashboard: a natural way to look at the metrics just added.
• Add dashboard auth: a login gate, which is what you do before anyone else opens a dashboard.

Every arrow is defensible, but the chain is not. Adding a login page to a dashboard has nothing to do with p95 latency, and the run got there in six locally sensible moves.

No local check catches this, because drift is a property of the path, not of any step on it. A checker that looks at one task and asks “is this reasonable?” says yes to all six. That is why the comparison has to be against the origin, and why the origin has to be beyond the agent’s reach.

Why the goal must be immutable

If the goal is stored somewhere the agent can edit, the detector stops working. It does not fail loudly; it keeps reporting healthy numbers while the run goes nowhere.

The trace below is that failure in four timestamps. t is the task number. drift_check(goal, queue) returns a score from 0.0 (fully aligned) to 1.0 (entirely unrelated), and anything under 0.3 is considered fine. Watch what the goal text says at t=0 versus t=17.

t=0    goal.txt  "Reduce p95 latency on /api/search to under 200ms."
       drift_check(read("goal.txt"), queue) -> 0.02   aligned

t=17   agent task t_17: "Update goal.txt to reflect refined understanding"
       goal.txt  "Improve observability of the search subsystem."
       (locally reasonable! it just spent four tasks on metrics)

t=20   queue: [build metrics dashboard, add dashboard auth, add SSO to dashboard]
       drift_check(read("goal.txt"), queue) -> 0.05   "fully aligned"

t=40   p95 on /api/search: 410ms, unchanged.
       drift_score never exceeded 0.08 for the entire run.
       Every alert threshold was respected. Nothing fired.

The goal here lives in a file the agent’s tools can write. At step 17 the agent edits it, and that edit is itself a locally reasonable task given the four metrics tasks it had just finished. From then on the detector compares the queue against the new goal, which the queue matches perfectly. (SSO in that queue is single sign-on, the corporate login flow, three steps removed from anything about latency.)

Drift detection compared the queue against a goal that had already drifted, and correctly reported 0.0 the whole way down. The detector was not broken. It was pointed at a moving target.

Three defenses, and which one actually holds

All three defenses are structural, not instructional. None ask the model to behave. They build on a Goal object with no setter, hashed at construction so a change to the text becomes detectable. A hash here is a SHA-256 digest: a short fixed-length fingerprint of some text, where different text almost certainly produces a different fingerprint. Store the fingerprint at construction, recompute it later, compare: if they differ, the text changed.

import hashlib

class Goal:
    """Loaded once at submit time. There is no setter."""
    def __init__(self, text: str):
        self._text = text
        self._sha = hashlib.sha256(text.encode()).hexdigest()

    @property
    def text(self) -> str:
        return self._text

    def assert_intact(self) -> None:
        if hashlib.sha256(self._text.encode()).hexdigest() != self._sha:
            raise RuntimeError("UNRECOVERABLE_ERROR: goal text mutated")

1. The goal lives in the harness, not on disk. If it is a file, it is inside a tool’s reach; if it is inside a tool’s reach, a locally reasonable task will eventually edit it. Load it once from the submission record. This is the defense that actually works: a goal held in the harness is unreachable by any tool the agent has, and unreachability is a property of the tool allowlist, not of a check.
2. Hash it and assert on every drift check. Cheap, and it converts one class of silent corruption (an in-process write to _text that forgets _sha, from a bad merge or a buggy library) into UNRECOVERABLE_ERROR. It is not a defense against the agent: no tool call can reach a private attribute, and anything that can reach one can set both fields. It is a cheap secondary check behind the real lock, which is that no tool exists to reach the goal at all.
3. The drift check runs in a fresh context. Fresh context means the judging model call starts from an empty conversation: it receives only (original_goal, queue), never the agent’s reasoning, never the transcript. This is the same idea as the evaluator in evaluator-optimizer. A judge holding the agent’s justifications in context is predicting the continuation of a conversation in which those justifications were persuasive, and it will rationalize the chain it just read.

The honest limit of the hash: it catches an in-process write that updates _text but forgets _sha, and nothing else. Two lines (set _text and recompute _sha) defeat it, and it never opens the goal file at all. That is exactly why defense 1, not the hash, is the one that holds.

The check itself

The check is one model call with a constrained output shape. Structured output means the model is forced to emit fields matching a declared schema, not free text, so the score comes back as a float you can compare against a threshold instead of a paragraph you would have to parse.

DriftCheck is that schema, and check_drift is the call. The system prompt defines “off-goal” for the judge and tells it explicitly that it has not been given the agent’s reasoning, so it cannot assume that reasoning was sound.

from pydantic import BaseModel, Field

class DriftCheck(BaseModel):
    reasoning: str                              # first field, on purpose
    off_goal_task_ids: list[str]
    drift_score: float = Field(ge=0, le=1)

def check_drift(goal: Goal, queue: list) -> DriftCheck:
    goal.assert_intact()
    r = client.messages.parse(                  # FRESH context: no agent history
        model="claude-opus-5", max_tokens=2048,
        system=("Score how far the task queue has drifted from the ORIGINAL goal. "
                "A task is off-goal if completing it would not measurably advance "
                "the original goal, even if it seems useful. "
                "Judge each task against the goal only. You have not been given "
                "the agent's reasoning and must not assume it was sound. "
                "0.0 = fully aligned, 1.0 = entirely unrelated."),
        messages=[{"role": "user", "content":
                   f"<original_goal>{goal.text}</original_goal>\n"
                   f"<queue>{fmt(queue)}</queue>"}],
        output_format=DriftCheck,
    )
    return r.parsed_output

reasoning is the first field on purpose. Constrained decoding (the mechanism that forces output to match the schema) emits fields in schema order (structured output is a guarantee). Put drift_score first and you force the model to commit to a number before it has written a word of analysis. Put reasoning first and the number is conditioned on the analysis.

Cadence and thresholds

Run the check every 5 tasks, not every task. It is a full Opus call (the most expensive model tier) and drift is a slow-moving property that will not appear and vanish inside five tasks. DRIFT_EVERY = 5 in the termination code is that cadence.

Two threshold bands do different things:

check_drift returns a score. drift_gate is the harness half that acts on it, and it is the only thing in the whole design that ever assigns state.drift_score: the value should_stop reads at the top of the next lap. The tasks it sheds go onto state.pruned, and they belong in the morning report next to the parked ones: a task the harness removed for drifting is at least as informative as one the agent gave up on.

PRUNE_AT, HALT_AT = 0.3, 0.5

def drift_gate(state, dc: DriftCheck) -> None:
    """The harness half of the check: record the score, then act on the band."""
    state.drift_score = dc.drift_score           # nothing else assigns this
    if dc.drift_score > HALT_AT:
        return                                   # should_stop halts on the next lap
    if dc.drift_score > PRUNE_AT:                # prune band: shed and continue
        off = set(dc.off_goal_task_ids)
        state.pruned += [t for t in state.queue if t.id in off]
        state.queue = [t for t in state.queue if t.id not in off]

Both thresholds use a strict >, so a score of exactly 0.5 prunes instead of halting, and exactly 0.3 does neither. A detector calibrated to “0.5” and a loop that fires at 0.5 are two different systems, so assert which one you shipped.

The detector has its own failure mode. It has a false-negative bias (it misses real drift more often than it invents fake drift) because a plausible task chain reads as reasonable. Calibrate it against 20 hand-labeled queues (10 aligned, 10 drifted at known severity), where calibrate means running the detector on inputs whose right answers you already know and measuring how often it agrees. Report that agreement rate before trusting a threshold. If it cannot separate the labeled sets, lower the threshold instead of shipping a detector you have not measured.

Checkpointing

Surviving interruption means saving the run’s state so a crash costs minutes, not the whole night. Less obviously, it means ensuring a half-written save file can never pass for a whole one, which is more dangerous than having no save at all.

An 8-hour run will be interrupted. A host reboot, a spot-instance reclaim (a cheap cloud machine taken back at short notice), an out-of-memory kill, a deploy. Design for resume from step one.

A checkpoint is a single file holding everything needed to pick the run back up: the queue, what finished, what was parked, what was in flight, and how much has been spent.

The two rows below are alternatives, not a sequence. The top row is the safe write: four steps ending in a committed checkpoint. The bottom row is the naive one-line write, which ends in a resume that is wrong without knowing it.

flowchart TD
    S[State in memory] --> W1["1. write state.json.tmp<br/>same filesystem"]
    W1 --> W2["2. f.flush() + os.fsync(fd)<br/>bytes are on the device"]
    W2 --> W3["3. os.replace(tmp, final)<br/>ATOMIC rename"]
    W3 --> W4["4. fsync the directory<br/>rename itself is durable"]
    W4 --> OK([Checkpoint committed])

    B1["Naive: open(final,'w').write(...)"] --> B2["crash at byte 184,301"]
    B2 --> B3["file exists, is truncated,<br/>and may still parse"]
    B3 --> BAD([Confident wrong resume])

    style W3 fill:#2d6a4f,color:#fff
    style OK fill:#2d6a4f,color:#fff
    style BAD fill:#9d0208,color:#fff

The top row, step by step:

1. Write the new state to a temporary file on the same filesystem as the real one.
2. Flush, then fsync. Flush pushes the program’s buffered bytes down to the operating system. fsync forces the operating system to push them onto the physical device. Both are needed: a flush alone leaves the bytes in the OS cache, where a power cut loses them.
3. Rename the temp file over the real one with os.replace. This is atomic, meaning any reader sees either the entire old file or the entire new one, never a mixture. This is the step that buys the whole guarantee.
4. fsync the directory, so the rename itself is durable and not just the bytes.

Only after step 4 is the checkpoint committed. The bottom row (open(final, 'w').write(...)) leaves, on a crash partway, a file that exists, is truncated, and may still parse as valid JSON. That is the dangerous outcome.

Why a torn checkpoint is worse than no checkpoint

A torn write is one that stopped partway, leaving a file that is neither the old state nor the new one. There are two ways a non-atomic write dies, and only one is survivable.

--- case A: it fails loudly (the lucky case) ---
$ python -c "import json; json.load(open('state/checkpoint.json'))"
json.decoder.JSONDecodeError: Unterminated string starting at:
  line 2841 column 18 (char 184301)

resume: refuses to start. You lose 8 hours. You know you lost 8 hours.

--- case B: it parses (the case that costs money) ---
the write was cut between records and the tail happened to close cleanly:

{"step": 34,
 "goal_sha": "9f2c...",
 "completed": [ ...22 entries... ],     <-- 12 entries lost
 "queue": [],                            <-- flushed before the queue was written
 "spent_usd": 11.40}

In case A the crash landed inside a string. The file no longer parses, so every attempt to resume fails immediately and loudly. You lost eight hours and you know it.

In case B the crash landed between records and the remaining braces closed cleanly. The file parses. Twelve of the 34 completed tasks are missing and the queue is empty, and nothing about the file says so. Resume proceeds, the goal predicate fails, and the 12 invisible completed tasks may be redone: anything non-idempotent runs twice. Idempotent means running an operation twice has the same effect as running it once; anything non-idempotent (a payment, a message, an append) does damage on the second run.

No checkpoint fails fast; a torn checkpoint that parses produces a confident wrong resume. That asymmetry is the argument for atomicity.

The implementation

Checkpoint.save is the four-step write from the diagram. Checkpoint.load is its inverse plus two refusals. Three things to watch for:

• The payload is framed: the state is serialized to a string, then wrapped in an outer object carrying a SHA-256 of that string. load rehashes and compares.
• Every refusal raises RuntimeError with UNRECOVERABLE_ERROR in the message, so a bad checkpoint arrives at the loop as one of the seven endings, not as a stray JSONDecodeError.
• artifacts stores paths, not contents, and history_summary stores a compacted summary, not the raw transcript (rule 4 below explains why).

import hashlib, json, os, pathlib, tempfile

SCHEMA_VERSION = 3

class Checkpoint:
    def __init__(self, path: str):
        self.path = pathlib.Path(path)
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def save(self, state) -> None:
        payload = {
            "schema_version": SCHEMA_VERSION,
            "goal": state.goal.text,               # original, never rewritten
            "goal_sha": state.goal._sha,
            "queue": [t.model_dump() for t in state.queue],
            "in_progress": state.in_progress.model_dump() if state.in_progress else None,
            "completed": [t.model_dump() for t in state.completed],
            "parked": [t.model_dump() for t in state.parked],
            "spent_usd": state.budget.spent,
            "step": state.step,
            "artifacts": state.artifact_paths,     # paths, not blobs
            "history_summary": state.summary,      # COMPACTED: the run's history
                                                   # summarized down by a model
                                                   # call, not the raw transcript
        }
        body = json.dumps(payload, indent=2, sort_keys=True)
        framed = json.dumps({"sha256": hashlib.sha256(body.encode()).hexdigest(),
                             "body": body})

        # temp file on the SAME filesystem, or the rename is not atomic
        fd, tmp = tempfile.mkstemp(dir=self.path.parent, suffix=".tmp")
        try:
            with os.fdopen(fd, "w") as f:
                f.write(framed)
                f.flush()
                os.fsync(f.fileno())               # bytes on the device
            os.replace(tmp, self.path)             # atomic rename
            dirfd = os.open(self.path.parent, os.O_DIRECTORY)
            try:
                os.fsync(dirfd)                    # the rename itself is durable
            finally:
                os.close(dirfd)
        except BaseException:
            pathlib.Path(tmp).unlink(missing_ok=True)
            raise

    def load(self) -> Optional[dict]:
        if not self.path.exists():
            return None
        try:                                            # a TORN file dies here,
            framed = json.loads(self.path.read_text())  # not at the checksum:
            body = framed["body"]                       # there is no body to hash
            body_sha = hashlib.sha256(body.encode()).hexdigest()
        except (ValueError, KeyError, TypeError, AttributeError) as e:
            raise RuntimeError(
                f"UNRECOVERABLE_ERROR: checkpoint unreadable: {e!r}") from e
        if body_sha != framed["sha256"]:
            raise RuntimeError("UNRECOVERABLE_ERROR: checkpoint checksum mismatch")
        payload = json.loads(body)
        if payload["schema_version"] != SCHEMA_VERSION:
            raise RuntimeError("UNRECOVERABLE_ERROR: checkpoint schema mismatch")
        return payload

Truncated at 200 random byte positions, every version is refused as UNRECOVERABLE_ERROR, each dying in json.loads before the checksum is ever compared, because a torn frame has no body to hash. The one thing neither the rename nor the checksum catches is a body that was already wrong before it was serialized: that frames, hashes, and reloads without complaint: case B, exactly.

Five rules do the work in that code, each ruling out a specific way the resume goes wrong.

1. Temp file on the same filesystem. os.replace across filesystems is a copy, not a rename, and a copy is not atomic. /tmp is usually a different mount, which is exactly why the temp file is created next to the checkpoint.
2. fsync the file, then fsync the directory. Without the second, the rename can be lost in a power failure even though the data was written.
3. Checksum the body, and be exact about which half does what. os.replace is what defeats case B, not the checksum: an atomic rename means a torn frame never reaches disk under the real name at all. The checksum adds everything the rename cannot cover: bit rot, a bad disk, a frame corrupted after commit. Neither covers a body that was already wrong when serialized. The refusal is worth having regardless, and it must arrive as UNRECOVERABLE_ERROR, not a raw JSONDecodeError.
4. Artifacts by path, never by content. An artifact is any file the run produced. A checkpoint that embeds file contents grows with the work and turns resume into a large prefill: a large block of text sent into the model’s context on the first call, paid for by the token.
5. Checkpoint after every task, and again before starting one, not on a timer. The checkpoint is a few kilobytes; a timer just chooses how much work you are willing to lose.

IN_PROGRESS: resume verifies, it does not re-execute

The dangerous window is a task that was mid-execution when the process died. Its side effects are half-applied and the checkpoint cannot know how far it got. That is what the IN_PROGRESS marker is for, and the rule is counter-intuitive: on resume you check whether the task already happened, and you never simply run it again.

03:11:02  t_27  status=IN_PROGRESS  "apply index migration 0042"
03:11:02  checkpoint saved (in_progress=t_27)
03:11:19  psql: CREATE INDEX CONCURRENTLY idx_search_ts ON search_events(created_at);
03:11:41  <SIGKILL — spot instance reclaimed>

--- resume at 07:02:10 ---
load()  -> in_progress = t_27

  naive (re-execute):
    psql: CREATE INDEX CONCURRENTLY idx_search_ts ...
    ERROR:  relation "idx_search_ts" already exists
    -> agent sees an error, "fixes" it by dropping and recreating,
       burns 40 minutes rebuilding an index that was already valid

  naive on a non-idempotent task class:
    t_27 = "post the summary to the #eng channel"
    -> posted twice. No error. Nobody notices it was the agent's fault.

  correct (verify, do not execute):
    t_27.verify()  ->  SELECT indisvalid FROM pg_index WHERE ... -> True
    mark t_27 DONE, continue at t_28

The task is a database migration (a schema change applied by running SQL) and SIGKILL is the signal that terminates a process instantly, with no chance to clean up. The index was in fact built before the kill, so re-executing produces an error that the agent then “fixes” destructively. The second case is worse, because posting a message twice raises no error at all.

Every task carries a cheap, idempotent verify(). On resume, an IN_PROGRESS task is verified, never re-executed. A task that cannot be verified is not eligible for autonomous execution.

resume is the first thing the harness calls after loading a checkpoint. It reads in_progress out of the checkpoint dict, runs that task’s verify(), and routes on the answer. There is no execute call anywhere in it: that absence is the entire point.

def resume(state, ckpt: dict):
    t = ckpt.get("in_progress")
    if t is None:
        return state
    task = Task(**t)
    outcome = task.verify()                  # cheap, idempotent, read-only
    if outcome.done:
        state.completed.append(task)
    elif outcome.partially_applied:
        state.parked.append(task.with_reason(
            f"resumed into a partially applied state: {outcome.detail}"))
    else:
        state.queue.insert(0, task)          # nothing happened; safe to re-run
    state.in_progress = None
    return state

The three-way outcome is where most implementations are only two-way, and the missing third is the expensive one:

• done means the work landed. Mark it complete and move on.
• Nothing happened means the task never took effect, so it is safe to put back at the front of the queue.
• partially_applied is the case implementations forget, and the one that must go to a human. A half-applied migration is not something an agent should improvise against unsupervised.

Budget ceilings

One spending limit is not enough. The two used here fail in opposite directions, so each covers the other.

The budget ledger is the harness’s running total of money spent. It is consulted before every model call, and what it does depends on how much of the cap is gone.

flowchart LR
    B[Budget ledger] --> W1["< 70%<br/>run freely"]
    B --> W2["70-90%<br/>tell the agent to prioritize"]
    B --> W3["90-100%<br/>finish current task only"]
    B --> W4["> 100%<br/>halt + report partial"]

    style W2 fill:#bc6c25,color:#fff
    style W3 fill:#bc6c25,color:#fff
    style W4 fill:#9d0208,color:#fff

With a $28 cap, those bands are concrete dollar figures: run freely below $19.60 (70%); inject a notice to prioritize between $19.60 and $25.20 (70–90%); finish the current task only between $25.20 and $28.00 (90–100%); halt and report partial past $28.00.

Two ceilings, and you need both

The difference is enforcement versus pacing. The hard cap is a control: plain harness code, invisible to the model, refusing the API call before it is made. The model has no say in it. The task budget is a request: a number placed in the model’s own configuration so it can pace itself and wrap up gracefully. A model stuck in a tool loop sails straight past it, because nothing enforces it. Neither alone is enough, so run both.

The ledger, in code

Ledger has two methods that run at different times. reserve runs before a call and can refuse it. record runs after a call and books what it actually cost. PRICES[model] returns an (input_price, output_price) pair in dollars per million tokens.

class Ledger:
    def __init__(self, hard_cap_usd: float):
        self.hard_cap = hard_cap_usd
        self.spent = 0.0
        self.blocked = False               # read by should_stop; see below

    def reserve(self, model: str, est_in: int, max_out: int) -> None:
        """Pre-flight. Check the WORST case before the call, not the actual after."""
        pin, pout = PRICES[model]                       # $/MTok
        worst = (est_in * pin + max_out * pout) / 1e6
        if self.spent + worst > self.hard_cap:
            self.blocked = True            # the ONLY thing that makes
            raise BudgetExceeded(          # BUDGET_EXHAUSTED reachable
                f"would reach ${self.spent + worst:.2f} of ${self.hard_cap:.2f}")

    def record(self, usage, model: str) -> None:
        pin, pout = PRICES[model]
        self.spent += (usage.input_tokens * pin
                       + usage.cache_read_input_tokens * pin * 0.1
                       + usage.cache_creation_input_tokens * pin * 1.25
                       + usage.output_tokens * pout) / 1e6

Prices are in $/MTok, dollars per million tokens. A token is the few-character chunk models read and write text in (tokens). Dividing by 1e6 converts a token count times a per-million price back into dollars.

Notice that reserve checks the worst case before the call, not the actual after it. The worst case is the whole prompt plus the maximum output the call is allowed to produce, because max_tokens is the only bound you have on output before the call happens. On Opus 5 at $25/MTok output, a single call with max_tokens=64000 is $1.60 of output exposure before you count a token of input. Checking after the fact makes the ceiling an observation, not a control: you find out you blew it once you already have.

Why spent >= hard_cap is not enough on its own

The pre-flight has a consequence people miss. reserve refuses the call that would cross the cap, so the run stops with money still on the ledger, and spent never reaches hard_cap. In the earlier run, spend ends at $27.75 of a $28.00 cap: the next call would have taken it to $28.75, so it was refused.

Read literally, then, spent >= hard_cap inside should_stop is a condition that never fires. BUDGET_EXHAUSTED becomes an ending nothing produces: a row in the table with an exit code and no code path. The fix is one flag, wired across three places:

1. reserve sets blocked = True before it raises.
2. run catches BudgetExceeded, sets nothing else, and continues.
3. should_stop reads blocked at the top of the next lap and reports BUDGET_EXHAUSTED: PARTIAL, exit 2, with the report the table promises instead of a traceback.

An alternative is to make the stop test spent + typical_call_cost >= hard_cap, which works but puts an estimate in the control path. The flag puts the decision in exactly one place.

Count all four usage fields

Prompt caching lets a repeated prefix be stored once and re-sent cheaply: cache reads bill at ~0.1x the input price and cache writes at 1.25x (prompt caching). A turn dominated by cache reads and writes therefore costs far more than counting input_tokens alone would suggest: for a turn of 500 fresh input tokens, 20,000 cache reads and 3,500 cache writes on Opus, about 13.75x more ($0.0344 versus $0.0025). Count all four usage fields, as record does, or your ceiling is nowhere near where you think it is.

The visible half: task_budget

The second ceiling is set per call, as a budget the model can see, and task_budget takes a token count, not a dollar amount. Converting remaining dollars into it requires a price, and the price depends on which model the next call uses, so the harness does that conversion and the model never sees a dollar figure. The ledger is denominated in dollars; the visible budget is denominated in tokens. They are two different instruments, and confusing their units is a sign the design has not been built.

with client.beta.messages.stream(
    model="claude-opus-5",
    max_tokens=64000,
    betas=["task-budgets-2026-03-13"],
    output_config={"effort": "high",
                   "task_budget": {"type": "tokens", "total": remaining_tokens}},
    tools=TOOLS,
    messages=messages,
) as stream:
    resp = stream.get_final_message()

At 70% spend, inject a budget notice into the conversation so the agent reprioritizes instead of discovering the ceiling by hitting it. Put that notice at the end of the message list, not in the system prompt. The system prompt is the standing instruction block sent at the front of every call; the notice is volatile (its dollar figure changes each call), so putting it at the front would invalidate the cached prefix. Concretely, it belongs after the last cache breakpoint, the marker up to which the prefix is reused. The end of the window is also where recall is best (why quality degrades in long contexts), so the placement is right for two independent reasons.

Memory

“Memory” in an eight-hour agent is not one thing: it is five different stores with five different lifetimes, and each exists because some mechanism above needs data that lives exactly that long.

Ordered by how long each one lives:

• Working memory is the message list of the current model call. It dies with the call.
• The checkpoint survives a crash. That is its whole job.
• The original goal survives everything, including the agent’s own opinions about it.
• Episodic memory is the run’s accumulated lessons. Episodic means drawn from specific past episodes, not from general instructions.
• The filesystem holds the actual outputs, and this is what keeps context flat. The context window is the text the model is given on a single call: finite, and paid for by the token. Because the work lives in files and only the paths are carried forward, the context window stops growing even as the pile of work grows.

The original goal must be stored immutably and compared against literally. If the agent can rewrite it, drift detection compares the queue against a goal that has already drifted, and reports 0.0 while the run goes off the rails. That is the trace above, and it is the central failure this case study guards against.

The episodic layer needs the same discipline as Reflexion lessons (Reflexion), where an agent writes down what it learned from a failure and reads it back on later attempts. A lesson must be specific and falsifiable, or it is a permanent tax on every future prompt:

✗  "Be careful with database migrations."
✓  "CREATE INDEX CONCURRENTLY cannot run inside a transaction block;
    the migration runner wraps everything in one. Use raw_sql_outside_tx()."

The first costs tokens on every call and changes no decision. The second names the constraint, the cause, and the workaround, and it can be proved wrong, which a vague warning cannot.

What a night costs

Deriving the cost of an overnight run from token counts turns the ceiling into a number you can defend, not a guess.

The run to be priced: 8 hours, 40 tasks, roughly 6 model calls per task. Prices: Opus 5 at $5/$25 (input/output per million tokens), and Sonnet 5, the cheaper mid-tier model, at $3/$15.

Each task is a short ReAct loop, and its context grows on every turn, because the whole conversation is re-sent each time: the model is stateless, so call 4 does not remember calls 1 through 3. With a 3.5k first call and 3.3k of new material each turn after, the size of call t is:

input(t) = 3.5k + (t - 1) x 3.3k

Summed over six calls, that is ~70.5k input (not 20k: you pay for the whole context on every call, and 70.5k is the sum of all six) plus ~4.2k output per task. Across 40 tasks: 2.82M input, 168k output. The total grows with the square of the loop length, because a seventh call would add its own ~23.3k, not 3.3k, which is the lever the optimization table pulls hardest on.

Pricing every kind of call the run makes:

The call counts: 40 tasks x 6 calls = 240 execution calls; one verification per task = 40; a drift check every 5 tasks = 8; plus 12 replans and 6 compactions. Compaction is the periodic call that summarizes the history so the context stops growing.

83% of the run is the 240 task-execution calls: $18.30 of $22.01. Everything else is rounding error, including the drift checks: eight Opus calls cost $0.35, which buys the one detector that catches the failure mode defining this architecture.

What each optimization is worth

Each row changes one thing against the $22.01 baseline. The savings are comparable but do not add up: apply two and you get less than the sum, because they compete for the same tokens.

The loop cap is the biggest single win because dropping the 5th and 6th calls removes their own input and every re-send those calls would have carried. Cap the loop first: it costs nothing and also reduces drift surface.

Two caveats decide whether the caching row is real:

• Minimum cacheable length. A prefix is cached only above the model’s floor, which is per model, not per generation. On claude-opus-5 the floor is 512 tokens, so a 3.5k prefix clears comfortably. On claude-haiku-4-5 the floor is 4,096 (above 3.5k) and the entire schedule collapses: every write is refused, every read is a miss, all six calls bill at full input price (prompt caching leverage). Check usage.cache_creation_input_tokens on the first call instead of assuming the marker did something.
• Cache TTL is five minutes. A task whose tool calls are slow (a test suite, a migration, a build) will exceed five minutes between calls. Then every turn pays a 1.25x write with no read to amortize it, which is worse than not caching at all. Measure the inter-call gap per task class, cache only the fast ones, or buy the extended TTL.

The number to lead with

About $22 for an overnight run, so set the hard ceiling at $28: $5.99 of headroom over the derived $22.01, so the ledger refuses the next call instead of cutting the agent mid-edit. The visible task budget is a token count, seeded per task from the derivation (70.5k in, 4.2k out for a six-call task).

The figure that actually matters is cost per completed and verified task, because it is the only one that moves when quality moves:

headroom          $28.00 - $22.01              =  $5.99
verified tasks    40 tasks x 75% pass rate     =  30 tasks
cost per verified $22.01 / 30                  =  $0.73

If the pass rate drops below ~70%, the agent is burning budget on work that gets thrown away, so halt and alert instead of continuing to pay for it. A cheaper run that verifies less is not cheaper.

Failure modes

Everything above compresses into one table: each way the run goes wrong, the signal that reveals it, and the mechanism that contains it. The bolded rows are the load-bearing ones: reconstruct the Guard column from the Failure column and you can rebuild the design.

Two rows have a dash in the detection column, and that is deliberate: a crash and an irreversible mistake cannot be detected after the fact, so they are handled entirely by prevention: checkpoint everything, keep production credentials out of the process, and give the agent no tool that can do damage it cannot undo.

The last two rows are the most important, because the harness evaluates the goal predicate itself. The agent’s claim of success is an input to the report, never the basis for it.

Alternatives considered and rejected

Fully autonomous operation is not always the right tool. Each row names why the alternative is tempting before why it loses.

Three terms in that table deserve plain definitions:

• A cron job is a script run on a fixed schedule by the operating system, with no model involved.
• Fan-out means splitting work across several agents running at once. Reading in parallel is safe because nothing is changed. Writing in parallel means two agents editing the same state with nobody awake to reconcile them.
• A tool allowlist is the explicit list of tools the agent is permitted to call. The constraint is enforced by what exists, not by what the prompt asks for, which is why “no publish verb in the allowlist” is a stronger guarantee than “do not publish anything” in a system prompt.

Evals

Before trusting a night to this design, you need evidence that it works. Eval is short for evaluation: an automated test whose subject is the agent’s behavior, not a single function’s return value.

An eval that restates the implementation cannot fail. “Mutating the goal text raises UNRECOVERABLE_ERROR” is the assert_intact body written out in English. It passes on the one input the author had in mind and says nothing about the input an attacker has in mind. That is why every eval row that names a mechanism also names the case that defeats it and asserts the defeat: the row records what the guard does not do as well as what it does.

The four layers do different jobs:

• Unit tests a single function in isolation.
• Component tests one mechanism, such as the drift detector, against inputs whose right answers you already know.
• Integration runs the whole agent end to end in a sandbox: a throwaway environment with no access to anything real.
• Chaos is the odd one out: it deliberately breaks the machine mid-run to see whether recovery works.

Run integration evals at 10x speed with a shrunken budget: 4 tasks, a $2 cap, a 20-minute wall clock. The shape of every failure above appears at that scale, and you can iterate twenty times a day. Full overnight runs are for release candidates only.

The chaos tests catch what design review cannot. kill -9 sends the same uninterruptible termination signal a cloud provider does when it reclaims your machine, and firing it at a random step is a two-line test harness. It is the only thing that proves your atomic write is actually atomic on the filesystem you deploy to.

Conclusion

The ReAct loop at the center of this design is the easy part. Everything that makes an eight-hour unattended run trustworthy sits outside it, in the harness: the code the model cannot reach.

flowchart TD
    subgraph HARNESS["Harness — non-model code the agent cannot reach"]
        GOAL["Immutable, hashed goal"]
        LEDGER["Dollar ledger + hard cap"]
        PRED["Success predicate"]
        subgraph LOOP["Agent loop (ReAct)"]
            RE[Reason] --> AC[Act] --> OB[Observe] --> RE
        end
        VER["Verify every task<br/>(machine-checkable)"] -.-> LOOP
        CP["Atomic checkpoint<br/>before + after each task"] -.-> LOOP
        DR["Drift check vs goal"] -.-> GOAL
        TERM["7-member termination set,<br/>every exit reported"] -.-> PRED
    end

The load-bearing takeaways:

• Build for autonomy only when actions are reversible, success is machine-checkable, and the budget has a hard ceiling. If the success predicate cannot be written as code, the task is not a candidate.
• The harness owns the verdict. The success predicate, the goal, and the ledger all live in code no tool can influence. An agent that grades its own work reports success by default.
• Every exit is a named, classified, reported ending. An empty queue with an unmet goal is a failure, not a finish, and an exception out of the loop body is a real exit that needs a class.
• Drift is a property of the path, so measure it against an immutable origin held by the harness and judged in a fresh context.
• A torn checkpoint that still parses is worse than none. Atomic rename plus a checksum, and on resume verify the in-flight task instead of re-running it.
• A night costs about $22, dominated by task-execution calls; set the ceiling with headroom and track cost per verified task, not cost per call.
• Prefer the supervised hybrid whenever a human can look at a review queue in the morning.

One line to remember: the ReAct loop is the easy part; every mechanism that makes eight unattended hours trustworthy lives in harness code the agent cannot reach.

Further reading

• Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models” (arXiv:2210.03629): the reason-act loop this design wraps.
• Shinn et al., “Reflexion: Language Agents with Verbal Reinforcement Learning” (arXiv:2303.11366): the discipline behind the episodic-memory layer.
• Anthropic, “Building Effective Agents” (2024): when an agentic loop is worth its cost versus a fixed workflow.

Next: 06 — Customer Support Agent.