Document Processing Agent

Extracting 5,000 invoices/month into a ledger is a fixed pipeline (not an agent) with a deterministic validator, confidence routing, and a narrow agent fallback for the ~5% it cannot classify.

Vocabulary

• Agent: model loops, choosing the next action at run time; step count/order varies per input.
• Workflow / pipeline / DAG: steps fixed in code ahead of time; every input walks the same path, no loops back.
• Ledger: the book of record money is paid from; a wrong row costs real money.
• OCR: pixels to characters, where a smudged 8 becomes a 3.

Not an agent: three questions

Invoice extraction fails all three on the common path, so it lands on a workflow.

1. Do step count/order depend on the environment? No — always classify, extract, validate, route.
2. Is output cheaply verifiable in code? Yes — an invoice self-checks arithmetically (lines sum to subtotal, subtotal + tax = total).
3. Does adaptivity buy more than unpredictability costs? Only matters if 1 and 2 went the other way. Here the steps are writable in advance, so nothing to buy.

Why determinism wins (accuracy first)

• Template errors are bimodal: exact copy, or a loud miss that routes elsewhere.
• Model errors are continuous: smooth spread; worst region is confident-wrong — schema-valid, arithmetically consistent, but not what the page says. Undetectable, so it gets paid.
• An agent makes this worse: every extra ability (zoom, re-read, reconcile) is another chance to construct a plausible number not on the page.

Cost, at 5,000/mo

• Pipeline model spend ~27x cheaper ($80 vs $2,185). Agent-on-everything pays a quadratic history cost (turn n resends all prior turns).
• Human review is 5-40x the model column: the bill is auto-post rate, not cost per call.

Architecture (deterministic, one loop)

PDF -> dedupe(hash) -> invisible-text prefilter -> classify(Haiku)
   -> known layout? yes: template ($0) / no: structured extract (Sonnet)
   -> validate -> calibrated p_correct
        above threshold -> auto-post to ledger
        mid band        -> human review
        low/unparseable -> agent fallback -> human

• Everything before the one agent step is deterministic.

Structured extraction

• Single constrained generation (output_format=Invoice), not a tool loop. Money in integer cents.
• Every numeric field nullable: structured output is enforced by logit masking (schema-breaking tokens set to -inf). A required non-nullable int forces the model to emit a derived number; nullable gives “unreadable” a value.
• “Do not compute values not on the document” is load-bearing: it preserves redundancy. If the model computes total = qty x price and subtotal = sum(lines), the arithmetic check compares the model against itself (a tautology, zero information).
• Prompt caching: cache_control marker set, but SYSTEM (~56 tokens) is below floors, so no discount yet. Minimum cacheable length is not price-ordered: Opus 5 = 512, Sonnet 5 = 1,024, Haiku 4.5 = 4,096 tokens.

Validation (where accuracy comes from)

Four layers, increasing knowledge, feeding a feature vector:

1. Schema — types, required fields.
2. Arithmetic — lines sum to subtotal; subtotal + tax = total; qty x price = line total. (<= 2 cents rounding slack.)
3. Cross-field — due after issue, currency consistent.
4. Business rules — vendor allowlist, PO match, no duplicate, amount in p5..p95 of vendor history.

• Gotcha: “check passed” ≠ “check never ran.” all([]) is True — guard empty line lists; a null field means not-examined, not passed.
• Arithmetic only helps documents with redundancy. A single-line invoice has zero redundancy, so single_line is its own risk class with its own threshold. in_vendor_range and the allowlist are the only features consulting data outside the model’s output.

Confidence: an artifact, not a probability

• Naive 1 - 0.25*len(issues) is wrong: emits only 5 discrete values (tuned threshold does nothing), weights all issues equally (unknown vendor ~2.1% error vs subtotal mismatch ~41%), and isn’t a probability.
• Calibrator (logistic regression, ~500 labelled docs) turns validator outputs into an honest probability: when it says 0.97, ~97/100 are correct. FEATURES order is fixed (one weight per slot). Refit monthly on a held-out month (vendor mix drifts).

Routing threshold from measured costs

Auto-post while P(wrong) x cost_of_error < cost_of_review.

• cost_of_review = 3 min at $30/hr fully loaded = $1.50.
• cost_of_error ≈ $180 (from finance’s write-offs).
• Break-even = 1.50 / 180 = 0.83% error.

• Threshold lands at 0.97 (not the guessed 0.95). Auto-post = top two bands = 60%; 40% of 5,000 to humans = $3,000/mo.
• Threshold is a function of your business, not the model. Publish the inequality, re-derive when a side moves. Raising auto-post is a modeling problem (sharpen calibrator) — splitting “unknown vendor” out lifts 60% -> 70%.

Templates

• Learn: 20 docs with agreeing field positions -> induce (anchor text + bbox + regex). Validate on 5 held-out docs -> live at $0.
• Retire: 5% shadow sample runs the model alongside; disagreement > 2% auto-disables and relearns. Anchor to nearby static text, not absolute coordinates.
• Economics small: saves ~$0.0186/doc, ~$56/mo at 60% coverage vs $3,000 review. Real reasons: bimodal (detectable) errors, and a broken template announces itself. Only induce for vendors above ~15 docs/quarter.

Prompt injection via PDF

• Supplier controls the bytes -> invoice is adversarial input. Payload hidden white-on-white 2pt in the text layer (pdftotext and the extractor read it; humans don’t).
• Four defenses, strongest first:
  1. No injectable field — no approved/remit_to in schema; those tokens have probability zero.
  2. Extractor has no tools — output is one JSON object.
  3. Money never moves on document data — remittance from vendor master under dual control (stops BEC).
  4. Cheap prefilter — flag near-background colour, size <= 3pt, off-canvas, invisible render mode, or absent-from-render. Weakest; a tripwire, not the stop.
• Gotchas: compare colours by perceptual distance, not == (#fffffe, #fff, rgb(255,255,255) are one colour); use <= 3.0, not < 3.0.

Agent fallback

• One place a model may loop; runs on ~5% (multi-page, handwritten, cut-off scans, credit notes).
• Five read-only tools: read_page, crop_and_enhance, lookup_vendor, lookup_po, flag_for_human. Nothing writes/approves/touches bank details. Max privilege: “read this document harder.”
• flag_for_human mandatory (else it forces a confident guess). AGENT_STEP_CAP = 8 must be a harness constant, never model-influenced. Every exit routes to a human.

Cost breakdown & memory

• Per doc: classify (Haiku) $0.0021; standard extract (Sonnet) $0.0207; agent fallback (7 turns) $0.15. Total ≈ $80/mo; agent path is 5% of docs, 47% of the bill.
• Memory layers: working (this doc), procedural (templates), semantic (vendors, tax IDs, ranges), episodic (corrections). Each correction = 3 artifacts: ledger fix, calibrator label, permanent regression case (+ few-shot).

Gotchas

• A failure you cannot detect (wrong amount, arithmetic still checks) is the one your bank tells you about — audit 1% of auto-posted.
• Field-level precision/recall, not document-level: nullable schema aims for high-precision/low-recall fields.
• Calibration is financial: if it says 0.95 but reality is 0.88, you pay wrong invoices at (12%/5%) = 2.4x assumed. Track reliability curve + Brier score.
• Run integration set at N=3 majority: temperature 0 isn’t deterministic. Dedupe by hashing PDF bytes, not model output.