27 — Design A Payment System

“Design the payment flow for an e-commerce checkout.”

Charging a credit card is one HTTP call. Knowing, afterwards, that it charged the right amount exactly once is the real problem, and this chapter builds the part of an online store that solves it.

Three mechanisms do that work, and each gets its own section:

1. Double-entry bookkeeping (Deep dive 1 double entry derived) — a way of recording money that makes a half-finished movement impossible to write down in the first place.
2. Idempotency (Deep dive 3 idempotency across a boundary you cannot roll back) — a way of making a retried charge harmless.
3. Reconciliation (Deep dive 5 reconciliation) — a daily line-by-line comparison against the card processor’s own records.

Each one catches a different class of failure, and no one of them covers another’s class. Being able to say which failure belongs to which mechanism — and which failures none of them catch — is most of what an interviewer is listening for. By the end you should be able to explain to someone else why “just retry the payment” is the single most expensive sentence in this design.

What goes in and what comes out. The input is one HTTP request from a checkout page, carrying:

• an order id, an amount, a currency, and a token standing in for the customer’s card (a meaningless string the PSP will exchange for the real card number, so the card number never touches your servers);
• a header called an idempotency key — a caller-supplied string that says “this is the same request I sent a moment ago, not a new one.”

The output is three things:

• a payment record whose state is one of AUTHORIZED, DECLINED, or UNKNOWN;
• a set of immutable bookkeeping rows called ledger entries that record where the money went;
• and, when the system genuinely cannot tell what happened at the card network, a case in a queue that a human works.

That third output is not a failure of the design. It is a designed output, and it is sized in Failure modes.

Vocabulary. Four terms, all of them used from here on:

• Payment service provider (PSP) — the outside company (Stripe, Adyen, Braintree) that actually talks to the card networks on your behalf. It is the third party whose behaviour drives every hard decision below.
• Ledger — the append-only list of rows that is the system’s record of every movement of money. It is the truth, and nothing else is.
• Minor units — whole pennies. $50.00 is written 5000, never 50.00.
• Double-charge — the specific catastrophe this chapter exists to prevent: one customer intent that results in two real charges on one card.

A payment system is the rare design where throughput is trivial and correctness is everything. The workload below peaks at 300 requests/s, which one machine handles without effort.

The chapter is entirely about a different problem: some of those requests move money through a company you do not control, on a timescale you do not control, with no rollback.

Three numbers carry the whole answer. Each is derived in the section named beside it, so treat the table as a map rather than a summary:

Three ideas from elsewhere in the repo are used here. Each is restated in place, so you never have to leave this page:

• Idempotency — the property that doing something twice has the same effect as doing it once. The four rules for building it (hash the request to get the key, namespace the key per operation, claim it with a single atomic database statement, put it on every mutating endpoint) are derived in 9a the idempotency rules in one place and restated with their reasoning in Deep dive 3 idempotency across a boundary you cannot roll back.
• Isolation levels — the database settings that decide which concurrency surprises a transaction is allowed to see. The full table of which setting permits which surprise is in sql 03.
• Delivery semantics — what a message queue promises about duplicates. The one that matters here is at-least-once: a message may be delivered more than once, but never zero times. Ch 20 owns that topic.

1. Framing: what decision, and what breaks

The whole design turns on one decision: where the truth about money lives, and what makes a lie detectable. Everything else — which payment service provider, which queue, how many copies of the database — is downstream of that.

What breaks is not the happy path. It is the three-second window where your process has told the PSP to charge a card and has not yet heard back.

In that window the system genuinely does not know whether the customer has been charged, and no consistency protocol can tell it. No amount of clever agreement between machines helps here, because the other participant is a company that will not join your transaction — it will not accept an instruction to commit or abort its side along with yours.

So the design goal is: every unknown becomes a detected unknown, and every detected unknown has an owner. A payment system that silently loses $50 is worse than one that stops and shouts, because the shout is a page and the loss is a regulator six months later.

Requirements

Four verbs recur throughout and are worth pinning down before they appear in a requirements list:

• Authorize — ask the customer’s bank to set aside the money and promise to hand it over later. No money has moved yet.
• Capture — claim that promised money. This is when it actually starts moving.
• Void — cancel an authorization before capturing it.
• Refund — send captured money back.

A fifth term is a noun, not a verb: settlement is the PSP wiring you the accumulated money a day or two later.

Functional

• Authorize, capture, void, and refund a card payment through one or more PSPs.
• Maintain a ledger that answers “what is the balance of account X at time T” exactly.
• Reconcile daily against the PSP’s settlement file — that is, line up our own records against the provider’s list of what it actually processed, and give every disagreement a name and an owner.
• Expose payment state to the order service, both by webhook (the PSP calls a URL of ours when something changes) and by polling (we ask, on a timer).

Non-functional. Two shorthands in the table. p99 is the 99th percentile: the latency that 99 out of 100 requests come in under, so it describes the slow tail rather than the typical case. RPO is recovery point objective, the amount of recent work you are willing to lose if a machine dies; RPO 0 means none at all, which forces every ledger write to be copied to a second machine before it is acknowledged.

Turn the availability target into a number of minutes so it means something. A month is 30 days of 1,440 minutes, and 99.99% available means 0.01% — a factor of 0.0001 — unavailable:

minutes/month:            30 x 1,440             =  43,200
allowed downtime, min:    43,200 x 0.0001        =  4.32

But our own availability is capped by the PSP’s. If the provider is up 99.9% of the time, a payment cannot succeed more often than that, however perfect our code is. The whole chain is only as available as its least available link, and that link is not ours.

The fix is a second provider you can fail over to. If provider A is down 0.1% of the time (0.001) and provider B is down 0.1% of the time independently, then both being down at the same instant has probability 0.001 x 0.001, and availability is one minus that:

both down at once:        0.001 x 0.001          =  0.000001
availability, two PSPs:   1 - 0.000001           =  0.999999

Two providers take 99.9% to 99.9999%: four extra nines from an integration, not from an architecture. That one line is the entire argument for multi-PSP routing, and it belongs in the requirements section, not the scaling section.

“Independently” is doing real work in that sentence. If both providers sit behind the same upstream card network and that network has the outage, the failures are correlated and the multiplication does not hold.

Back-of-envelope

Four numbers decide the architecture: how many requests per second, how many bytes, how much money is at stake when the accounting is wrong, and how many people the exceptions need.

Assume a large e-commerce checkout: 10 M payments/day, $50.00 average order. Money is in minor units throughout, so $50.00 is 5,000 minor units.

Two conventions before the arithmetic. The 100,000 below is the customary round stand-in for the 86,400 seconds in a day; using it costs about 15% of accuracy and saves you doing long division on a whiteboard, which is the right trade at this stage. GMV is gross merchandise value: the total money customers spend, before any fees are taken out.

payments/s, average:      10,000,000 / 100,000   =  100
payments/s, peak:         100 x 3                =  300
GMV/day, minor units:     10,000,000 x 5,000     =  50,000,000,000

50,000,000,000 minor units is $500 M/day. Next, what the provider takes. PSPs charge a percentage of the amount plus a flat fee per transaction; at a typical 2.9% + $0.30, on the average $50.00 order:

percentage fee, minor:    5,000 x 0.029          =  145
flat fee, minor:          0.30 x 100             =  30
fee per order, minor:     145 + 30               =  175
fees/day, minor:          10,000,000 x 175       =  1,750,000,000

1,750,000,000 minor units is $17.5 M/day in fees. Hold that figure: a 0.1% error in fee accounting is $17,500 a day walking out of the building unnoticed, which is why Deep dive 5 reconciliation exists.

How much storage

Storage is the next question, and it starts from the size of a single ledger entry, field by field. The numbers on the right are bytes; the last line adds them up:

entry_id            16   B   (UUID, binary)
transaction_id      16   B
account_id           8   B
amount_minor         8   B   (int64, signed)
currency             4   B
entry_type           1   B
created_at           8   B
idempotency_ref     16   B
payload, bytes:     16+16+8+8+4+1+8+16           =  77

77 B is the data you wrote. It is not what the row costs on disk, for two reasons.

First, round 77 B up to 80 B: databases pad rows out to convenient boundaries, so the last few bytes are free anyway.

Second, multiply by 3. The factor of 3 is an estimate of everything the database stores alongside your columns:

• a per-row header — bookkeeping the storage engine keeps about each row (its transaction visibility, its length);
• the primary key index;
• two secondary indexes, meaning extra sorted copies of a few columns that let a query find rows by something other than the primary key. Ours are on (account_id, created_at) and on transaction_id. Both are in the schema below because both read paths need them.

on disk, bytes/entry:     80 x 3                 =  240

Now multiply out. A single payment produces four ledger entries — receivable, fee, revenue, and one settlement leg — and there are 10 M payments a day:

entries/day:              10,000,000 x 4         =  40,000,000
bytes/day:                40,000,000 x 240       =  9,600,000,000

9,600,000,000 bytes/day is 9.6 GB/day. Retention is 7 years, and each of those years is 365 days:

GB over 7 years:          9.6 x 365 x 7          =  24,528
GB with 3 replicas:       24,528 x 3             =  73,584

74 TB of replicated ledger is three commodity machines’ worth of disk. Storage is not the constraint, queries per second (QPS) is not the constraint, and latency is not the constraint. Say this in the first two minutes: “the numbers say this fits on one box, so the interesting part is not scale — it’s that a third party holds half the state.”

The one number that sizes a component

One figure here does drive a real design choice: how much memory the open authorizations need.

An authorization sits open from the moment the card is approved until the goods ship and you capture. Assume that lag is 4 hours at the median, which is 14,400 seconds. At 100 payments/s, the working set — the rows the system has to keep at its fingertips rather than merely on disk — is:

open auths, rows:         100 x 14,400           =  1,440,000
memory, bytes:            1,440,000 x 240        =  345,600,000

345,600,000 bytes is 346 MB, which fits in RAM on any machine you would buy. So “which authorizations are still open” is an in-memory question, and the sweep that expires stale ones is cheap enough to run constantly.

What being wrong costs

The last number is the one that funds the rest of the chapter. Suppose one payment in ten thousand results in a double-charge — the same customer intent billed twice:

double charges/day:       10,000,000 / 10,000    =  1,000
dollars/day:              1,000 x 50             =  50,000
dollars/year:             50,000 x 365           =  18,250,000

A one-in-ten-thousand double-charge rate costs $18.25 M a year, and that is the budget every mechanism in Deep dive 3 idempotency across a boundary you cannot roll back is spent against. Quote it whenever someone calls the idempotency work over-engineering.

API sketch

The interface is where the design’s commitments become visible, so three of them are deliberately hard to miss: money is an integer, “I do not know” is a legal answer, and every endpoint that changes something takes an idempotency key.

Read the block below as two halves. The top half is one request and the four responses it can produce; the bottom half is the rest of the endpoints, compressed onto one line each. The arrows in the top half are annotations, not part of the wire format.

POST /v1/payments
Idempotency-Key: 8f14e45f...         <- REQUIRED. sha256 of the canonical request
{ "order_id": "9001", "amount_minor": 5000, "currency": "USD",
  "method_token": "tok_live_...", "capture": "manual" }

201 { "payment_id": "pay_a1", "state": "AUTHORIZED", "psp_ref": "ch_..." }
202 { "payment_id": "pay_a1", "state": "AUTHORIZING" }   <- outcome unknown, poll
409 { "error": "idempotency_key_reused_with_different_body" }
402 { "error": "card_declined", "decline_code": "insufficient_funds" }

POST /v1/payments/{id}/capture   Idempotency-Key: ...   { "amount_minor": 5000 }
POST /v1/payments/{id}/void      Idempotency-Key: ...
POST /v1/payments/{id}/refund    Idempotency-Key: ...   { "amount_minor": 5000,
                                                          "reason": "..." }
GET  /v1/payments/{id}
GET  /v1/accounts/{id}/entries?from=...&to=...&cursor=...

Three things a reviewer looks for in that sketch.

amount_minor is an integer, and the field name says so. No endpoint in this system accepts a decimal string that a client might have produced with a float — the computer’s fast but approximate representation of a fractional number, whose failures are the subject of Deep dive 2 money is never a float.

202 is a first-class response, not an error. HTTP 202 means “accepted, outcome not yet determined”. “I do not yet know whether the card was charged” is a legitimate outcome of a payment request, and an API with no way to say it forces the caller to guess — which, in practice, means the caller retries and charges the card again.

Idempotency-Key is on every mutating endpoint, not just create. A mutating endpoint is any one that changes state rather than merely reading it. This is rule R4 of the four idempotency rules (9a the idempotency rules in one place), and it is the one an API sketch most often gets wrong.

Here is why R4 matters. A create call that double-charges is the bug everyone designs against. A retried refund that refunds twice moves the same money in the other direction, out of the same books, and nothing in a create-only contract stops it.

The key is not free-form either. It is a canonical hash of the request body: a fixed-length fingerprint computed after sorting the fields into one agreed order, so that the same intent always produces the same fingerprint. A retry of the same refund therefore produces the same key by construction, rather than by the client remembering to reuse one.

Data model

Four tables carry everything:

• accounts — what accounts exist;
• ledger_transactions — what money movements happened;
• ledger_entries — the individual signed lines of each movement;
• psp_calls — a record of every call made to the PSP.

The accounting vocabulary in the first table is small. An asset is something you own (cash, money the PSP owes you). A liability is something you owe (money held on a customer’s behalf). Revenue is money earned, and an expense is money spent — here, the PSP’s fees.

CREATE TABLE accounts (
  account_id   BIGINT PRIMARY KEY,
  kind         TEXT NOT NULL,     -- asset | liability | revenue | expense
  currency     CHAR(3) NOT NULL
);

CREATE TABLE ledger_transactions (
  transaction_id  UUID PRIMARY KEY,
  kind            TEXT NOT NULL,  -- CAPTURE | SETTLE | REFUND | CHARGEBACK | ADJUST
  payment_id      UUID,
  created_at      TIMESTAMPTZ NOT NULL
);

CREATE TABLE ledger_entries (          -- APPEND ONLY. No UPDATE, no DELETE.
  entry_id        UUID PRIMARY KEY,
  transaction_id  UUID NOT NULL REFERENCES ledger_transactions,
  account_id      BIGINT NOT NULL REFERENCES accounts,
  amount_minor    BIGINT NOT NULL CHECK (amount_minor <> 0),
  currency        CHAR(3) NOT NULL,
  created_at      TIMESTAMPTZ NOT NULL
);
CREATE INDEX ON ledger_entries (account_id, created_at);
CREATE INDEX ON ledger_entries (transaction_id);

CREATE TABLE psp_calls (               -- written BEFORE the network call
  idempotency_key TEXT PRIMARY KEY,  -- "<client key>:<operation>", ch 23 R2
  payment_id      UUID NOT NULL,
  request_hash    BYTEA NOT NULL,
  state           TEXT NOT NULL,   -- IN_FLIGHT | DONE | UNKNOWN | MANUAL
  started_at      TIMESTAMPTZ NOT NULL,   -- IN_FLIGHT ages out of it, section 4
  psp_ref         TEXT,
  response        JSONB
);
CREATE INDEX ON psp_calls (state, started_at);   -- the recovery worker's query

Two things in that schema deserve unpacking.

amount_minor is signed, meaning it can be negative. A debit (money into an account) is positive; a credit (money out of it) is negative. An account’s balance is the sum of its entries — notice that there is no balance column anywhere in this schema. That omission is deliberate and it is the subject of the next section.

psp_calls is the row that must exist before any call to the provider. Its state column takes four values, and all four recur throughout the chapter:

The index on (state, started_at) exists purely so a background worker can cheaply ask “which rows have been IN_FLIGHT for longer than the call timeout” without scanning the table.

High-level architecture

The diagram below is the whole system on one page. Read it as three journeys that share a database: the synchronous charge path down the left, the asynchronous notification path up from the provider, and the daily comparison at the bottom.

flowchart TD
    C["Checkout service"] -->|"POST /payments + Idempotency-Key"| API["Payment API"]
    API --> IDEM["psp_calls row<br/>state IN_FLIGHT<br/>committed BEFORE the call"]
    IDEM --> RT["PSP router<br/>primary and failover"]
    RT -->|"HTTPS, p99 2 s"| PSP1["PSP A"]
    RT -->|"failover"| PSP2["PSP B"]
    RT --> LED["Ledger writer<br/>append-only, zero-sum txns"]
    LED --> DB[("Postgres<br/>ledger and accounts<br/>sync replica, RPO 0")]
    PSP1 -->|"webhook"| WH["Webhook ingest<br/>verify signature, dedupe by psp_ref"]
    WH --> Q["Queue, ch 20"]
    Q --> SM["Payment state machine worker"]
    SM --> LED
    SM --> DLQ["Dead letter"]
    DLQ --> OPS["Human queue<br/>sized in section 7"]
    PSP1 -->|"daily settlement file"| REC["Reconciler"]
    DB --> REC
    REC --> BRK["Break records<br/>five classes"]
    BRK --> OPS

    style IDEM fill:#1d3557,color:#fff
    style LED fill:#2d6a4f,color:#fff
    style REC fill:#bc6c25,color:#fff
    style OPS fill:#7f1d1d,color:#fff

Journey 1: the charge, going out. The Checkout service is the caller — the part of the store that has a cart and a card token and wants a charge. It sends POST /payments with an idempotency key. The Payment API writes the psp_calls row and commits it with state IN_FLIGHT before anything leaves the building. The PSP router then picks a provider, primary or failover, and makes the actual charge; that is the one hop in the diagram with an HTTPS p99 of 2 s, four thousand times the cost of an internal call. The Ledger writer turns the outcome into append-only, zero-sum transactions, which land in the Postgres ledger and accounts database, kept honest by a synchronous replica so RPO 0 holds.

Journey 2: the notification, coming back. Later, the PSP calls us back over the webhook. Webhook ingest verifies the provider’s cryptographic signature so a forged callback is rejected, deduplicates on the provider’s own reference id (psp_ref) so a repeated callback is a no-op, and puts the event on a queue. The payment state machine worker applies it as a transition (Deep dive 4 async settlement and why state is a machine) and writes any resulting ledger entries. Anything it still cannot process once its retries are exhausted goes to a dead letter store, and from there to the human queue.

Journey 3: the daily comparison. Once a day the provider drops a settlement file. The reconciler joins that file against our own ledger and emits break records — a break being one row the two sides disagree about — in the five classes of Deep dive 5 reconciliation. Those also feed the human queue.

Four boxes are load-bearing and the rest is plumbing: the idempotency row written before the call, the append-only ledger, the reconciler, and the human queue. An answer that draws the first three and omits the fourth has not run a payment system.

2. Deep dive 1 — double-entry, derived

Double-entry is the six-hundred-year-old accounting rule that every movement of money is recorded as two or more signed lines that add up to zero: money leaving one account is the same event as money arriving in another, so it is written once, in two halves. The whole system rests on it, so it is worth deriving from the naive design and the specific bug that kills it — which is also how you learn what the rule catches, what it provably cannot catch, and what it costs to verify.

2.1 What a single balance column loses

Start with the failure that motivates everything else; it takes nothing more exotic than two lines of Python.

The obvious model is one row per account with a balance column, and a transfer is two UPDATEs — two statements that overwrite a stored number.

Here is the failure, with no distributed systems in it at all. Watch the two final assertions: the customer’s 5,000 has left, and revenue never received it.

# The single-column model: two updates, and no way to tell they both landed.
accounts = {"customer": 5000, "revenue": 0}

def transfer_single_column(src, dst, amount, crash=False):
    accounts[src] -= amount
    if crash:
        raise RuntimeError("process died between the two updates")
    accounts[dst] += amount

try:
    transfer_single_column("customer", "revenue", 5000, crash=True)
except RuntimeError:
    pass

# Both rows are individually plausible. No constraint is violated, no CHECK
# fires. Zero is a completely normal thing for an account balance to be.
assert accounts["customer"] == 0
assert accounts["revenue"] == 0

5,000 minor units left the system and no query can find them. The corruption is not in any row — both rows hold a number an account is perfectly entitled to hold. It is in the relationship between two rows, and the schema does not record that relationship anywhere.

The obvious objection: wrap both updates in one database transaction. That fixes this instance and nothing else, because in a real payment system the two sides of a money movement are frequently not available to one transaction. One side may be:

• a card network, which will not join your transaction at all;
• another shard — one of several machines the data has been split across, so no single machine holds it all (ch 28);
• a batch file that does not arrive until tomorrow.

The single-column model has no representation for “half of this movement has happened”, so it represents that state as a normal balance.

2.2 The invariant

What replaces the balance column is a single rule — an invariant, a statement that must be true of the data at all times. The value of having one is that checking it is a single query rather than a judgement call.

Double-entry replaces the balance column with an append-only list of signed entries — append-only meaning rows are only ever added, never edited and never deleted — grouped into transactions, under one rule:

Every transaction’s entries sum to zero. Therefore the sum of all entries in the ledger is zero, at every instant, forever.

Both halves matter, for different reasons. The per-transaction rule is enforceable at write time, as a constraint the database refuses to violate. The global rule follows from it, and it is what makes the books auditable: checking them is one aggregate over one table — no joins, no interpretation, no judgement call.

The code below is the whole scheme in fifteen lines. post is the write path and refuses anything that does not balance; balance derives an account’s balance by summing its entries rather than reading a column; global_invariant is the audit query. The last three lines re-run the crash from §2.1 and show it now producing a number that screams.

LEDGER = []   # append-only: (transaction_id, account, signed_minor_units)

def post(txn_id, entries):
    """Refuses to write unless the transaction is zero-sum, so a half-written
    money movement cannot be represented, let alone persisted."""
    if sum(amount for _, amount in entries) != 0:
        raise ValueError("transaction does not balance")
    for account, amount in entries:
        LEDGER.append((txn_id, account, amount))

def balance(account):
    return sum(a for _, acct, a in LEDGER if acct == account)

def global_invariant():
    return sum(a for _, _, a in LEDGER)

post("t1", [("customer_payable", -5000),
            ("psp_receivable", 4825),
            ("psp_fee_expense", 175)])
assert balance("psp_receivable") == 4825
assert global_invariant() == 0

# The same crash as above, now representable and therefore detectable.
LEDGER.append(("t2", "customer_payable", -5000))   # a half-written movement
assert global_invariant() == -5000

Here is that same $50.00 order captured, written the way it appears in the books.

Two account names carry accounting meaning. A receivable is money someone owes you but has not yet handed over — here, the PSP is holding your $48.25. A payable is money you owe someone else — here, the $50.00 of goods you now owe the customer.

The three amounts come from the fee arithmetic above: the customer’s 5,000 minor units split into the 175 the PSP keeps and the 5,000 - 175 = 4,825 it will eventually pay you. The numbers in the left column are conventional account codes, and the signs follow the schema: debits positive, credits negative.

txn CAPTURE order 9001, 5,000 minor units, PSP fee 2.9% plus 30
  1101  psp_receivable      asset      +4825
  5010  psp_fee_expense     expense     +175
  2101  customer_payable    liability  -5000
                                       ------
  sum                                       0

Settlement two days later moves the receivable into cash and touches nothing else. Same 4,825, one account to another, and the transaction still sums to zero:

txn SETTLE batch 2026-07-29
  1001  bank_cash           asset      +4825
  1101  psp_receivable      asset      -4825
                                       ------
  sum                                       0

Note what is not here: the authorization. An authorization is a promise from the issuer — the bank that gave the customer the card — and a promise is not a movement of money, so it produces no ledger entries. It produces a hold record with an expiry instead. Booking the authorization into the ledger is the most common modelling error in this design, and it corrupts the books the moment an authorization expires unused.

2.3 What the invariant catches that a balance column does not

Now the honest accounting of the rule’s coverage: five bugs it catches, and one large class it is structurally unable to catch.

Read the table one row at a time, comparing the middle column to the right one. The row to slow down on is the fifth, because it is the one where double-entry loses.

The retry row is the honest one. Double-entry catches asymmetry, not duplication. A transaction applied twice is internally consistent and sums to zero perfectly — it simply describes a world in which the customer bought two things. Saying this unprompted is what separates a memorized answer from a used one.

Per currency is not a footnote either. A single global SUM over a multi-currency ledger is meaningless: 100 JPY and 100 USD are different quantities, and adding them produces a number that means nothing. So the real invariant is SUM(amount_minor) = 0 GROUP BY currency, and every transaction is single-currency.

That raises an obvious question: how do you record a currency conversion, which is inherently two currencies? A foreign-exchange (FX) conversion is two transactions joined by an FX position account, never one transaction with two currencies in it. The position account is a holding place that ends up short one currency and long the other, so each of the two transactions still balances within its own currency and the per-currency invariant survives.

2.4 What checking the invariant costs

An invariant you cannot afford to check is a wish. So price the check.

The check is a full aggregate: one pass that reads every row and adds up a column. Its cost is therefore just the time to read the data. Price it against the 24,528 GB of ledger computed above and the 1 GB/s sequential-read figure from the standing latency table. Dividing gigabytes by gigabytes per second leaves seconds:

full history, seconds:    24,528 / 1             =  24,528
full history, hours:      24,528 / 3,600         =  6.8

Six hours is not a monitor, it is a quarterly exercise. A drift introduced at 09:00 would be found some time after lunch, having been replicated, reported on, and paid out against in the meantime.

The fix uses the one property the ledger has that ordinary data does not: the past does not change. Because entries are append-only, yesterday’s sum is a constant, and there is no reason to recompute it.

So store the running total at each day boundary — that stored total is a checkpoint — and check only today’s rows against it. Splitting a table into one physical chunk per day is called partitioning; each chunk is a partition, and the one being written to right now is the open partition. Today’s partition is one day of ledger, which the storage arithmetic above put at 9.6 GB:

today only, seconds:      9.6 / 1                =  9.6

A 9.6-second query, run every five minutes, means ledger drift is detected within five minutes of being introduced. That is the number to quote.

The full-history recompute still runs, monthly, but its job is different: it is what validates the checkpoints themselves. Without it, a corrupted checkpoint would make every 5-minute check agree with a lie.

Four layers enforce the invariant, and each one catches what the layer before it missed:

1. CHECK (amount_minor <> 0) plus a per-transaction zero-sum constraint — a rule the database itself refuses to violate, applied at write time, cheap, and it catches the ordinary application bug.
2. REVOKE UPDATE, DELETE ON ledger_entries — REVOKE is the SQL command that takes a permission away, so this makes “correct it by editing the row” impossible rather than merely discouraged. Nobody, including the application, holds the privilege.
3. The 5-minute partition sum — catches whatever got past 1 and 2.
4. Monthly full recompute against the checkpoints — catches a corrupted checkpoint, which is the one thing level 3 trusts blindly.

3. Deep dive 2 — money is never a float

Keeping cents from evaporating is a type discipline, and it compresses to three rules you can enforce in a code review: store whole pennies as integers, use an exact decimal type only for genuine fractions, and never divide money without handing out the leftover.

A float, short for floating-point number, is how computers usually store fractional values: a fixed number of binary digits plus an exponent. It is fast, and it is wrong in exactly the way that matters here.

The reason is that binary fractions cannot represent 0.1, 0.2, or 0.3 exactly, any more than decimal digits can represent one third exactly. 0.1 in a float is a nearby number, not one tenth. Arithmetic on floats is therefore arithmetic on nearby numbers, and the error survives into whatever you store.

The block below has four parts, numbered in its comments: (1) the failure, (2) the integer fix, (3) the Decimal fix for genuine fractions, and (4) division, which is where money actually goes missing. Every line is an assertion, so the block passing is the claim.

from decimal import Decimal, ROUND_HALF_UP

# 1. The failure, stated exactly.
assert 0.1 + 0.2 != 0.3
assert repr(0.1 + 0.2) == "0.30000000000000004"

# It accumulates. Ten additions of a dime do not make a dollar.
total = 0.0
for _ in range(10):
    total += 0.1
assert total != 1.0

# 2. Fix A -- integer minor units. Exact, fast, and what wire formats use.
def cents(dollars: str) -> int:
    return int(Decimal(dollars).scaleb(2).to_integral_value(ROUND_HALF_UP))

assert cents("0.10") + cents("0.20") == cents("0.30") == 30

# 3. Fix B -- Decimal, for intermediates that genuinely need more than two
#    places: FX rates, interest accrual, per-unit prices.
assert Decimal("0.1") + Decimal("0.2") == Decimal("0.3")

# 4. Division is where money is actually lost, and integers do not save you.
#    1,000 minor units split three ways is 333 each, and one unit vanishes.
def split(total_minor: int, n: int) -> list:
    base, remainder = divmod(total_minor, n)
    return [base + (1 if i < remainder else 0) for i in range(n)]

parts = split(1000, 3)
assert parts == [334, 333, 333]
assert sum(parts) == 1000        # the invariant that makes the split safe

Three rules follow, and all three are enforceable in code review:

• Store and transmit integer minor units. amount_minor BIGINT. A 64-bit signed integer (int64) holds 9.22e18 minor units — larger than any figure you will ever transact, by many orders of magnitude.
• Use Decimal only where a fraction of a minor unit is genuinely needed, and round to minor units before anything is posted to the ledger. Decimal is an exact base-ten number type — it stores the digits you actually wrote, so 0.1 is really one tenth — at the cost of being slower than a float. A rate is a Decimal; a ledger entry is an int.
• Never divide without redistributing the remainder. The split above is the pattern: compute the base, then hand the remainder out one unit at a time so the parts sum to the whole. The assert sum(parts) == 1000 is not a test, it is the design.

The connection back to Deep dive 1 double entry derived is worth making explicit. Suppose someone writes the naive split anyway. The transaction’s entries then come to 333 + 333 + 333 = 999 against a stated 1,000, the sum is not zero, and the zero-sum constraint rejects the write. The bad transaction never lands, rather than drifting a unit at a time into the books. Two mechanisms, one failure caught twice.

One last trap: declaring the column DECIMAL/NUMERIC in the database is not enough on its own. The value passes through JSON, through a client library, and through application code on its way to that column, and any of those hops can put it in a double and hand back a rounded number. The type discipline has to hold end to end or it holds nowhere, which is why the API field is named amount_minor and typed as an integer.

4. Deep dive 3 — idempotency across a boundary you cannot roll back

This is the problem the chapter exists for: making a retried charge harmless when the thing being retried happens at another company. The answer ends in a small state machine you could implement tomorrow.

4.1 The four rules, restated

Four rules make a key work, and they are the same four whether you are booking a hotel room or charging a card. 9a the idempotency rules in one place derives them in a setting with no third party; here they are restated so you need not go and look:

• R1 — the key is a hash of the request, not a random number. Compute it from the request’s meaningful fields, in a canonical order, leaving out anything that changes per attempt.
• R2 — namespace the key per operation. Append the name of the operation to the key so different operations on one intent get different keys.
• R3 — claim the key with a single atomic statement. One statement both inserts the key and tells you whether you were the one who inserted it.
• R4 — put the key on every call that changes something, not just on the create call.

Those four are stated abstractly. The table below says what each one turns into once the operation being retried is a card charge — read the right-hand column as the concrete instruction and the middle one as the reminder of why:

Two words in R3 are worth spelling out. A race is what you have when two copies of the same code run at once and the outcome depends on which one gets there first. A round trip is one request to another machine plus its reply — about half a millisecond for a database on the same network, and an eternity when a charge can slip into it.

4.2 What changes when the other side is another company

The PSP call cannot be rolled back, and it happens in the middle of your transaction.

A hotel booking is a row you own. Change your mind and you issue ROLLBACK, and the database throws your changes away as though they never happened.

A card charge is an effect at Visa. There is no ROLLBACK for it. And no protocol lets you learn its outcome atomically with your own commit — “atomically” meaning as one indivisible step that either wholly happens or wholly does not.

That single asymmetry forces one rule:

The idempotency key is generated before the external call and committed with the record of that call, in its own transaction, before a single byte goes over the wire.

The argument for it is a two-case proof. Consider a crash at the worst possible moment — after the charge has gone out, before you have written anything down.

Key written after the call. The crash leaves no evidence a call was ever made. Recovery finds a payment with no PSP record and does the only thing it can with that information, which is call again. The customer is charged twice.

Key written before the call. The crash leaves an IN_FLIGHT row. Recovery finds a call whose outcome is unknown — a completely different situation, and a recoverable one, because the row carries the key you need to go ask what happened.

4.3 The mechanism, in code

The block below is that mechanism in runnable form. It has four parts:

• payment_key — the key derivation of R1: hash the request, drop the per-attempt fields.
• op_key — the namespacing of R2: one row per (intent, operation).
• PspCallStore.begin — the atomic claim of R3, and the three answers it can give: PROCEED (you own this call, make it), REPLAY (already done, here is the stored result), RECOVER (someone else started it and we do not know how it ended).
• PspCallStore.recover — the path that gets a stuck row out of IN_FLIGHT, by asking the PSP rather than charging again.

The assertions after the class are the interesting part. They start eight threads at the same instant to show that the one-statement claim really does admit exactly one charge, and then they walk a stuck row to a human case and, separately, to a resolved one.

import hashlib, json, threading, time

VOLATILE = {"attempt", "retry_count", "client_ts", "requested_at", "trace_id"}

def payment_key(request: dict) -> str:
    """R1: a canonical hash of the intent, per-attempt fields excluded."""
    canonical = {k: v for k, v in sorted(request.items()) if k not in VOLATILE}
    return hashlib.sha256(
        json.dumps(canonical, sort_keys=True, separators=(",", ":")).encode()
    ).hexdigest()

def op_key(key: str, operation: str) -> str:
    """R2: one psp_calls row per (intent, operation)."""
    return key + ":" + operation


class PspCallStore:
    """One row per (intent, operation), committed BEFORE the network call, so a
    crash always leaves evidence that a call may be in flight."""

    LATENCY = 0.002                     # one statement's round trip to the DB

    def __init__(self):
        self.rows = {}
        self.states = set()             # every state this code has ever produced

    def begin(self, key, request, now=None):
        h = hashlib.sha256(json.dumps(request, sort_keys=True).encode()).hexdigest()
        fresh = {"state": "IN_FLIGHT", "hash": h, "result": None,
                 "started": time.time() if now is None else now}
        time.sleep(self.LATENCY)
        # R3. ONE statement claims the key:
        #   INSERT INTO psp_calls (...) VALUES (...) ON CONFLICT DO NOTHING
        # `setdefault` is that statement -- it returns the row that is in the
        # table after the call, so the winner is known by identity and there is
        # no gap between the check and the act.
        row = self.rows.setdefault(key, fresh)
        self.states.add(row["state"])
        if row is fresh:
            return "PROCEED", None
        if row["hash"] != h:
            raise ValueError("idempotency key reused with a different body")
        if row["state"] == "DONE":
            return "REPLAY", row["result"]
        return "RECOVER", None          # a prior attempt's outcome is unknown

    def finish(self, key, result):
        self.rows[key].update(state="DONE", result=result)
        self.states.add("DONE")

    def recover(self, key, psp_lookup, now, timeout_s=30):
        """IN_FLIGHT is not a resting state. Past the call timeout the row moves
        to UNKNOWN, and UNKNOWN is worked by a LOOKUP -- never by re-sending the
        charge. An inconclusive lookup is MANUAL, which is the human queue of
        section 7 and the only exit this design refuses to automate."""
        row = self.rows[key]
        if row["state"] == "IN_FLIGHT" and now - row["started"] > timeout_s:
            row["state"] = "UNKNOWN"
            self.states.add("UNKNOWN")
        if row["state"] != "UNKNOWN":
            return row["state"], row["result"]
        outcome = psp_lookup(key)                 # GET /charges?idempotency_key=...
        if outcome is None:
            row["state"] = "MANUAL"
        else:
            row.update(state="DONE", result=outcome)
        self.states.add(row["state"])
        return row["state"], row["result"]


req = {"order_id": 9001, "amount_minor": 5000, "currency": "USD"}
key = payment_key(req)
assert key == payment_key(dict(req, attempt=3, client_ts="2026-07-30T10:00:01Z"))

store = PspCallStore()
assert store.begin(op_key(key, "auth"), req) == ("PROCEED", None)
store.finish(op_key(key, "auth"), {"psp_ref": "ch_abc", "state": "AUTHORIZED"})
action, replay = store.begin(op_key(key, "auth"), req)
assert action == "REPLAY" and replay["psp_ref"] == "ch_abc"

# Capture is a DIFFERENT PSP call on the SAME intent. Without the namespace it
# lands on the authorize row and is rejected as "reused with a different body"
# -- a correct capture refused as a client bug.
cap = {"payment_id": "pay_a1", "amount_minor": 5000}
assert store.begin(op_key(key, "capture"), cap) == ("PROCEED", None)
store.finish(op_key(key, "capture"), {"psp_ref": "cap_abc"})
assert store.begin(op_key(key, "refund"), cap)[0] == "PROCEED"

# --- eight retries of one intent, landing at once ---------------------------
charges, guard = [], threading.Lock()
gate = threading.Barrier(8)
hot = PspCallStore()
hot_key = op_key(payment_key(req), "auth")

def attempt():
    gate.wait()
    if hot.begin(hot_key, req)[0] == "PROCEED":
        with guard:
            charges.append(1)                     # the card is charged HERE

ts = [threading.Thread(target=attempt) for _ in range(8)]
for t in ts:
    t.start()
for t in ts:
    t.join()
assert sum(charges) == 1, f"{sum(charges)} of 8 concurrent retries charged the card"

# --- IN_FLIGHT is not a trap: it ages into UNKNOWN, and out of it -----------
stuck = PspCallStore()
k2 = op_key(payment_key(dict(req, order_id=9002)), "auth")
stuck.begin(k2, req, now=0.0)
assert stuck.begin(k2, req)[0] == "RECOVER"
assert stuck.recover(k2, lambda _: None, now=31.0) == ("MANUAL", None)

k3 = op_key(payment_key(dict(req, order_id=9003)), "auth")
stuck.begin(k3, req, now=0.0)
assert stuck.recover(k3, lambda _: {"psp_ref": "ch_x"}, now=31.0)[0] == "DONE"
assert stuck.begin(k3, req) == ("REPLAY", {"psp_ref": "ch_x"})

# Every state the DDL declares is a state some code path actually produces.
assert stuck.states == {"IN_FLIGHT", "UNKNOWN", "DONE", "MANUAL"}

4.4 Five details that get probed

• The claim is one statement, and the eight-thread assertion is the proof. setdefault stands in for INSERT ... ON CONFLICT DO NOTHING: it returns the row that is in the table after the call, so the winner is known by identity and there is no gap between checking and acting. Write it instead as a get followed by an assignment — which is what a SELECT and an INSERT are, with a round trip in the gap — and all eight threads see no row, all eight return PROCEED, and eight charges go to the PSP. This is not a Python artifact; it is the reason R3 says what it says.
• request_hash. The same key arriving with a different body is a client bug, and silently returning the cached result is the wrong kindness: the caller then believes it charged $80 when you charged $50. Return 409 instead. This check is only usable because of R2 — without the operation suffix, a legitimate capture would arrive on the authorize row with a different body and trip it.
• RECOVER is a lookup, not a retry. GET /charges?idempotency_key=... at the PSP. Every serious provider supports this precisely because everyone hits this case. If a provider does not, that is a procurement decision, not an engineering one.
• IN_FLIGHT has an exit, and something owns it. A row that can only ever answer RECOVER is a payment nobody will ever resolve and money nobody will ever account for. The recovery worker sweeps WHERE state = 'IN_FLIGHT' AND started_at < now() - 30s, moves the row to UNKNOWN, and looks it up; a conclusive answer is DONE, an inconclusive one is MANUAL and a case in the Failure modes queue. A state that the schema declares and no code path produces is a state that will be wrong when you finally need it — which is what the last assertion checks, by collecting every state the code actually emitted and comparing that set against the four the psp_calls table allows.
• Key lifetime. The key must outlive the longest possible retry, which is bounded by the authorization expiry of about 7 days, not by an HTTP timeout. Expiring keys after an hour reintroduces the double-charge you just designed away.

5. Deep dive 4 — async settlement, and why state is a machine

A payment cannot be modelled as a boolean, and barely as a status string. Laying the four clocks a single payment runs on side by side shows why.

A state machine is a named list of the states a thing can be in, plus the transitions allowed between them. The discipline it buys you is that code changes state by naming a transition, never by writing a status directly — so an illegal jump is a rejected transition rather than a plausible-looking row.

Authorization, capture, and settlement happen on three different timescales. A fourth clock — the dispute window, the period during which the customer can contest the charge with their bank and have it reversed — runs for months after the other three have finished.

The column to read is the middle one. The spread in it is the argument:

Put the fastest and slowest clocks in the same unit — seconds — and take the ratio. A day is 86,400 seconds, and an authorization answers in about 2:

dispute window, seconds:  120 x 86,400           =  10,368,000
vs a 2-second auth:       10,368,000 / 2         =  5,184,000

A factor of 5.2 million — more than six orders of magnitude — separates the fastest and slowest transitions in one payment’s life. No boolean models that, and neither does status = 'paid'.

What models it is a state machine whose states are the states the money is actually in. In the diagram below, follow the happy path first (CREATED → AUTHORIZED → CAPTURED → SETTLED → CLOSED), then come back for the branches: everything hanging off AUTHORIZING is a payment whose outcome you do not know, and everything after SETTLED is money that can still come back out.

stateDiagram-v2
    [*] --> CREATED
    CREATED --> AUTHORIZING: submit to PSP
    AUTHORIZING --> AUTHORIZED: approved, seconds
    AUTHORIZING --> DECLINED: declined
    AUTHORIZING --> UNKNOWN: timeout or no response
    UNKNOWN --> AUTHORIZED: PSP lookup says charged
    UNKNOWN --> DECLINED: PSP lookup says not charged
    UNKNOWN --> MANUAL: lookup inconclusive
    MANUAL --> AUTHORIZED: operator resolves
    MANUAL --> DECLINED: operator resolves
    AUTHORIZED --> CAPTURED: goods shipped, hours
    AUTHORIZED --> VOIDED: order cancelled
    AUTHORIZED --> EXPIRED: 7 days, no capture
    CAPTURED --> SETTLED: funds land, T plus 2 days
    SETTLED --> REFUNDED: merchant-initiated
    SETTLED --> DISPUTED: issuer chargeback, up to 120 days
    DISPUTED --> SETTLED: dispute won
    DISPUTED --> REVERSED: dispute lost
    SETTLED --> CLOSED: 120-day window elapses
    DECLINED --> [*]
    VOIDED --> [*]
    EXPIRED --> [*]
    REFUNDED --> [*]
    REVERSED --> [*]
    CLOSED --> [*]

Four things this diagram earns you:

UNKNOWN is a real state with a real row, and something moves rows into it. Most candidates draw the happy path and handle timeouts with a retry. Naming UNKNOWN forces the design of Deep dive 3 idempotency across a boundary you cannot roll back to exist, and gives the recovery worker something to query for: psp_calls WHERE state = 'IN_FLIGHT' AND started_at < now() - 30s. The transition out is a lookup, and its two outcomes are the two edges on the diagram — AUTHORIZED/DECLINED when the PSP answers, MANUAL when it cannot. A state on the diagram that no code path enters or leaves is decoration, and the test at the end of §4 is there to keep all four honest.

EXPIRED costs money and has a size. Card authorizations expire in about 7 days. Anything that ships later than that must be re-authorized, and a re-authorization can decline — the card may be cancelled, over its limit, or simply out of funds now.

Assume 2% of orders ship after the window, and 5% of those re-authorizations decline:

late shipments/day:       10,000,000 x 0.02      =  200,000
re-auth declines/day:     200,000 x 0.05         =  10,000

10,000 declines/day is a customer flow — email the customer, let them retry with another card — not an operations queue. Sizing it is what tells you it has to be automated: nobody is hand-working ten thousand cases a day.

CAPTURED is where ledger entries are written; SETTLED is where cash appears. Between those two states the PSP owes you money, which is to say you are its creditor. That is exactly the psp_receivable account from §2.2, and reconciliation checks it: its balance should equal the PSP’s reported pending payout, to the minor unit.

CLOSED is 120 days after settlement. Until then the payment can reverse, so no ledger transaction may be archived and no account may be treated as final. This single fact sets the hot-data retention floor and is why “we keep 30 days hot” is a wrong answer in this domain.

6. Deep dive 5 — reconciliation

The third and last correctness mechanism is the one that looks outward.

Reconciliation compares your own records against someone else’s, item by item, and gives every disagreement a class and an owner. It is the only mechanism that detects a disagreement between your books and the outside world. The invariant of Deep dive 1 double entry derived cannot help with that: your ledger can be perfectly zero-sum and still describe a day that did not happen.

Sizing the job

Each morning the PSP drops a settlement file: a list of every transaction it processed the previous day, each with the provider’s own reference id, the amount, and the status.

How big is that file? Of the 10 M payments a day, only the ones the card network approved appear in it. At a 95% authorization success rate, and about 200 B per row:

rows in the file:         10,000,000 x 0.95      =  9,500,000
file bytes:               9,500,000 x 200        =  1,900,000,000
read seconds at 1 GB/s:   1.9 / 1                =  1.9

1.9 GB, read in under two seconds. Now the match itself, which is a hash join on psp_ref, the provider’s reference id.

A hash join loads one of the two datasets into an in-memory lookup table keyed on the join column — that side is called the build side — and then streams the other side past it, looking each row up as it goes. It is fast as long as the build side fits in memory. So the only question is how big our side is.

Our side is the day’s captures, cut down to the three columns the match needs: a 16 B reference, an 8 B amount, and a 1 B status.

build row, bytes:         16 + 8 + 1             =  25
build side, bytes:        9,500,000 x 25         =  237,500,000

237,500,000 bytes is 238 MB. 238 MB fits in memory, so reconciliation is a single-pass hash join that takes seconds, not an overnight batch window. People assume this job is heavy because it feels important. It is one of the cheapest jobs in the system, which removes every excuse for running it less often than daily.

The five classes of mismatch

Every reference in either dataset lands in exactly one bucket. That totality — every input accounted for, none silently dropped — is what makes the output trustworthy: a row cannot quietly fall out of the process, so “the buckets are empty” really does mean “nothing is wrong”.

The function below is the classifier. ours and theirs are both dictionaries keyed by psp_ref, with (amount, status) as the value. The first loop walks our rows and sorts each into one of four buckets; the second loop picks up anything that exists only on their side.

The last assertion is the proof of totality: the bucket sizes add up to the size of the union of both key sets, so nothing was counted twice and nothing was lost.

def reconcile(ours, theirs):
    """ours / theirs: {psp_ref: (amount_minor, status)}.
    Every ref lands in exactly one bucket -- that totality is the point."""
    out = {"matched": [], "missing_at_psp": [], "missing_locally": [],
           "amount_mismatch": [], "status_mismatch": []}
    for ref, (amount, status) in ours.items():
        if ref not in theirs:
            out["missing_at_psp"].append(ref)
        elif theirs[ref][0] != amount:
            out["amount_mismatch"].append(ref)
        elif theirs[ref][1] != status:
            out["status_mismatch"].append(ref)
        else:
            out["matched"].append(ref)
    for ref in theirs:
        if ref not in ours:
            out["missing_locally"].append(ref)
    return out

ours = {"a": (4825, "CAPTURED"), "b": (1000, "CAPTURED"), "c": (200, "CAPTURED")}
theirs = {"a": (4825, "CAPTURED"), "b": (900, "CAPTURED"), "d": (750, "CAPTURED")}
r = reconcile(ours, theirs)
assert r["matched"] == ["a"]
assert r["amount_mismatch"] == ["b"]
assert r["missing_at_psp"] == ["c"]
assert r["missing_locally"] == ["d"]
assert sum(len(v) for v in r.values()) == len(set(ours) | set(theirs))

Four buckets in the code, plus one the code cannot see (a duplicate reference inside the provider’s own file), make five classes. Each one means something different and gets a different response:

The timing class, which is what separates people

There is a sixth category that is not a mismatch at all, and mistaking it for one is what makes reconciliation alerting useless in practice.

The provider closes each day’s file at a fixed cutoff time. Payments captured in the last moments before that cutoff land in the next day’s file instead. They are not missing; they are late. That is not a break.

How much of it is there? At 100 payments/s, one minute of traffic straddling the cutoff is:

last minute of traffic:   100 x 60               =  6,000

On day 0 you will see roughly 6,000 unmatched rows that are pure timing noise, and they clear themselves the next morning. An alert wired to “unmatched count > 0” therefore fires on 6,000 non-problems a day, gets muted in week one, and is not looked at again on the day it matters.

The fix is to age the unmatched set: carry each unmatched row forward and only call it a break once it has survived several settlement cycles. One cycle is one daily file.

In steady state the day-3 unmatched count is zero. “Zero after aging” is the alertable condition, and getting that threshold right is worth more than any amount of matching cleverness.

Two more things the reconciler owns:

• Fee verification. Recompute the expected fee for each row from the contracted schedule and compare it to what the provider actually charged. Providers do get fee tiers wrong, and the exposure is large — take 0.1% of the $17.5 M/day fee bill computed in Back of envelope:

0.1% of daily fees, $:    17,500,000 x 0.001     =  17,500

At 17,500 x 365 = 6,387,500, that is $6.39 M a year going out on a fee difference nobody is checking.

• The receivable tie-out. A tie-out is a check that two independently computed totals agree. Here, balance(psp_receivable) from The invariant must equal the PSP’s reported pending balance. This is a second, independent check on the same money, and it catches a class the row-by-row match cannot: a systematic error that is consistent on both sides, so that every individual row matches and the total is still wrong.

7. Failure modes

Every way the design breaks, how you would notice, and what happens next — followed by prices on the two things everyone gets wrong about the recovery path: how many retries a checkout can afford, and how many people the leftovers need.

The middle column of the table is the one to argue about. A failure with no detection line is a failure you find out about from a customer:

Retry policy, priced

The reflex answer is “retry with exponential backoff”. Backoff means you wait before retrying; exponential means each wait is roughly double the last, so a struggling service gets progressively less traffic rather than more. Right instinct, wrong budget here.

Price it. Each attempt burns the full 10 s timeout before it gives up, and the waits between attempts are 1 s, 2 s, and 4 s. Four attempts is therefore four timeouts plus three waits:

four attempts, seconds:   10 + 1 + 10 + 2 + 10 + 4 + 10   =  47

47 seconds is not a checkout experience. The customer has closed the tab, or pressed the button again.

So the in-request retry budget is one attempt. After that the payment goes to UNKNOWN, the API returns 202, and retries continue asynchronously — against the lookup endpoint, never the charge endpoint.

Ch 20 owns the delivery semantics of that asynchronous path, and they are worth naming here. The queue guarantees at-least-once delivery, so duplicates will arrive. Combine that with the idempotency key of Deep dive 3 idempotency across a boundary you cannot roll back and you get effectively-once — duplicates still arrive, but the second one changes nothing — at the PSP, which is the only place it matters.

Retries also need full jitter: instead of sleeping exactly the backoff interval, sleep a random amount drawn uniformly from [0, backoff].

Without jitter, a PSP outage synchronizes every one of your workers. They all failed at the same instant, so they all retry at the same instant, forever after. 300 payments/s of synchronized retry is how you turn the provider’s brownout — a period of degraded but partial service — into their outage, and lose the failover you were counting on.

The dead-letter path and the human queue

After the asynchronous retries are exhausted, a payment lands in a dead-letter queue (DLQ): a side queue holding messages the system could not process, kept so they stay visible rather than being discarded. A DLQ nobody drains is a data-loss mechanism with extra steps.

In this domain the drain is a person, so size that person the way you would size any other capacity. Two assumptions go in: 0.01% of payments end up unresolved, and each case takes 3 minutes to work. A full-time equivalent (FTE) is one person working one full shift, taken here as 8 hours:

cases/day:                10,000,000 x 0.0001    =  1,000
minutes/day:              1,000 x 3              =  3,000
hours/day:                3,000 / 60             =  50
FTE at 8 h/shift:         50 / 8                 =  6.25

6.25 people — and that is before shift coverage (someone has to work nights) and before the queue’s own tail (some cases take an hour, not three minutes).

Now drive the unresolved rate down by a factor of ten, from 0.01% to 0.001%, and run the same arithmetic:

cases/day at 0.001%:      10,000,000 x 0.00001   =  100
FTE:                      100 x 3 / 60 / 8       =  0.625

6.25 people become 0.625. That factor of ten is the business case for the automated PSP lookup in Deep dive 3 idempotency across a boundary you cannot roll back, stated in headcount rather than in adjectives.

A human queue is a legitimate design element here, not an admission of defeat. Some cases are genuinely undecidable by software: the lookup comes back inconclusive, the settlement file and the API disagree, a customer disputes a charge the network says succeeded.

So the engineering question is not “how do we eliminate it”. It is the three questions you would ask of any queue:

• What is the arrival rate? (1,000 cases/day.)
• What is the service time? (3 minutes/case.)
• What is the service-level agreement (SLA) — the promised time to resolve a case?

8. Bottlenecks and scaling

What actually runs out first is not a machine you own — and that reframing is the point: capacity planning in a payment system is a procurement activity.

Three of the five rows below resolve to “nothing to do”. The two that do not — the PSP and the human queue — are the two you cannot fix by adding machines:

The bottleneck is a company you do not run. That reframing is the scaling answer: capacity planning here means negotiating rate limits and integrating a second provider, not adding replicas.

One number makes the point concrete. The PSP call has a p99 of 2 s, which is 2,000 ms. A round trip inside your own datacenter is about 0.5 ms. Divide:

external vs internal hop: 2,000 / 0.5            =  4,000

A single external call costs what four thousand internal ones cost. Every design instinct built on 500 us hops is wrong on the other side of that boundary, which is why the external call gets a state machine, a persisted intent, and a human fallback, while the internal calls get none of that.

9. Alternatives rejected

Each row below is a design somebody proposes in every review of this system, and the one-line reason it does not survive contact with the failure modes above.

10. Interviewer pushback

These are the seven questions this design reliably attracts, with the answers that show you have used the design rather than memorized it.

“Isn’t double-entry just accounting ceremony? I have a transfer table with from, to, and amount — same thing, fewer rows.”

It is the same thing for a two-legged, same-currency, fully internal movement, and it stops being the same thing immediately. A capture has three legs: receivable, fee, revenue. An FX conversion has two currencies. A chargeback has a fee that lands in a different account than the reversal. The from/to shape forces you to encode those as multiple rows anyway, at which point you have double-entry with a worse schema and no zero-sum constraint. The constraint is the product; the row layout is incidental.

“You said the invariant catches bugs. Name one it does not catch.”

Duplication. A transaction applied twice sums to zero both times and the ledger is perfectly consistent — it just describes a world where the customer bought two things. That is why idempotency is a separate mechanism with a separate key, and why reconciliation against the provider is a third check. The invariant validates internal shape, idempotency validates write-once, reconciliation validates against the outside world. Three mechanisms, three failure classes, and no one of them subsumes another.

“The PSP timed out. Why not just retry? Their API is idempotent.”

If they honour my idempotency key, retrying is safe and I do it. The design has to work when they do not, or when the retry is issued by a process that lost the key — a queue consumer replaying an at-least-once message after the original worker died, say. The rule that makes both cases safe is that the key is committed before the call, so the recovery path always has something to look the charge up by. Retry is the optimization; lookup is the correctness argument. And the IN_FLIGHT row is not where it ends: a sweeper picks up anything older than the call timeout, moves it to UNKNOWN, and works it with a lookup — conclusive answers close it, inconclusive ones become MANUAL and a case in the human queue. A row that can only ever say “unknown” is money with no owner.

“Where does the idempotency key come from? Who generates it, and what is in it?”

It is a SHA-256 over the canonical request body with the per-attempt fields stripped — no timestamp, no attempt counter, no trace id — so a retry of the same intent produces the same key by construction instead of by the client remembering to reuse one. A uuid4() per call is the failure mode, not the design: it makes every retry a distinct intent, which is exactly what the key exists to prevent. Then I suffix the operation, key:auth, key:capture, key:refund, because one payment makes several PSP calls and my psp_calls row is keyed on that string — without the suffix the capture collides with the authorize row and my own hash check rejects it as a client bug. Same four rules as the booking write in ch 23; I do not want two idempotency designs in one company.

“Why is capture the point where you write ledger entries, and not authorization?”

Because no money moves at authorization. An auth is an issuer’s promise to honour a capture for about seven days, and roughly 2% of my orders ship after that window and need re-authorization, with about 5% of those declining. If I booked revenue at auth, every expired auth would leave revenue in the books with no cash behind it, and my receivable would never tie out to the provider’s pending balance. Auth gets a hold record with an expiry; capture gets a ledger transaction.

“Your reconciler found 6,000 unmatched rows this morning. Do you page?”

No, and paging on that number is the mistake. At 100 payments/s, 100 x 60 = 6,000 is exactly one minute of traffic straddling the provider’s cutoff, and it clears itself in tomorrow’s file. I alert on the set that has survived three settlement cycles, which in steady state is zero, so one item there is a real signal. The threshold is derived from the traffic rate and the cutoff, not chosen.

“Six FTE for exceptions sounds like a failure of engineering.”

It is a measurement, and it is the one that funds the engineering. At a 0.01% exception rate the queue needs 6.25 people; at 0.001% it needs 0.625. That factor of ten is what the automated lookup buys, expressed in a unit the business acts on. And the residual is irreducible — some cases are genuinely undecidable by software, like an inconclusive lookup on a charge the network says succeeded. My job is to make the queue small, bounded, and monitored, not to pretend it is zero.

Assumption ledger

Every number above rests on something assumed. This table separates the assumptions you should simply state and move past from the ones worth asking about, and marks which ones actually change the design if they turn out differently. Load-bearing means the architecture changes if the assumption is wrong; if it is not load-bearing, being wrong only moves a number.

Cheat sheet

Next: 28 — Digital Wallet takes the same ledger inside the building, where there is no third party to blame and the hard part becomes 10,000 writes/s landing on one account.