How to design a payment system

A shopper clicks “pay” on a checkout page. In this lesson, we’ll build everything on the far side of that click: the flow that actually moves their money for an online store. Charging a credit card is one HTTP call; knowing afterwards that it charged the right amount exactly once is the hard part, and closing that gap is the entire design.

By the end you’ll be able to reason about the three mechanisms that do the work, each of which catches a class of failure the others cannot:

1. Double-entry bookkeeping (Deep dive 1) records money so that a half-finished movement is impossible to write down in the first place.
2. Idempotency (Deep dive 3) makes a retried charge harmless.
3. Reconciliation (Deep dive 5) compares the books line by line against the card processor’s own records, once a day.

No one mechanism covers another’s failures. That is why “just retry the payment” is the most expensive instinct in this design: a blind retry charges the card again.

Input and output. The input is one HTTP request from the checkout page, carrying:

• an order id, an amount, a currency, and a token standing in for the customer’s card (a meaningless string the PSP exchanges for the real card number, so the card number never touches your servers);
• an idempotency key: a caller-supplied string that says “this is the same request I sent a moment ago, not a new one.”

The output is three things:

• a payment record in state AUTHORIZED, DECLINED, or UNKNOWN;
• a set of immutable ledger entries recording where the money went;
• and, when the system genuinely cannot tell what happened at the card network, a case in a queue that a human works. That third output is designed, not a failure of the design; it is sized in Failure modes.

Vocabulary. Four terms, all used from here on:

• Payment service provider (PSP): the outside company (Stripe, Adyen, Braintree) that talks to the card networks on your behalf. Its behaviour drives every hard decision below.
• Ledger: the append-only list of rows that is the system’s record of every movement of money. It is the truth, and nothing else is.
• Minor units: whole pennies. $50.00 is written 5000, never 50.00.
• Double-charge: the catastrophe this lesson exists to prevent, one customer intent that results in two real charges on one card.

A payment system is the rare design where throughput is trivial and correctness is everything. The workload below peaks at 300 requests/s, which one machine handles without effort. The whole difficulty is that some of those requests move money through a company you do not control, on a timescale you do not control, with no rollback.

Three numbers carry the answer, each derived in the section beside it:

Three ideas from other lessons appear here, each restated in place so you never have to leave this page:

• Idempotency: doing something twice has the same effect as doing it once. Its four rules (hash the request to get the key, namespace the key per operation, claim it with a single atomic statement, put it on every mutating endpoint) are derived in the hotel reservation chapter and restated with their reasoning in Deep dive 3.
• Isolation levels: the database settings that decide which concurrency surprises a transaction is allowed to see. The full table is in the database internals chapter.
• Delivery semantics: what a message queue promises about duplicates. The one that matters here is at-least-once: a message may arrive more than once, but never zero times. See the message queue chapter.

Framing: what decision, and what breaks

The whole design turns on one decision: where the truth about money lives, and what makes a lie detectable. Everything else (which provider, which queue, how many copies of the database) is downstream of that.

What breaks is not the happy path but the three-second window where your process has told the PSP to charge a card and has not yet heard back. In that window the system genuinely does not know whether the customer has been charged, and no consistency protocol can tell it. No amount of agreement between machines helps, because the other participant is a company that will not join your transaction: it will not accept an instruction to commit or abort its side along with yours.

So the design goal is: every unknown becomes a detected unknown, and every detected unknown has an owner. A payment system that silently loses $50 is worse than one that stops and shouts, because the shout is a page and the loss is a regulator six months later.

Requirements

Four verbs recur throughout:

• Authorize: ask the customer’s bank to set aside the money and promise to hand it over later. No money has moved yet.
• Capture: claim that promised money. This is when it actually starts moving.
• Void: cancel an authorization before capturing it.
• Refund: send captured money back.

A fifth term is a noun: settlement is the PSP wiring you the accumulated money a day or two later.

Functional

• Authorize, capture, void, and refund a card payment through one or more PSPs.
• Maintain a ledger that answers “what is the balance of account X at time T” exactly.
• Reconcile daily against the PSP’s settlement file: line up our own records against the provider’s list of what it actually processed, and give every disagreement a name and an owner.
• Expose payment state to the order service, both by webhook (the PSP calls a URL of ours when something changes) and by polling (we ask, on a timer).

Non-functional. p99 is the 99th percentile: the latency 99 out of 100 requests come in under, so it describes the slow tail, not the typical case. RPO is recovery point objective, the amount of recent work you are willing to lose if a machine dies; RPO 0 means none, which forces every ledger write to be copied to a second machine before it is acknowledged.

99.99% availability allows only 4.32 minutes of downtime a month (0.01% of ~43,200 minutes).

But our own availability is capped by the PSP’s. If the provider is up 99.9% of the time, a payment cannot succeed more often than that, however perfect our code is. The chain is only as available as its least available link, and that link is not ours.

The fix is a second provider to fail over to. If A and B each fail 0.1% of the time independently, both are down at the same instant only 0.001 x 0.001 = one in a million, so availability is 1 - 0.000001 = 99.9999%. Two providers take 99.9% to 99.9999%: four extra nines from an integration, not from an architecture. That is the entire argument for multi-PSP routing.

“Independently” carries the argument. If both providers sit behind the same upstream card network and that network has the outage, the failures are correlated and the multiplication does not hold.

Back-of-envelope

Four numbers decide the architecture: requests per second, bytes, money at stake when the accounting is wrong, and people needed for exceptions.

Assume a large e-commerce checkout: 10 M payments/day, $50.00 average order (5,000 minor units). That is ~100 payments/s average, ~300/s at peak (a 3x day/night multiplier), and $500 M/day of GMV: gross merchandise value, the total customers spend before fees.

At a typical 2.9% + $0.30 fee, each $50 order costs 145 + 30 = 175 minor units, so ~$17.5 M/day in fees across 10 M payments. A 0.1% error in fee accounting is $17,500 a day walking out unnoticed, which is why Deep dive 5 exists.

How much storage

A single ledger entry is ~77 B of data. That is not what it costs on disk. Round to 80 B (databases pad rows to convenient boundaries) and multiply by 3 for everything the database stores alongside your columns:

• a per-row header (the storage engine’s bookkeeping about visibility and length);
• the primary key index;
• two secondary indexes: extra sorted copies of a few columns that let a query find rows by something other than the primary key. Ours are on (account_id, created_at) and transaction_id; both read paths need them.

So call it 240 B/entry on disk. A payment produces four entries (receivable, fee, revenue, one settlement leg), so at 10 M payments/day that is 40 M entries = 9.6 GB/day. Over 7 years with 3 replicas, ~74 TB, three commodity machines’ worth of disk. Storage is not the constraint, QPS is not the constraint, latency is not the constraint. The numbers fit on one box; the interesting part is that a third party holds half the state.

The one number that sizes a component

One figure drives a real design choice: how much memory the open authorizations need.

An authorization stays open from approval until you capture (goods ship). At a 4-hour median lag and 100 payments/s, the working set is ~100 x 14,400 = 1.44 M rows, about 346 MB, small enough for RAM on any machine. So “which authorizations are still open” is an in-memory question, and the sweep that expires stale ones is cheap enough to run constantly.

What being wrong costs

This number funds the rest of the lesson. At a 1-in-10,000 double-charge rate: 1,000 double charges/day × $50 = $50,000/day, or $18.25 M/year. That is the budget every mechanism in Deep dive 3 is spent against, and the answer to anyone who calls the idempotency work over-engineering.

API sketch

Three commitments in this interface are deliberately hard to miss: money is an integer, “I do not know” is a legal answer, and every endpoint that changes something takes an idempotency key.

The top half is one request and the four responses it can produce; the bottom half is the rest of the endpoints, one line each. The arrows are annotations, not part of the wire format.

POST /v1/payments
Idempotency-Key: 8f14e45f...         <- REQUIRED. sha256 of the canonical request
{ "order_id": "9001", "amount_minor": 5000, "currency": "USD",
  "method_token": "tok_live_...", "capture": "manual" }

201 { "payment_id": "pay_a1", "state": "AUTHORIZED", "psp_ref": "ch_..." }
202 { "payment_id": "pay_a1", "state": "AUTHORIZING" }   <- outcome unknown, poll
409 { "error": "idempotency_key_reused_with_different_body" }
402 { "error": "card_declined", "decline_code": "insufficient_funds" }

POST /v1/payments/{id}/capture   Idempotency-Key: ...   { "amount_minor": 5000 }
POST /v1/payments/{id}/void      Idempotency-Key: ...
POST /v1/payments/{id}/refund    Idempotency-Key: ...   { "amount_minor": 5000,
                                                          "reason": "..." }
GET  /v1/payments/{id}
GET  /v1/accounts/{id}/entries?from=...&to=...&cursor=...

amount_minor is an integer, and the field name says so. No endpoint accepts a decimal string a client might have produced with a float: the computer’s fast but approximate representation of a fractional number, whose failures are Deep dive 2.

202 is a first-class response, not an error. HTTP 202 means “accepted, outcome not yet determined”. “I do not yet know whether the card was charged” is a legitimate outcome, and an API with no way to say it forces the caller to guess, which in practice means the caller retries and charges the card again.

Idempotency-Key is on every mutating endpoint, not just create. A mutating endpoint changes state instead of merely reading it. A create call that double-charges is the bug everyone designs against; a retried refund that refunds twice moves the same money the other way, out of the same books, and nothing in a create-only contract stops it.

The key is a canonical hash of the request: a fixed-length fingerprint computed after sorting the fields into one agreed order, so the same intent always produces the same fingerprint. A retry of the same refund therefore produces the same key by construction, not by the client remembering to reuse one.

Data model

Four tables carry everything:

• accounts: what accounts exist;
• ledger_transactions: what money movements happened;
• ledger_entries: the individual signed lines of each movement;
• psp_calls: a record of every call made to the PSP.

The accounting vocabulary is small. An asset is something you own (cash, money the PSP owes you). A liability is something you owe (money held on a customer’s behalf). Revenue is money earned; an expense is money spent: here, the PSP’s fees.

CREATE TABLE accounts (
  account_id   BIGINT PRIMARY KEY,
  kind         TEXT NOT NULL,     -- asset | liability | revenue | expense
  currency     CHAR(3) NOT NULL
);

CREATE TABLE ledger_transactions (
  transaction_id  UUID PRIMARY KEY,
  kind            TEXT NOT NULL,  -- CAPTURE | SETTLE | REFUND | CHARGEBACK | ADJUST
  payment_id      UUID,
  created_at      TIMESTAMPTZ NOT NULL
);

CREATE TABLE ledger_entries (          -- APPEND ONLY. No UPDATE, no DELETE.
  entry_id        UUID PRIMARY KEY,
  transaction_id  UUID NOT NULL REFERENCES ledger_transactions,
  account_id      BIGINT NOT NULL REFERENCES accounts,
  amount_minor    BIGINT NOT NULL CHECK (amount_minor <> 0),
  currency        CHAR(3) NOT NULL,
  created_at      TIMESTAMPTZ NOT NULL
);
CREATE INDEX ON ledger_entries (account_id, created_at);
CREATE INDEX ON ledger_entries (transaction_id);

CREATE TABLE psp_calls (               -- written BEFORE the network call
  idempotency_key TEXT PRIMARY KEY,  -- "<client key>:<operation>", rule R2
  payment_id      UUID NOT NULL,
  request_hash    BYTEA NOT NULL,
  state           TEXT NOT NULL,   -- IN_FLIGHT | DONE | UNKNOWN | MANUAL
  started_at      TIMESTAMPTZ NOT NULL,   -- IN_FLIGHT ages out of it, section 4
  psp_ref         TEXT,
  response        JSONB
);
CREATE INDEX ON psp_calls (state, started_at);   -- the recovery worker's query

amount_minor is signed. A debit (money into an account) is positive; a credit (money out) is negative. An account’s balance is the sum of its entries: there is no balance column anywhere. That omission is deliberate and is the subject of the next section.

psp_calls must exist before any call to the provider. Its state column takes four values, all of which recur below:

The index on (state, started_at) lets a background worker cheaply ask “which rows have been IN_FLIGHT longer than the call timeout” without scanning the table.

High-level architecture

The diagram is the whole system on one page: three journeys that share a database: the synchronous charge path, the asynchronous notification path, and the daily comparison.

flowchart TD
    C["Checkout service"] -->|"POST /payments + Idempotency-Key"| API["Payment API"]
    API --> IDEM["psp_calls row<br/>state IN_FLIGHT<br/>committed BEFORE the call"]
    IDEM --> RT["PSP router<br/>primary and failover"]
    RT -->|"HTTPS, p99 2 s"| PSP1["PSP A"]
    RT -->|"failover"| PSP2["PSP B"]
    RT --> LED["Ledger writer<br/>append-only, zero-sum txns"]
    LED --> DB[("Postgres<br/>ledger and accounts<br/>sync replica, RPO 0")]
    PSP1 -->|"webhook"| WH["Webhook ingest<br/>verify signature, dedupe by psp_ref"]
    WH --> Q["Message queue"]
    Q --> SM["Payment state machine worker"]
    SM --> LED
    SM --> DLQ["Dead letter"]
    DLQ --> OPS["Human queue"]
    PSP1 -->|"daily settlement file"| REC["Reconciler"]
    DB --> REC
    REC --> BRK["Break records<br/>five classes"]
    BRK --> OPS

Journey 1: the charge, going out. The checkout service sends POST /payments with an idempotency key. The payment API writes the psp_calls row and commits it with state IN_FLIGHT before anything leaves the building. The PSP router picks a provider, primary or failover, and makes the actual charge, the one hop with an HTTPS p99 of 2 s, four thousand times the cost of an internal call. The ledger writer turns the outcome into append-only, zero-sum transactions in the Postgres ledger, kept honest by a synchronous replica so RPO 0 holds.

Journey 2: the notification, coming back. Later the PSP calls us over the webhook. Webhook ingest verifies the provider’s cryptographic signature (so a forged callback is rejected), deduplicates on the provider’s reference id (psp_ref, so a repeated callback is a no-op), and queues the event. The payment state machine worker applies it as a transition (Deep dive 4) and writes any resulting ledger entries. Anything it still cannot process after its retries goes to a dead letter store and from there to the human queue.

Journey 3: the daily comparison. Once a day the provider drops a settlement file. The reconciler joins it against our ledger and emits break records (one row the two sides disagree about) in the five classes of Deep dive 5. Those also feed the human queue.

Four boxes are load-bearing and the rest is plumbing: the idempotency row written before the call, the append-only ledger, the reconciler, and the human queue. The fourth is the one most designs omit.

Deep dive 1 — double-entry, derived

Double-entry is the six-hundred-year-old accounting rule that every movement of money is recorded as two or more signed lines that add up to zero: money leaving one account is the same event as money arriving in another, written once, in two halves. The whole system rests on it, so it is worth deriving from the naive design and the specific bug that kills it, which is also how you learn what the rule catches, what it provably cannot catch, and what it costs to verify.

What a single balance column loses

The obvious model is one row per account with a balance column, and a transfer is two UPDATEs. Here is the failure, with no distributed systems in it at all:

# The single-column model: two updates, and no way to tell they both landed.
accounts = {"customer": 5000, "revenue": 0}

def transfer_single_column(src, dst, amount, crash=False):
    accounts[src] -= amount
    if crash:
        raise RuntimeError("process died between the two updates")
    accounts[dst] += amount

try:
    transfer_single_column("customer", "revenue", 5000, crash=True)
except RuntimeError:
    pass

# Both rows are individually plausible. No constraint is violated, no CHECK
# fires. Zero is a completely normal thing for an account balance to be.
assert accounts["customer"] == 0
assert accounts["revenue"] == 0

5,000 minor units left the system and no query can find them. The corruption is not in any row: both hold a number an account is entitled to hold. It is in the relationship between two rows, and the schema records that relationship nowhere.

The obvious objection: wrap both updates in one database transaction. That fixes this instance and nothing else, because in a real payment system the two sides of a movement are frequently not available to one transaction. One side may be:

• a card network, which will not join your transaction at all;
• another shard: one of several machines the data has been split across (the digital wallet chapter);
• a batch file that does not arrive until tomorrow.

The single-column model has no representation for “half of this movement has happened”, so it represents that state as a normal balance.

The invariant

What replaces the balance column is an invariant: a statement that must be true of the data at all times. Its value is that checking it is a single query, not a judgement call.

Double-entry replaces the balance column with an append-only list of signed entries (rows are only ever added, never edited or deleted) grouped into transactions, under one rule:

Every transaction’s entries sum to zero. Therefore the sum of all entries in the ledger is zero, at every instant, forever.

Both halves matter. The per-transaction rule is enforceable at write time, as a constraint the database refuses to violate. The global rule follows from it, and it is what makes the books auditable: checking them is one aggregate over one table, no joins, no interpretation.

The code below is the whole scheme in fifteen lines. post refuses anything that does not balance; balance sums an account’s entries instead of reading a column; global_invariant is the audit query:

LEDGER = []   # append-only: (transaction_id, account, signed_minor_units)

def post(txn_id, entries):
    """Refuses to write unless the transaction is zero-sum, so a half-written
    money movement cannot be represented, let alone persisted."""
    if sum(amount for _, amount in entries) != 0:
        raise ValueError("transaction does not balance")
    for account, amount in entries:
        LEDGER.append((txn_id, account, amount))

def balance(account):
    return sum(a for _, acct, a in LEDGER if acct == account)

def global_invariant():
    return sum(a for _, _, a in LEDGER)

post("t1", [("customer_payable", -5000),
            ("psp_receivable", 4825),
            ("psp_fee_expense", 175)])
assert balance("psp_receivable") == 4825
assert global_invariant() == 0

# The same crash as above, now representable and therefore detectable.
LEDGER.append(("t2", "customer_payable", -5000))   # a half-written movement
assert global_invariant() == -5000

Here is that same $50.00 order captured, written the way it appears in the books. A receivable is money someone owes you but has not handed over: the PSP holding your $48.25. A payable is money you owe someone else: the $50.00 of goods you now owe the customer. The customer’s 5,000 splits into the 175 the PSP keeps and the 5,000 - 175 = 4,825 it will pay you; debits positive, credits negative:

txn CAPTURE order 9001, 5,000 minor units, PSP fee 2.9% plus 30
  1101  psp_receivable      asset      +4825
  5010  psp_fee_expense     expense     +175
  2101  customer_payable    liability  -5000
                                       ------
  sum                                       0

Settlement two days later moves the receivable into cash and touches nothing else, same 4,825, still summing to zero:

txn SETTLE batch 2026-07-29
  1001  bank_cash           asset      +4825
  1101  psp_receivable      asset      -4825
                                       ------
  sum                                       0

What is not here is the authorization. An authorization is a promise from the issuer (the bank that gave the customer the card), and a promise is not a movement of money, so it produces no ledger entries: only a hold record with an expiry. Booking the authorization into the ledger is the most common modelling error here, and it corrupts the books the moment an authorization expires unused.

What the invariant catches that a balance column does not

Five bugs it catches, and one large class it structurally cannot. The row to slow down on is the fifth, because it is the one where double-entry loses:

Double-entry catches asymmetry, not duplication. A transaction applied twice is internally consistent and sums to zero perfectly; it simply describes a world in which the customer bought two things.

Per currency is not a footnote. A single global SUM over a multi-currency ledger is meaningless: 100 JPY and 100 USD are different quantities. So the real invariant is SUM(amount_minor) = 0 GROUP BY currency, and every transaction is single-currency. A foreign-exchange (FX) conversion is two transactions joined by an FX position account, never one transaction with two currencies in it. The position account ends up short one currency and long the other, so each transaction still balances within its own currency and the per-currency invariant survives.

What checking the invariant costs

An invariant you cannot afford to check is a wish. The check is a full aggregate: one pass that reads every row. Reading the 24,528 GB of single-copy ledger at 1 GB/s (the estimation chapter) takes ~24,528 s ≈ 6.8 hours: a quarterly exercise, not a monitor. A drift introduced at 09:00 would surface after lunch, having been replicated and paid out against in the meantime.

The fix uses the one property the ledger has that ordinary data does not: the past does not change. Because entries are append-only, yesterday’s sum is a constant. So store the running total at each day boundary (that stored total is a checkpoint) and check only today’s rows against it. Splitting a table into one physical chunk per day is partitioning; the chunk being written now is the open partition, which the storage arithmetic put at 9.6 GB. Reading 9.6 GB at 1 GB/s is ~9.6 s.

A 9.6-second query, run every five minutes, means ledger drift is detected within five minutes of being introduced. The full-history recompute still runs monthly, but its job is different: it validates the checkpoints themselves. Without it, a corrupted checkpoint would make every 5-minute check agree with a lie.

Four layers enforce the invariant, each catching what the layer before it missed:

1. CHECK (amount_minor <> 0) plus a per-transaction zero-sum constraint: enforced at write time, cheap, catches the ordinary application bug.
2. REVOKE UPDATE, DELETE ON ledger_entries: REVOKE takes a permission away, so “correct it by editing the row” is impossible, not merely discouraged. Nobody, including the application, holds the privilege.
3. The 5-minute partition sum: catches whatever got past 1 and 2.
4. Monthly full recompute against the checkpoints: catches a corrupted checkpoint, the one thing level 3 trusts blindly.

Deep dive 2 — money is never a float

Keeping cents from evaporating is a type discipline that compresses to three rules: store whole pennies as integers, use an exact decimal type only for genuine fractions, and never divide money without handing out the leftover.

A float (floating-point number) is how computers usually store fractional values: a fixed number of binary digits plus an exponent. It is fast, and wrong in exactly the way that matters here. Binary fractions cannot represent 0.1, 0.2, or 0.3 exactly, any more than decimal digits can represent one third exactly. 0.1 in a float is a nearby number, so arithmetic on floats is arithmetic on nearby numbers, and the error survives into whatever you store.

The block below has four parts: the failure, the integer fix, the Decimal fix for genuine fractions, and division: where money actually goes missing. Every line is an assertion, so the block passing is the claim:

from decimal import Decimal, ROUND_HALF_UP

# The failure, stated exactly.
assert 0.1 + 0.2 != 0.3
assert repr(0.1 + 0.2) == "0.30000000000000004"

# It accumulates. Ten additions of a dime do not make a dollar.
total = 0.0
for _ in range(10):
    total += 0.1
assert total != 1.0

# Fix A -- integer minor units. Exact, fast, and what wire formats use.
def cents(dollars: str) -> int:
    return int(Decimal(dollars).scaleb(2).to_integral_value(ROUND_HALF_UP))

assert cents("0.10") + cents("0.20") == cents("0.30") == 30

# Fix B -- Decimal, for intermediates that genuinely need more than two
# places: FX rates, interest accrual, per-unit prices.
assert Decimal("0.1") + Decimal("0.2") == Decimal("0.3")

# Division is where money is actually lost, and integers do not save you.
# 1,000 minor units split three ways is 333 each, and one unit vanishes.
def split(total_minor: int, n: int) -> list:
    base, remainder = divmod(total_minor, n)
    return [base + (1 if i < remainder else 0) for i in range(n)]

parts = split(1000, 3)
assert parts == [334, 333, 333]
assert sum(parts) == 1000        # the invariant that makes the split safe

Three rules follow:

• Store and transmit integer minor units. amount_minor BIGINT. A 64-bit signed integer holds 9.22e18 minor units, larger than any figure you will transact by many orders of magnitude.
• Use Decimal only where a fraction of a minor unit is genuinely needed, and round to minor units before anything is posted. Decimal is an exact base-ten number type (it stores the digits you wrote, so 0.1 is really one tenth) at the cost of being slower than a float. A rate is a Decimal; a ledger entry is an int.
• Never divide without redistributing the remainder. Compute the base, then hand the remainder out one unit at a time so the parts sum to the whole. The assert sum(parts) == 1000 is not a test, it is the design.

The connection to Deep dive 1 is worth making explicit. If someone writes the naive split anyway, the entries come to 333 + 333 + 333 = 999 against a stated 1,000, the sum is not zero, and the zero-sum constraint rejects the write. The bad transaction never lands, instead of drifting a unit at a time into the books. One failure caught twice.

One last trap: declaring the column DECIMAL/NUMERIC is not enough on its own. The value passes through JSON, a client library, and application code on its way to that column, and any hop can put it in a double and hand back a rounded number. The type discipline has to hold end to end or it holds nowhere, which is why the API field is named amount_minor and typed as an integer.

Deep dive 3 — idempotency across a boundary you cannot roll back

This is the problem the lesson exists for: making a retried charge harmless when the thing being retried happens at another company. The answer is a small state machine.

The four rules, restated

Four rules make a key work, the same four whether you are booking a hotel room or charging a card. The hotel reservation chapter derives them with no third party; here they are restated so you need not go look:

• R1: the key is a hash of the request, not a random number. Compute it from the request’s meaningful fields, in a canonical order, leaving out anything that changes per attempt.
• R2: namespace the key per operation. Append the operation name so different operations on one intent get different keys.
• R3: claim the key with a single atomic statement. One statement both inserts the key and tells you whether you were the one who inserted it.
• R4: put the key on every call that changes something, not just create.

What each rule turns into once the operation being retried is a card charge:

Two words in R3 are worth spelling out. A race is when two copies of the same code run at once and the outcome depends on which gets there first. A round trip is one request to another machine plus its reply: about half a millisecond for a database on the same network, and an eternity when a charge can slip into it.

What changes when the other side is another company

The PSP call cannot be rolled back, and it happens in the middle of your transaction.

A hotel booking is a row you own: change your mind, issue ROLLBACK, and the database throws your changes away. A card charge is an effect at Visa: there is no ROLLBACK for it, and no protocol lets you learn its outcome atomically with your own commit (“atomically” meaning as one indivisible step that either wholly happens or wholly does not).

That asymmetry forces one rule:

The idempotency key is generated before the external call and committed with the record of that call, in its own transaction, before a single byte goes over the wire.

The argument is a two-case proof. Consider a crash at the worst moment: after the charge has gone out, before you have written anything down.

Key written after the call. The crash leaves no evidence a call was ever made. Recovery finds a payment with no PSP record and does the only thing it can: call again. The customer is charged twice.

Key written before the call. The crash leaves an IN_FLIGHT row. Recovery finds a call whose outcome is unknown: a different, recoverable situation, because the row carries the key you need to go ask what happened.

The mechanism, in code

The block below is that mechanism in runnable form. payment_key is the R1 derivation; op_key is the R2 namespacing; PspCallStore.begin is the atomic claim of R3 with its three answers: PROCEED (you own this call, make it), REPLAY (already done, here is the stored result), RECOVER (someone else started it and we do not know how it ended); PspCallStore.recover gets a stuck row out of IN_FLIGHT by asking the PSP, not charging again.

The assertions start eight threads at the same instant to show that the one-statement claim admits exactly one charge, then walk a stuck row to a human case and, separately, to a resolved one:

import hashlib, json, threading, time

VOLATILE = {"attempt", "retry_count", "client_ts", "requested_at", "trace_id"}

def payment_key(request: dict) -> str:
    """R1: a canonical hash of the intent, per-attempt fields excluded."""
    canonical = {k: v for k, v in sorted(request.items()) if k not in VOLATILE}
    return hashlib.sha256(
        json.dumps(canonical, sort_keys=True, separators=(",", ":")).encode()
    ).hexdigest()

def op_key(key: str, operation: str) -> str:
    """R2: one psp_calls row per (intent, operation)."""
    return key + ":" + operation


class PspCallStore:
    """One row per (intent, operation), committed BEFORE the network call, so a
    crash always leaves evidence that a call may be in flight."""

    LATENCY = 0.002                     # one statement's round trip to the DB

    def __init__(self):
        self.rows = {}
        self.states = set()             # every state this code has ever produced

    def begin(self, key, request, now=None):
        h = hashlib.sha256(json.dumps(request, sort_keys=True).encode()).hexdigest()
        fresh = {"state": "IN_FLIGHT", "hash": h, "result": None,
                 "started": time.time() if now is None else now}
        time.sleep(self.LATENCY)
        # R3. ONE statement claims the key:
        #   INSERT INTO psp_calls (...) VALUES (...) ON CONFLICT DO NOTHING
        # `setdefault` is that statement -- it returns the row that is in the
        # table after the call, so the winner is known by identity and there is
        # no gap between the check and the act.
        row = self.rows.setdefault(key, fresh)
        self.states.add(row["state"])
        if row is fresh:
            return "PROCEED", None
        if row["hash"] != h:
            raise ValueError("idempotency key reused with a different body")
        if row["state"] == "DONE":
            return "REPLAY", row["result"]
        return "RECOVER", None          # a prior attempt's outcome is unknown

    def finish(self, key, result):
        self.rows[key].update(state="DONE", result=result)
        self.states.add("DONE")

    def recover(self, key, psp_lookup, now, timeout_s=30):
        """IN_FLIGHT is not a resting state. Past the call timeout the row moves
        to UNKNOWN, and UNKNOWN is worked by a LOOKUP -- never by re-sending the
        charge. An inconclusive lookup is MANUAL, which is the human queue of
        section 7 and the only exit this design refuses to automate."""
        row = self.rows[key]
        if row["state"] == "IN_FLIGHT" and now - row["started"] > timeout_s:
            row["state"] = "UNKNOWN"
            self.states.add("UNKNOWN")
        if row["state"] != "UNKNOWN":
            return row["state"], row["result"]
        outcome = psp_lookup(key)                 # GET /charges?idempotency_key=...
        if outcome is None:
            row["state"] = "MANUAL"
        else:
            row.update(state="DONE", result=outcome)
        self.states.add(row["state"])
        return row["state"], row["result"]


req = {"order_id": 9001, "amount_minor": 5000, "currency": "USD"}
key = payment_key(req)
assert key == payment_key(dict(req, attempt=3, client_ts="2026-07-30T10:00:01Z"))

store = PspCallStore()
assert store.begin(op_key(key, "auth"), req) == ("PROCEED", None)
store.finish(op_key(key, "auth"), {"psp_ref": "ch_abc", "state": "AUTHORIZED"})
action, replay = store.begin(op_key(key, "auth"), req)
assert action == "REPLAY" and replay["psp_ref"] == "ch_abc"

# Capture is a DIFFERENT PSP call on the SAME intent. Without the namespace it
# lands on the authorize row and is rejected as "reused with a different body"
# -- a correct capture refused as a client bug.
cap = {"payment_id": "pay_a1", "amount_minor": 5000}
assert store.begin(op_key(key, "capture"), cap) == ("PROCEED", None)
store.finish(op_key(key, "capture"), {"psp_ref": "cap_abc"})
assert store.begin(op_key(key, "refund"), cap)[0] == "PROCEED"

# --- eight retries of one intent, landing at once ---------------------------
charges, guard = [], threading.Lock()
gate = threading.Barrier(8)
hot = PspCallStore()
hot_key = op_key(payment_key(req), "auth")

def attempt():
    gate.wait()
    if hot.begin(hot_key, req)[0] == "PROCEED":
        with guard:
            charges.append(1)                     # the card is charged HERE

ts = [threading.Thread(target=attempt) for _ in range(8)]
for t in ts:
    t.start()
for t in ts:
    t.join()
assert sum(charges) == 1, f"{sum(charges)} of 8 concurrent retries charged the card"

# --- IN_FLIGHT is not a trap: it ages into UNKNOWN, and out of it -----------
stuck = PspCallStore()
k2 = op_key(payment_key(dict(req, order_id=9002)), "auth")
stuck.begin(k2, req, now=0.0)
assert stuck.begin(k2, req)[0] == "RECOVER"
assert stuck.recover(k2, lambda _: None, now=31.0) == ("MANUAL", None)

k3 = op_key(payment_key(dict(req, order_id=9003)), "auth")
stuck.begin(k3, req, now=0.0)
assert stuck.recover(k3, lambda _: {"psp_ref": "ch_x"}, now=31.0)[0] == "DONE"
assert stuck.begin(k3, req) == ("REPLAY", {"psp_ref": "ch_x"})

# Every state the DDL declares is a state some code path actually produces.
assert stuck.states == {"IN_FLIGHT", "UNKNOWN", "DONE", "MANUAL"}

Five details worth pinning down

• The claim is one statement, and the eight-thread assertion is the proof. setdefault stands in for INSERT ... ON CONFLICT DO NOTHING: it returns the row in the table after the call, so the winner is known by identity and there is no gap between checking and acting. Write it instead as a get followed by an assignment (a SELECT and an INSERT with a round trip in the gap) and all eight threads see no row, all eight return PROCEED, and eight charges go to the PSP. This is not a Python artifact; it is the reason R3 says what it says.
• request_hash. The same key arriving with a different body is a client bug, and silently returning the cached result is the wrong kindness: the caller then believes it charged $80 when you charged $50. Return 409 instead. This check is only usable because of R2: without the operation suffix, a legitimate capture would arrive on the authorize row with a different body and trip it.
• RECOVER is a lookup, not a retry. GET /charges?idempotency_key=... at the PSP. Every serious provider supports this precisely because everyone hits this case. If a provider does not, that is a procurement decision, not an engineering one.
• IN_FLIGHT has an exit, and something owns it. A row that can only ever answer RECOVER is a payment nobody will resolve and money nobody will account for. The recovery worker sweeps WHERE state = 'IN_FLIGHT' AND started_at < now() - 30s, moves the row to UNKNOWN, and looks it up; a conclusive answer is DONE, an inconclusive one is MANUAL and a case in the Failure modes queue. A state the schema declares and no code path produces is a state that will be wrong when you finally need it, which is what the last assertion checks, by comparing every state the code emitted against the four the table allows.
• Key lifetime. The key must outlive the longest possible retry, which is bounded by the authorization expiry of about 7 days, not by an HTTP timeout. Expiring keys after an hour reintroduces the double-charge you just designed away.

Deep dive 4 — async settlement, and why state is a machine

A payment cannot be modelled as a boolean, and barely as a status string. A state machine is a named list of the states a thing can be in, plus the transitions allowed between them. The discipline it buys you is that code changes state by naming a transition, never by writing a status directly, so an illegal jump is a rejected transition instead of a plausible-looking row.

Authorization, capture, and settlement happen on three different timescales. A fourth clock, the dispute window (the period during which the customer can contest the charge with their bank and have it reversed) runs for months after the other three have finished:

Put the fastest and slowest in the same unit: the 120-day dispute window is ~10.4 M seconds, about 5.2 million times a 2-second authorization: six orders of magnitude between the fastest and slowest transitions in one payment’s life. No boolean models that, and neither does status = 'paid'.

What models it is a state machine whose states are the states the money is actually in. Follow the happy path first (CREATED → AUTHORIZED → CAPTURED → SETTLED → CLOSED), then the branches: everything hanging off AUTHORIZING is a payment whose outcome you do not know, and everything after SETTLED is money that can still come back out.

stateDiagram-v2
    [*] --> CREATED
    CREATED --> AUTHORIZING: submit to PSP
    AUTHORIZING --> AUTHORIZED: approved, seconds
    AUTHORIZING --> DECLINED: declined
    AUTHORIZING --> UNKNOWN: timeout or no response
    UNKNOWN --> AUTHORIZED: PSP lookup says charged
    UNKNOWN --> DECLINED: PSP lookup says not charged
    UNKNOWN --> MANUAL: lookup inconclusive
    MANUAL --> AUTHORIZED: operator resolves
    MANUAL --> DECLINED: operator resolves
    AUTHORIZED --> CAPTURED: goods shipped, hours
    AUTHORIZED --> VOIDED: order cancelled
    AUTHORIZED --> EXPIRED: 7 days, no capture
    CAPTURED --> SETTLED: funds land, T plus 2 days
    SETTLED --> REFUNDED: merchant-initiated
    SETTLED --> DISPUTED: issuer chargeback, up to 120 days
    DISPUTED --> SETTLED: dispute won
    DISPUTED --> REVERSED: dispute lost
    SETTLED --> CLOSED: 120-day window elapses
    DECLINED --> [*]
    VOIDED --> [*]
    EXPIRED --> [*]
    REFUNDED --> [*]
    REVERSED --> [*]
    CLOSED --> [*]

Four things this diagram earns you:

UNKNOWN is a real state with a real row, and something moves rows into it. Naming it forces the design of Deep dive 3 to exist, and gives the recovery worker something to query: psp_calls WHERE state = 'IN_FLIGHT' AND started_at < now() - 30s. The transition out is a lookup, and its two outcomes are the two edges: AUTHORIZED/DECLINED when the PSP answers, MANUAL when it cannot. A state on the diagram that no code path enters or leaves is decoration, and the test at the end of Deep dive 3 keeps all four honest.

EXPIRED costs money and has a size. Card authorizations expire in about 7 days; anything that ships later must be re-authorized, and a re-authorization can decline. Assume 2% of orders ship after the window and 5% of those re-authorizations decline: ~10,000 declines/day. That is a customer flow (email the customer, let them retry with another card) not an operations queue. Sizing it is what tells you it has to be automated: nobody hand-works ten thousand cases a day.

CAPTURED is where ledger entries are written; SETTLED is where cash appears. Between those two states the PSP owes you money (the psp_receivable account from The invariant) and reconciliation checks it: its balance should equal the PSP’s reported pending payout, to the minor unit.

CLOSED is 120 days after settlement. Until then the payment can reverse, so no ledger transaction may be archived and no account treated as final. This sets the hot-data retention floor and is why “we keep 30 days hot” is a wrong answer in this domain.

Deep dive 5 — reconciliation

The third and last correctness mechanism looks outward. Reconciliation compares your own records against someone else’s, item by item, and gives every disagreement a class and an owner. It is the only mechanism that detects a disagreement between your books and the outside world. The invariant of Deep dive 1 cannot help: your ledger can be perfectly zero-sum and still describe a day that did not happen.

Sizing the job

Each morning the PSP drops a settlement file: every transaction it processed the previous day, each with the provider’s reference id, the amount, and the status. Of 10 M payments, only the ~95% the card network approved appear: ~9.5 M rows, ~1.9 GB, read in under 2 s at 1 GB/s.

The match is a hash join on psp_ref: load one dataset into an in-memory lookup keyed on the join column (the build side), then stream the other side past it. It is fast as long as the build side fits in memory. Our build side is the day’s captures cut to three columns (16 B reference, 8 B amount, 1 B status) = 25 B each, so 9.5 M x 25 B ≈ 238 MB. That fits in memory, so reconciliation is a single-pass hash join that takes seconds, not an overnight batch window, one of the cheapest jobs in the system, which removes any excuse for running it less often than daily.

The five classes of mismatch

Every reference in either dataset lands in exactly one bucket. That totality (every input accounted for, none silently dropped) is what makes the output trustworthy: “the buckets are empty” really does mean “nothing is wrong”. The last assertion proves it: the bucket sizes add up to the size of the union of both key sets:

def reconcile(ours, theirs):
    """ours / theirs: {psp_ref: (amount_minor, status)}.
    Every ref lands in exactly one bucket -- that totality is the point."""
    out = {"matched": [], "missing_at_psp": [], "missing_locally": [],
           "amount_mismatch": [], "status_mismatch": []}
    for ref, (amount, status) in ours.items():
        if ref not in theirs:
            out["missing_at_psp"].append(ref)
        elif theirs[ref][0] != amount:
            out["amount_mismatch"].append(ref)
        elif theirs[ref][1] != status:
            out["status_mismatch"].append(ref)
        else:
            out["matched"].append(ref)
    for ref in theirs:
        if ref not in ours:
            out["missing_locally"].append(ref)
    return out

ours = {"a": (4825, "CAPTURED"), "b": (1000, "CAPTURED"), "c": (200, "CAPTURED")}
theirs = {"a": (4825, "CAPTURED"), "b": (900, "CAPTURED"), "d": (750, "CAPTURED")}
r = reconcile(ours, theirs)
assert r["matched"] == ["a"]
assert r["amount_mismatch"] == ["b"]
assert r["missing_at_psp"] == ["c"]
assert r["missing_locally"] == ["d"]
assert sum(len(v) for v in r.values()) == len(set(ours) | set(theirs))

Four buckets in the code, plus one the code cannot see (a duplicate reference inside the provider’s own file), make five classes, each meaning something different:

The timing class, which is what separates people

There is a sixth category that is not a mismatch at all, and mistaking it for one makes reconciliation alerting useless in practice.

The provider closes each day’s file at a fixed cutoff time. Payments captured in the last moments before the cutoff land in the next day’s file. They are not missing; they are late. At 100 payments/s, one minute of traffic straddling the cutoff is 100 x 60 = 6,000 rows. On day 0 you will see roughly 6,000 unmatched rows that are pure timing noise, and they clear themselves the next morning. An alert wired to “unmatched count > 0” therefore fires on 6,000 non-problems a day, gets muted in week one, and is not looked at again on the day it matters.

The fix is to age the unmatched set: carry each unmatched row forward and only call it a break once it has survived several settlement cycles (one cycle = one daily file):

In steady state the day-3 unmatched count is zero. “Zero after aging” is the alertable condition, and getting that threshold right matters more than the matching logic itself.

Two more things the reconciler owns:

• Fee verification. Recompute the expected fee for each row from the contracted schedule and compare it to what the provider actually charged. Providers do get fee tiers wrong, and 0.1% of the $17.5 M/day fee bill is $17,500/day ≈ $6.39 M/year going out on a fee difference nobody is checking.
• The receivable tie-out. A tie-out checks that two independently computed totals agree. Here, balance(psp_receivable) from The invariant must equal the PSP’s reported pending balance. This is a second, independent check on the same money, and it catches what the row-by-row match cannot: a systematic error consistent on both sides, so every row matches and the total is still wrong.

Failure modes

Every way the design breaks, how you notice, and what happens next. A failure with no detection line is a failure you find out about from a customer:

Retry policy, priced

The reflex answer is “retry with exponential backoff”. Backoff means you wait before retrying; exponential means each wait roughly doubles, so a struggling service gets progressively less traffic. Right instinct, wrong budget here.

Each attempt burns the full 10 s timeout before giving up, and the waits between attempts are 1 s, 2 s, 4 s. Four attempts is 10 + 1 + 10 + 2 + 10 + 4 + 10 = 47 seconds: not a checkout experience. The customer has closed the tab, or pressed the button again.

So the in-request retry budget is one attempt. After that the payment goes to UNKNOWN, the API returns 202, and retries continue asynchronously, against the lookup endpoint, never the charge endpoint.

The message queue chapter owns the delivery semantics of that asynchronous path. The queue guarantees at-least-once delivery, so duplicates will arrive. Combine that with the idempotency key of Deep dive 3 and you get effectively-once (duplicates still arrive, but the second one changes nothing) at the PSP, which is the only place it matters.

Retries also need full jitter: instead of sleeping exactly the backoff interval, sleep a random amount drawn uniformly from [0, backoff]. Without jitter, a PSP outage synchronizes every worker: they all failed at the same instant, so they all retry at the same instant, forever. 300 payments/s of synchronized retry turns the provider’s brownout (degraded but partial service) into an outage, and loses the failover you were counting on.

The dead-letter path and the human queue

After the asynchronous retries are exhausted, a payment lands in a dead-letter queue (DLQ): a side queue holding messages the system could not process, kept so they stay visible instead of discarded. A DLQ nobody drains is a data-loss mechanism with extra steps.

In this domain the drain is a person, so size that person like any other capacity. Two assumptions: 0.01% of payments end up unresolved, and each case takes 3 minutes. A full-time equivalent (FTE) is one person working one full shift (8 hours):

• 0.01% of 10 M = 1,000 cases/day × 3 min = 3,000 min = 50 hours = 6.25 FTE, and that is before shift coverage and the queue’s own tail (some cases take an hour, not three minutes).
• Drive the unresolved rate down tenfold, to 0.001%: 100 cases/day = 0.625 FTE.

That factor of ten is the business case for the automated PSP lookup in Deep dive 3, stated in headcount, not adjectives.

A human queue is a legitimate design element here, not an admission of defeat. Some cases are genuinely undecidable by software: the lookup comes back inconclusive, the settlement file and the API disagree, a customer disputes a charge the network says succeeded. So the engineering question is not “how do we eliminate it” but the three questions you ask of any queue: what is the arrival rate (1,000/day), what is the service time (3 min/case), and what is the service-level agreement (SLA), the promised time to resolve a case.

Bottlenecks and scaling

What runs out first is not a machine you own: capacity planning here is a procurement activity. Three of the five rows resolve to “nothing to do”; the two that do not (the PSP and the human queue) are the two you cannot fix by adding machines:

The bottleneck is a company you do not run. Capacity planning here means negotiating rate limits and integrating a second provider, not adding replicas.

One number makes it concrete: the PSP call is 2 s (2,000 ms) p99; a round trip inside your datacenter is ~0.5 ms. A single external call costs what 4,000 internal ones cost. Assumptions built on 500 us hops do not hold on the other side of that boundary, which is why the external call gets a state machine, a persisted intent, and a human fallback, while the internal calls get none of that.

Alternatives rejected

Each row is a design that looks reasonable and does not survive contact with the failure modes above:

Assumptions

Every number above rests on something assumed. Load-bearing means the architecture changes if the assumption is wrong; otherwise being wrong only moves a number:

Conclusion

Correctness, not throughput, is the whole problem: 300 requests/s and 74 TB over seven years fit on one box, but half the state lives at a company you do not control. Three mechanisms, each catching a class the others cannot, hold the design together. Double-entry makes a half-written money movement impossible to represent, so a single SUM = 0 query (per currency) audits the books. Idempotency, with the key committed before the external call, makes a retried charge harmless and turns an unknown outcome into a lookup, not a second charge. Reconciliation compares the books against the provider’s file daily, aging out cutoff noise so the alert means something. Money is always integer minor units, end to end. And where software genuinely cannot decide, a bounded, sized, monitored human queue owns the case: a designed output, not a defect.

One line to remember: you never re-send a charge you are unsure about, you ask what happened: the one thing you cannot roll back is the one thing you must never blindly repeat.

Cheat sheet

Further reading

• Martin Kleppmann, Designing Data-Intensive Applications: idempotency, exactly-once delivery, replication, and event logs.
• Pat Helland, “Life beyond Distributed Transactions: an Apostate’s Opinion”: why you cannot enlist an outside company in your transaction.
• Pat Helland, “Idempotence Is Not a Medical Condition” (ACM Queue): idempotency for at-least-once messaging.
• Stripe API reference, “Idempotent requests”: how a real PSP implements idempotency keys and lookups.
• Martin Fowler, “Accounting Patterns”: the entry/transaction/account model behind double-entry bookkeeping.

Next: Digital Wallet takes the same ledger inside the building, where there is no third party to blame and the hard part becomes 10,000 writes/s landing on one account.