How to design idempotent APIs and webhooks

In this lesson, we’ll make an API survive the real world: clients retry, networks drop responses, and traffic spikes without warning. We’ll build the three mechanisms that keep it correct and stable under that pressure. Idempotency makes retries safe, rate limiting shares capacity fairly, and webhooks push events back to clients so they stop polling. By the end you’ll be able to design a retry-safe write, throttle a caller without lying to it, and deliver an event that arrives at-least-once and out-of-order without corrupting the receiver.

Idempotency makes a retried write safe

Picture a client that sends POST /payments and never receives a response. It cannot tell whether the payment succeeded or the reply was lost, so it retries. A naive server treats the retry as a fresh request and charges twice.

The fix is an idempotency key: the client generates a unique key per logical operation and sends it as a header, and the server guarantees that every request carrying the same key produces one single effect. The first request does the work and remembers the outcome under that key; every later request with the same key gets the remembered outcome back instead of doing the work again. (This is exactly the double-charge problem that the payment system chapter has to solve.)

flowchart TD
  A["POST /payments<br/>Idempotency-Key: abc123"] --> B{"Key seen<br/>before?"}
  B -- "no" --> C["Process once,<br/>store key + response"]
  C --> D["Return result"]
  B -- "yes, completed" --> E["Return stored response<br/>(no re-processing)"]
  B -- "yes, in progress" --> F["Return 409 / retry-after"]

The diagram shows the three cases the server must handle: unseen key, key already completed, key still running. Getting them right in code comes down to a few notes:

Persist the key with the request fingerprint and the final response. Set a TTL (for example 24 hours) that comfortably exceeds any client’s retry window, so the record is still there when a slow client finally retries but does not live forever.
Insert the key inside the same transaction that performs the effect, so a crash cannot leave the effect done but the key unrecorded. A unique constraint on the key turns a duplicate into a caught conflict rather than a second charge. This is the crux: if the key and the effect could commit separately, a crash between them reopens the double-charge you were trying to close.
If the same key arrives with a different body, reject it with 422: that signals a client bug, not a retry.

PUT and DELETE are already idempotent by definition; idempotency keys exist mainly to make POST safe to retry. Idempotency protects the write once a request reaches the service. The next mechanism decides which requests are allowed to reach it at all.

Rate limiting protects shared capacity

Rate limits stop one caller from exhausting capacity everyone shares, and they give every client predictable service instead of a system that is fast until a neighbor floods it. The token bucket is the common choice because it allows short bursts while still bounding the long-run rate.

Here is the idea before the parameters. Imagine a bucket that tops up at a steady drip and holds only so much. Every request spends one token; when the bucket runs dry, requests are refused until it drips full enough again. The steady drip caps your sustained rate, and the bucket’s size caps how big a burst you can spend at once. The rate limiter chapter covers the algorithms and their trade-offs in more depth; here is the short version.

A bucket holds up to B tokens and refills at R tokens per second. Each request removes one token; if the bucket is empty, the request is rejected with 429.
B caps burst size; R caps the sustained rate. A fixed-window counter is simpler but allows a double-rate burst across the window boundary; a sliding window smooths that at higher cost.

Why the fixed-window burst happens is worth holding in your head: if the limit is B per minute, a client can spend B at the very end of one window and B at the very start of the next, landing 2B requests inside a couple of seconds while never breaking the per-window count. The token bucket has no window boundary to exploit, which is why it bounds the burst cleanly at B.

Always tell the client where it stands with response headers so it can self-throttle instead of hammering you:

RateLimit-Limit: 100
RateLimit-Remaining: 12
RateLimit-Reset: 30
Retry-After: 30

Enforce limits at the gateway (per API key or per IP) so rejected traffic never reaches the service. On the client side, retry 429 and 5xx with exponential backoff and jitter to avoid a synchronized retry storm: backoff spreads retries out over time, and jitter scatters them so a thousand clients that failed at the same instant do not all retry at the same instant and re-crash you. Rate limiting decides what gets in and idempotency guards the write. The third mechanism carries the result back out to the client.

Webhooks push events instead of making clients poll

Polling for state changes is wasteful and slow: the client asks “is it done yet?” on a loop, and most of those calls learn nothing. Webhooks invert the flow. The client registers a URL once, and the server sends an HTTP POST to it the moment an event actually occurs.

sequenceDiagram
  participant S as Your service
  participant Q as Delivery queue
  participant C as Client endpoint
  S->>Q: enqueue event (order.paid)
  Q->>C: POST /webhook (signed payload)
  alt 2xx received
    C-->>Q: 200 OK (ack)
  else failure or timeout
    C-->>Q: error / no response
    Q->>C: retry with backoff
  end

That flow only holds up in production if four things are handled, and each one answers a specific failure:

Signing. Include an HMAC signature header computed over the raw body with a shared secret so the receiver can verify authenticity and reject forgeries. Without it, your webhook URL is a public endpoint anyone can POST fake events to.
Retries with backoff. Deliver from a durable queue and retry failed deliveries over a decaying schedule; after a maximum window, mark the endpoint failing and alert the owner. The queue is what lets a delivery survive the receiver being briefly down.
At-least-once delivery. A delivered event may arrive more than once (a retry after the receiver processed but before its ack landed), so make the receiver idempotent by de-duplicating on the event ID, the same discipline as request idempotency, applied on the receiving side.
Ordering. Do not assume events arrive in order. Attach a monotonically increasing sequence or timestamp and let receivers reconcile.

At-least-once delivery drags idempotency back into the picture, this time on the client’s side, and that is the thread tying all three mechanisms together.

How the three fit together

The three mechanisms sit at different points of a request’s life, and each hands off to the next. Rate limiting runs first, at the edge, so rejected traffic never reaches the service. Idempotency guards the write itself, so a retried request has the same effect as the first. Webhooks push the result back, and the receiving side reuses the same idempotency discipline to absorb duplicate deliveries.

flowchart TD
  CL["Client"] -->|"POST + Idempotency-Key"| GW["Gateway<br/>token bucket per key/IP<br/>429 + RateLimit headers"]
  GW -->|"under limit"| SVC["Service<br/>key seen? one effect only<br/>store key + response in txn"]
  SVC --> DB[("Store<br/>key, fingerprint, response")]
  SVC -->|"event"| Q["Durable delivery queue"]
  Q -->|"signed POST, retry w/ backoff"| RCV["Client endpoint<br/>verify signature<br/>de-dupe on event ID"]

Following the diagram left to right, the same idea shows up twice: de-dupe on the way in (the idempotency key at the service) and de-dupe on the way out (the event ID at the receiver). Both are the same move, “act on this only once,” applied at the two ends of the round trip.

Conclusion

Idempotency keys make non-idempotent writes safe to retry. Store the key inside the same transaction that performs the effect, guarded by a unique constraint, so a crash or a duplicate cannot cause a second effect.
Rate limit at the gateway with a token bucket (bursts up to B, sustained rate R), reject with 429, and return RateLimit headers so clients self-throttle. Clients should retry 429 and 5xx with exponential backoff and jitter.
Deliver webhooks from a durable queue: sign the payload, retry with backoff, and assume at-least-once, out-of-order delivery. The receiver de-duplicates on event ID and reconciles order from a sequence number or timestamp.

One line to remember: build every write so it can be tried twice and land once, and every event so it can arrive twice and count once.

Idempotency makes a retried write safe

Rate limiting protects shared capacity

Webhooks push events instead of making clients poll

How the three fit together

Conclusion

Further reading