Three mechanisms keep an API correct under retries, spikes, and pushed events: idempotency makes retries safe, rate limiting shares capacity, webhooks deliver events at-least-once.
Idempotency (retry-safe writes)
- Idempotency key: client sends a unique key per logical operation as a header; same key produces one single effect.
- First request does the work and stores the outcome under the key; later same-key requests return the stored outcome.
- Persist key with request fingerprint + final response; set a TTL (e.g. 24h) that exceeds any client’s retry window.
- Insert the key inside the same transaction as the effect; a unique constraint turns a duplicate into a caught conflict, not a second charge.
- Same key with a different body -> reject
422(client bug, not a retry). PUT/DELETEare already idempotent; keys mainly makePOSTsafe.
Three server cases per key:
| Key state | Response |
|---|---|
| Unseen | Process once, store key + response |
| Completed | Return stored response, no re-processing |
| In progress | 409 / retry-after |
Rate limiting (shared capacity)
- Token bucket: holds up to
Btokens, refills atRtokens/sec; each request removes one; empty -> reject429. Bcaps burst size;Rcaps sustained rate.- Fixed-window counter is simpler but allows a
2Bburst across the window boundary; sliding window smooths it at higher cost. Token bucket has no boundary to exploit. - Return headers so clients self-throttle:
RateLimit-Limit,RateLimit-Remaining,RateLimit-Reset,Retry-After. - Enforce at the gateway (per API key or IP) so rejected traffic never reaches the service.
- Clients retry
429and5xxwith exponential backoff and jitter; jitter scatters retries so failed clients do not re-crash you in sync.
Webhooks (push, not poll)
- Client registers a URL once; server
POSTs to it when an event occurs. Four requirements: - Signing: HMAC signature header over the raw body with a shared secret, so the receiver rejects forgeries.
- Retries with backoff: deliver from a durable queue; after a max window, mark endpoint failing and alert the owner.
- At-least-once: an event may arrive more than once -> receiver de-duplicates on event ID.
- Ordering: do not assume in-order; attach a monotonic sequence or timestamp and reconcile.
How they fit
- Rate limiting runs first at the edge; idempotency guards the write; webhooks push the result back.
- De-dupe appears twice: idempotency key on the way in, event ID on the way out. Same move, both ends.
Remember
- Build every write so it can be tried twice and land once.
- Build every event so it can arrive twice and count once.