How to design a metrics and alerting system

In this lesson, we’ll design the system that watches a whole fleet. Every service reports numbers about itself: request counts, memory, latencies. A monitoring system gathers all of it, draws the dashboards, and wakes someone the moment something goes wrong. By the end you’ll be able to size each tier from a workload, say which number forces the next decision, and defend the whole design in an interview.

Reach for the usual pipeline diagram (agents, a queue, a store, a dashboard) and you’ll size the wrong thing. The quantity that sets the bill is not how much data flows through the system, but the number of distinct label combinations that are actively receiving samples. Get that wrong by a factor of a thousand, which takes one careless label, and the fleet you must buy is a thousand times larger. Hold onto that: cardinality, not throughput, is the real size.

Vocabulary

Five terms carry the rest of this lesson, so let’s pin them down before we build anything.

• A metric is a named quantity a process reports about itself: http_requests_total, process_memory_bytes.
• A label is a key/value pair attached to a metric so you can slice it: service="api", status="500". Labels are how you ask for “errors, but only on checkout.”
• A series is one metric name plus one complete set of label values. http_requests_total{service="api", endpoint="/checkout", status="500"} is one series. Change any single label value and you have a different series.
• A sample is one (timestamp, value) pair appended to a series. A scrape collects one sample for every series a process currently exposes.
• A series is active if it is still receiving samples. Active series are the ones the system must keep resident in memory. A series nobody writes to any more has been flushed to disk and costs almost nothing.

Cardinality is the count of active series, and the thing to internalize is that it is a product, not a sum. Count it once with tiny numbers: three services x five endpoints x four status codes is 60 series. Now add one more label carrying a thousand distinct values and the whole product is multiplied by a thousand, not increased by a thousand. That multiplication is the heart of this design, and it is what makes every sizing question below turn on which labels you allow.

The three metric types

Everything the fleet reports is one of three types, and they cost wildly different amounts of storage.

• A counter only ever goes up: total requests served, total bytes sent. You never read a counter directly; you ask the server for its rate().
• A gauge goes up and down: memory in use, queue depth, temperature.
• A histogram is a set of counters, one per latency bucket (“requests under 10 ms”, “under 25 ms”, and so on), plus a running _sum and _count. One histogram is therefore many series, which is why histograms dominate the size estimate.

What the system must do

Functional:

• Ingest counters, gauges, and histograms from every process, each tagged with a label set.
• Query by label selector over a time range, with aggregation pushed into the store, not done in the dashboard. The three aggregations that matter: sum adds series together, rate() converts a counter into per-second change, and histogram_quantile() reads a percentile out of histogram buckets.
• Evaluate alert rules on a schedule, with a for duration (the condition must hold continuously for that long before the alert fires) and a routing/deduplication layer.
• Keep retention tiers: full resolution for recent data, downsampled for the long term.
• Support multi-tenancy with per-tenant series limits. A tenant is one team or environment sharing the cluster, and the per-tenant limit is the only real defence against cardinality.

Non-functional: two properties are non-negotiable, and both come from the same fact: a dropped sample never comes back, but a dropped dashboard recovers on the next refresh.

• Ingest availability sits strictly above query availability. A monitoring system that returns errors is annoying. One that silently drops the samples belonging to the service currently on fire is worse than no monitoring at all, because people trust it.
• Load shedding is loud, attributed, and bounded. When the system refuses data it says so, names who and what it refused, and refuses a known amount.

Concrete targets: sample loss below 0.01% (an order of magnitude under the tightest budget these samples police), total-outage detection under a minute, query p99 under a second for an hour of 120 series, ingest availability above 99.99%, and a cardinality ceiling of 75,000 active series per tenant. Every one of these numbers is derived later; none was picked for looking round.

Sizing: cardinality first, everything else second

Let’s size a concrete fleet. Assume 500 hosts running 20 services, each service exposing 50 endpoints, each endpoint reporting 10 status codes, scraped every 30 seconds. So host, service, endpoint, and status are the four label dimensions.

The obvious move is to multiply all four: 500 x 20 x 50 x 10 = 5,000,000. That is wrong, and here is why: a host runs exactly one service. Once you know the host you know the service, so service carries no information the host label did not already carry. Multiplying by it invents 19 impossible combinations per host and counts every real series 20 times.

When one label is fully determined by another, that is a functional dependency, and it always overcounts by exactly the cardinality of the dependent label, here 20x. Cardinality is the product of only the labels that vary independently: 500 x 50 x 10 = 250,000. Spotting one functional dependency saved an order of magnitude before anything was designed, which tells you these are the first questions to ask, not the last.

Now the real total, three groups added at the end. A counter contributes one series per label tuple; a histogram contributes one series per bucket plus _sum and _count, so 12 buckets become 14 series, which is why histograms are the largest group.

request counters:                   500 x 50 x 10   =  250,000
latency histograms (14 series each): 500 x 50 x 14   =  350,000
process + host + runtime, ~300/host: 500 x 300       =  150,000
active series (total)                                =  750,000
samples/s                            750,000 / 30    =   25,000

25,000 samples/s is nothing. One machine handles it easily, and that is the point: monitoring is a cardinality problem, not a throughput problem. The fleet you buy is set by how many series must stay resident in memory, not by how many samples per second arrive.

What one active series costs

Every active series costs memory whether or not anyone queries it, because to accept the next sample the process must already hold the label set, the index entry, and the half-filled chunk the sample will be appended to. None of that can live on disk when a sample arrives every 30 seconds.

Adding it up: the label set (~300 B), the series-ref-to-label-set map entry, the inverted-index postings, and the one or two compressed head chunks come to roughly 1 KB of live heap per series. A garbage collector that runs when the heap reaches twice the live data leaves about half of resident memory as garbage awaiting collection, so the figure to carry is 2 KB of resident memory (RSS) per active series. (RSS is the memory the operating system is actually holding for the process; divide an RSS budget by an RSS figure or you overstate the box by 2x.)

The baseline: 3% of one machine

A 64 GB box has about 48 GB usable after the OS, page cache, and query working memory take their share, which holds roughly 23 million series. So:

baseline memory:  750,000 x 2 KB  =  1.5 GB
boxes needed:     1.5 GB / 48 GB  =  0.03  (3% of one box)

The entire monitoring system for a 500-host fleet fits in 3% of one machine. You run two, and the second is purely for availability, not capacity.

Add one label: 2,134 machines

A developer breaks the request counter down by customer with a user_id label carrying 200,000 distinct values a day. It multiplies only the 250,000 request counters (the conservative reading, and still catastrophic):

with user_id:   250,000 x 200,000  =  50,000,000,000 series
capacity:       50 B / 23.4 M/box  ≈  2,134 boxes  (vs 1)
cost at ~$1/box-hour:  ~$18.7 M/yr  (vs ~$8,760/yr)

One line of code took an $8,760/year bill to $18.7 M/year, a factor of 2,134. Both sides of that ratio are bare capacity, both rounded up to whole machines. If you put redundancy on one side only, the ratio starts lying: capacity compares to capacity, or HA to HA, never one of each. That is why the defence is a hard per-tenant limit, not a code-review guideline.

An unbounded label is a different kind of mistake

user_id at least has a ceiling. request_id does not: a new value is minted for every request, forever. At 500 hosts serving 100 requests/s, that is ~4.3 billion new series a day, each receiving exactly one sample. The index needed to describe those series (~1 KB each) is then about 731x larger than the sample data it points at (~1.4 B/sample).

A label with one distinct value per event turns a time-series database into a log store with none of a log store’s optimizations. The rule follows directly:

A label is legal only if its value set is bounded and does not grow with traffic.

status, region, version, instance are fine: new values appear rarely and the set has a ceiling. user_id, request_id, url, error_message are not, and no amount of tuning makes them fine. Those belong in logs or traces, where the engine indexes high-cardinality fields on purpose and charges per event instead of per series.

Two defences, not interchangeable

• A per-tenant limit on total active series bounds the damage and stops the ingester dying. It tells you nothing about the cause, so you still get paged and go looking.
• A per-label limit on distinct values, enforced at ingest and reported with the offending label name in the rejection body, turns a 3 a.m. page into a one-line fix. The rejection says user_id and the engineer knows exactly which pull request to revert.

Run both. Then alert on the rate of new series creation, d(series)/dt, not on memory: an explosion is visible in the creation rate within a minute, but only visible in memory once the ingester is already most of the way to dying.

Data model: series, chunk, block, index

Four nested units, smallest to largest.

series  = metric name + sorted label set  ->  a 64-bit series ref
sample  = (series ref, timestamp ms, float64)
chunk   = 120 consecutive samples of ONE series, compressed together
block   = 2 hours of chunks + an inverted index, immutable once written
index   = label=value  ->  sorted list of series refs (a postings list)

Two of those lines need unpacking. The series ref is a 64-bit integer the store assigns to each distinct label set, so a sample on the wire carries 8 bytes instead of a repeated 300-byte label set. A postings list is the classic inverted-index structure: for each label=value pair, the sorted list of series refs carrying it, so looking up {status="500"} fetches one list instead of scanning every series.

flowchart TD
    subgraph BLK["Block: 2 hours, immutable once written"]
        subgraph CHK["Chunk: 120 samples of ONE series, compressed together"]
            SMP["Sample = (timestamp, value)"]
        end
        IDX["Inverted index: label=value -> sorted series refs"]
    end

The unit that matters is the chunk. Storing samples of one series adjacently is what makes compression possible and turns a range query into a handful of reads instead of thousands of lookups. A design that keys samples by timestamp instead of by series has thrown away the only locality this workload has, and every other decision here follows from keeping it.

Architecture

Two paths run through the system. The write path (down the left): targets are scraped, samples land in a local write-ahead log, a distributor hashes each series to three ingesters, ingesters flush two-hour blocks to object storage, and a compactor merges and downsamples them. The read path (up the right): a query frontend splits a request by time, answering recent data from the ingesters and older data from a store gateway reading object storage.

The one box on neither path is the shed door: the point where the system refuses data instead of buffering it.

flowchart TB
    T1["Targets: ~2,500 /metrics endpoints"] -->|"pull every 30 s"| SC["Scrapers, sharded by hash(target)"]
    SD["Service discovery"] -.-> SC
    SC --> WAL["Local WAL + head block, 2 h in memory"]
    WAL -->|"remote write"| DIST["Distributor: hash(series ref) picks ingesters"]

    DIST -->|"429 when a limit is breached"| SHED(["Shed at the door: never queue"])
    DIST --> I1["Ingester 1, replication factor 3"]
    DIST --> I2["Ingester 2"]
    DIST --> I3["Ingester 3"]

    I1 --> BLK["2 h blocks to object storage"]
    BLK --> CMP["Compactor: merge + downsample to 5 m, 1 h"]
    CMP --> OBJ[("Object storage, ~194 GB per replica")]

    Q["Query frontend: split by time, cache"] --> I1
    Q --> SG["Store gateway: reads blocks + index"]
    SG --> OBJ

    RUL["Ruler: evaluates every 15 s"] --> Q
    RUL --> AM["Alertmanager: group, dedupe, silence, route"]

Two labels there are shorthand. A WAL (write-ahead log) is an append-only file a process flushes before it acknowledges, so a crash loses nothing already accepted. Replication factor 3 means every series is held on three ingesters, so losing one loses no data.

Push vs pull collection

The usual comparison is bytes on the wire. It is the wrong axis. Pull (the server scrapes GET /metrics from each target on a schedule) uses about 2% of a 1 Gbps network card; push (each process sends samples on its own schedule) uses roughly 2x more in a naive protocol, and label interning (sending each label set once, then referring to it by an integer) erases even that gap. Bandwidth does not decide it.

Four properties do. One term first: backpressure is a receiver’s ability to make a sender slow down, instead of absorbing the excess in a buffer or dropping it silently.

Service discovery is the registry that says which processes currently exist and where. sample_limit is a pull-side setting: if a target’s /metrics response contains more than N series, the scraper discards the entire response and records the scrape as failed. That is the blast-radius row in action. A bad deploy that adds user_id to a metric on 100 targets would push 10 billion series with push; with pull’s sample_limit those 100 scrapes fail wholesale, admit nothing, and fire up == 0 on exactly those 100 targets. Pull converts an unbounded global failure into a bounded local one plus an alert.

Pull has two honest costs. Service discovery becomes a hard dependency: no discovery, no targets, no metrics. And the interval quantizes everything: sampling at interval T resolves only features longer than 2T (the Nyquist limit), so a 30 s scrape cannot see a 5-second latency spike. A counter still captures the spike’s total, though, because it accumulates between scrapes and the next scrape reports the accumulated value. That is why you export counters and let the server compute rate(), instead of exporting a pre-computed rate that gets sampled and loses the spike.

The practical answer is both: pull for anything long-lived you can reach, and a push gateway for batch jobs and functions that will not exist by the next scrape.

Why a general key-value store is the wrong shape

The cheap-sounding answer is to put samples in Cassandra, RocksDB, or DynamoDB. Let’s see why it loses on writes by two to three orders of magnitude and on reads by two, and why both losses are structural, not tunable.

Writes

Here is the intuition first: if the store keeps reorganizing files on disk, it rewrites the same byte many times, and you pay for every rewrite. Most modern key-value stores use an LSM tree (log-structured merge tree): it buffers writes in memory, flushes them to a sorted file, then repeatedly compacts by merging small files into larger ones. Every merge rewrites data already on disk, and that rewriting is the cost, measured as write amplification: bytes actually written to the device per byte the application handed over.

A sample’s payload is (series ref 8 B, timestamp 8 B, value 8 B) = 24 bytes, which becomes about a 50-byte row once the store adds its own key, header, and index share. Then:

• Leveled compaction amplifies roughly 42x (one write-ahead log copy, one flush, then rewritten once per level down a 4-level ladder with a 10x size ratio). So ~2,100 B written per sample against a time-series store’s ~1.4 B: about 1,500x.
• Size-tiered compaction amplifies ~4.5x instead, merging files of similar size instead of pushing every byte down the ladder: about 161x. It writes less but leaves more files for a read to search, a write-cost-for-read-cost trade.

Always say which compaction strategy you mean, because the two differ by an order of magnitude and the choice is a config knob, not a property of the engine. Either way, two mechanisms produce the gap: per-row overhead dwarfs a 24-byte payload, and nothing is compressed against its neighbours (each sample is an independent row).

Reads

A one-hour dashboard panel over 120 series scraped every 30 s is 120 x 3,600 / 30 = 14,400 samples. In a key-value store that is 14,400 point lookups. In a time-series store, where a chunk holds 120 samples of one series, it is 14,400 / 120 = 120 chunk reads: one I/O returns an hour of one series.

Do not turn that into a latency claim by multiplying 14,400 by an SSD’s 100 us access time. That figure is queue-depth-1 latency, not a throughput ceiling; a modern NVMe drive sustains 500,000 to 1,000,000 random IOPS by servicing hundreds of requests at once, so one 14,400-lookup panel finishes in about 29 ms, well inside the 1 s target. The real argument is throughput: one device serves about 35 key-value-shaped panels per second against about 4,167 time-series-shaped ones. That 120x gap is exactly the I/O count, and a dashboard is thousands of panels against a shared fleet, so 120x fewer device operations per panel means 120x fewer devices.

The index and the query that kills the box

The index is an inverted index, not a B-tree over rows: label=value -> sorted postings list of series refs, exactly like a search engine mapping a word to the documents containing it. A selector with two matchers fetches two sorted lists and intersects them with galloping search (jumping ahead exponentially in the longer list instead of walking it), so the work is driven by the shorter list. On this schema {service="api"} matches 37,500 series, and intersecting is ~20x cheaper than scanning all 750,000.

The structure also explains the one query that reliably kills the box. A regex matcher cannot use a postings list, because a postings list is keyed by an exact label=value pair. endpoint=~".*checkout.*" forces the store to enumerate every distinct value of endpoint, test each against the pattern, fetch a postings list per match, and union them, which is 50 lists here. On the exploded schema, user_id=~"..." would be 200,000 lists in one query from one panel. Never regex a high-cardinality label: it is the difference between one lookup and 200,000.

Delta-of-delta and XOR: 16 bytes down to 1.4

A raw sample on the wire is 8 + 8 = 16 bytes: an 8-byte millisecond timestamp and an 8-byte float64 value. Gorilla-style encoding gets that to about 1.4 bytes by exploiting two facts about how metrics are produced: scrapes happen on a fixed schedule, and most values barely move between scrapes.

Timestamps: delta-of-delta

Do not store the timestamp; store how much the gap changed, D = (t[n] - t[n-1]) - (t[n-1] - t[n-2]). On a perfect 30 s schedule every gap is 30,000 ms, so every D is zero, and zero costs one bit. Real scrapes jitter, so a variable-length prefix code spends a few tag bits to say how wide the payload is:

With most samples landing exactly on schedule and a few jittering into the small buckets, the timestamp costs about 1.35 bits per sample.

Values: XOR against the previous value

A float64 is a sign bit, an 11-bit exponent, and a 52-bit mantissa (the significant digits). A gauge going 0.4100 -> 0.4103 keeps the same sign and exponent and changes only low mantissa bits. XOR the two 64-bit patterns and identical bits cancel to zero, leaving leading zeros, a short window of bits that differ, and trailing zeros. Store only that middle window, and re-use it when the next XOR fits:

With about half of values unchanged and most of the rest fitting the previous window, the value costs about 9.84 bits per sample. So 1.35 + 9.84 = 11.19 bits, which is 1.40 bytes per sample, an 11.4x reduction from 16.

The entire gain comes from storing samples of one series adjacently: delta-of-delta needs the previous two samples of the same series, XOR needs the previous one. A key-value layout that never puts two samples of a series next to each other has nothing to compress against, which is the structural reason it cannot be rescued with a better compressor.

The compression also forces two constraints. Chunks are immutable and append-only, because a bit-packed stream cannot have a sample inserted in the middle, so out-of-order writes must go to a separate head or be rejected (this is why a silently dropped backfill is such a common surprise). And counters compress far better than gauges: a steadily rising counter has near-constant deltas and a stable XOR window, while a noisy gauge takes the 32-bit path more often, which is one more reason to prefer counters plus server-side rate().

Retention tiers, and why downsampling does not save storage

Downsampling replaces N raw samples with one summary point covering the same span. A 5-minute tier replaces the ten 30-second samples in each window with one point. That point must carry four aggregates (min, max, sum, count), not one, or you cannot recover a peak or divide by the right denominator for a mean or rate.

Three tiers at 1.4 bytes/sample: raw (30 s, 15 days), 5-minute (90 days), and 1-hour (400 days) come to about 194.5 GB per replica, 389 GB replicated (the figure on the object-storage box above).

Now the surprise. The 5-minute tier costs 2.4x more than the raw tier it summarizes. Ten samples become four values, so the rate drops only 10 / 4 = 2.5x, not 10x, because of the four aggregates, while the tier is kept 6x longer (90 days vs 15). 6 / 2.5 = 2.4. Downsampling does not save storage; it buys query speed.

Where it pays is the read path. A 30-day panel over 100 series is 8.64 million raw samples rendered into a 1,000-pixel-wide panel, which is 8,640 samples decoded per pixel. The 5-minute tier cuts that decode 10x and changes nothing a human can see. That is why step_s is a query parameter: the store, not the dashboard, should pick the tier and answer a wide panel from a coarse one instead of decoding samples the caller will discard.

Raw retention length is chosen against how far back an incident review needs full resolution, not against a disk quota. And the whole 389 GB of metrics sits against a log pipeline that runs to terabytes per day, because metrics cost per series and stay flat as traffic grows, while logs cost per event and grow linearly with it. That asymmetry is why metrics and logs are separate products with separate retention.

Alerting: for does not fix a bad alert

Alert on a per-series threshold and the false-alarm rate scales with the fleet, not with the incident rate. Watch what that does with numbers. Take a threshold a perfectly healthy series still exceeds with probability 0.001 on any evaluation, roughly a 3-sigma line (the textbook “basically never by chance” cutoff). Evaluated every 15 s across 10,000 series:

evaluations/s:   10,000 / 15         =  667
false trips/s:   667 x 0.001         =  0.667
false trips/day: 0.667 x 86,400      =  57,600

57,600 pages a day with nothing wrong. That is not noisy, it is structurally unusable.

The reflex fix is a for clause requiring the condition to hold for a while first. for: 5m covers 20 evaluations. If those trips were independent the chance of all 20 firing is 0.001^20, effectively zero, and the problem would be solved. But metric noise is autocorrelated: a series does not bounce randomly, it drifts into a bad regime and stays there. Model regime lengths as an exponential with a 120-second mean, and for: 5m survives whenever a regime outlasts 300 s, which happens with probability e^(-300/120) = 0.082. That cuts 57,600 pages to about 4,729, a suppression of only 12x. You needed a factor of 57,600. for suppresses flapping within one alert; it cannot touch the fleet-size term.

The fix is to change what is measured. An SLO (service level objective) is a stated target for a user-visible quantity, such as “99.9% of checkout requests succeed over 30 days.” Aggregate the 10,000 series into 20 SLO signals and threshold the aggregate. Two independent wins multiply: 500x fewer evaluations, and averaging 500 series shrinks the noise of the average by sqrt(500) = 22.4x. A threshold sitting 3.1 standard deviations out on one series sits about 69 standard deviations out on the group, where random excursions never reach it and only a real shift in the underlying error rate does.

That is the mechanical version of a folk rule: alert on symptoms, not causes.

Keep the cause metrics, because they are how you diagnose after a symptom alert fires, but do not page on them. That is what dashboards and, at most, tickets are for.

Burn-rate alerting, derived from the budget

The idea is to treat the SLO as a spending account and page when we are draining it too fast. Start from the SLO: 99.9% availability over 30 days. The error budget is what the SLO permits you to fail, 1 - 0.999 = 0.001, and 30 days is exactly 720 hours. Burn rate is the observed error rate divided by the budgeted one: burn 1 consumes the whole budget in 30 days, burn 10 in 3 days, burn 1,000 means everything is failing.

The budget consumed by sustaining burn rate B for W hours is B x W / 720. Invert it: decide how much budget you will lose before someone is told, and the threshold is forced. Willing to lose 2% in one hour? 0.02 x 720 / 1 = 14.4. That is where the famous number comes from; it is a consequence of one policy decision, not a tuned constant.

Every row is that same equation with different inputs:

Four rows, because one rule cannot do the job. A single fast rule misses slow burns: a service failing 0.5% of requests burns at 5, exhausts the budget in six days, and never trips 14.4. A single slow rule is blind to a total outage for hours, because a 72-hour average takes hours to move. And the severities differ because the response differs: 2% gone in an hour needs someone awake now; 10% over three days needs a ticket in the morning.

Detection time falls out of the threshold. A total outage burns at 1 / 0.001 = 1,000, so the 1-hour rule fires once its window’s average crosses 14.4, which at burn 1,000 takes 14.4 / 1,000 of the hour, about 51.8 seconds. That is where the sub-minute detection target came from; it was implied, not chosen.

Each row also has a second, shorter window, because the long window has memory on the way down. After a 10-minute outage ends, the 1-hour window still holds those bad minutes and keeps firing for nearly an hour, which is how on-call engineers learn to ignore an alert. Requiring the short window to also be above threshold clears it within 5 minutes. The long window controls precision; the short window controls reset time; neither alone gives both.

flowchart LR
    E["Error ratio = rate(errors) / rate(total)"] --> L1["1 h burn > 14.4"]
    E --> S1["5 m burn > 14.4"]
    L1 --> A1{"AND"}
    S1 --> A1
    A1 -->|"2% budget, ~52 s on a full outage"| PAGE(["Page"])
    E --> P2["6 h burn > 6 AND 30 m burn > 6"] -->|"5% budget, slower burns"| PAGE
    E --> T3["24 h burn > 3"] --> TIC(["Ticket"])
    E --> T4["72 h burn > 1"] --> TIC

The write path: shed, don’t queue

The write path is a fan-in: thousands of independent senders, one logical sink, with the arrival rate set by the senders. No amount of provisioning changes who controls that rate.

Steady state (25,000 samples/s) is one machine’s work. Recovery is what sizes the tier. During a network partition every sender keeps scraping and buffers to its local WAL; when connectivity returns they all replay at once, a thundering herd. A 10-minute partition leaves a 15-million-sample backlog per the fleet; if senders drain it within 30 s, that is 15,000,000 / 30 = 500,000 samples/s, a 20x burst. The 20x is not a constant, it is 600 s of backlog / 30 s of catch-up deadline; halve the deadline and the herd doubles to 40x.

The trap is that a queue looks like it absorbs the burst while it is quietly setting up a feedback loop. Suppose the tier is provisioned for 100,000 samples/s and queues the excess instead of rejecting it. The queue grows at 500,000 - 100,000 = 400,000/s, so after ten seconds it is 4 million deep, and an item entering it waits 4,000,000 / 100,000 = 40 seconds (Little’s law). The sender’s timeout is 30 s, so it gives up, marks the batch un-acked, and retries. The same samples now arrive twice, arrivals rise, the queue grows faster, more senders time out. That loop is congestion collapse: throughput falls as offered load rises, caused by the buffer, not relieved by it.

An immediate HTTP 429 does three things a queue cannot: it costs one cheap rejection instead of a buffered sample plus a doomed write; it pushes the backlog into the sender’s bounded, disk-backed WAL, which is built for exactly this and holds hours; and it gives the sender an explicit signal to back off exponentially, so the herd de-synchronizes. Queueing moves an overload from a place with backpressure to a place without one; rejecting keeps it where the buffer already exists.

Shed selectively. First-come-first-served lets the loudest tenant win, so give each tenant a limit: fair share doubled, 750,000 / 20 x 2 = 75,000 series, so a tenant with genuinely more series is not punished. Within a tenant, reject samples for new series before samples for existing ones: a series created ten seconds ago is on no dashboard and in no alert rule yet, so dropping it costs nothing today, while an existing series is being evaluated right now and dropping it makes an alert go blind. That one ordering rule is what makes a cardinality explosion degrade only the tenant causing it. Finally, the 429 body must name the tenant, the metric, and the label that breached: the difference between a four-minute fix and a four-hour one is whether the rejection said which label exploded.

Bottlenecks and scaling

The axis to remember: this system scales on series count, not sample rate. Doubling the scrape frequency doubles samples at ~1.4 bytes each and costs almost nothing. Doubling label cardinality doubles memory, index size, query cost, and box count at once. “Can we scrape every 10 seconds?” is usually a cheap yes. “Can we add a label?” requires arithmetic.

Failure modes

The dead-man’s switch is worth its own note. Every alert here fires on the presence of bad data. A pipeline that stops delivering data fires nothing at all, which looks exactly like health. One rule that always fires, routed to a receiver that pages when it stops arriving, is the only thing that covers that case.

Alternatives rejected

Conclusion

• The system is sized by active series, and series count is a product of independent label cardinalities. Spotting a functional dependency (a host runs one service) removes a 20x overcount before anything is designed.
• The whole 750,000-series baseline fits in 1.5 GB, about 3% of one machine, and throughput (25,000 samples/s) is trivial. One unbounded label (user_id) takes that to 50 billion series and 2,134 machines. A label is legal only if its value set is bounded and does not grow with traffic; enforce it with a per-tenant and per-label limit, and alert on d(series)/dt.
• Store samples of one series adjacently. That locality is what makes delta-of-delta + XOR compression reach 1.4 bytes per sample, and it is the single reason a general key-value store loses by two to three orders of magnitude.
• Downsampling buys query speed, not storage; a coarser tier can even cost more, so keep it for the read path and let the store pick the tier via step_s.
• Alert on symptoms via SLO burn rate, not on per-series thresholds. Aggregation, not a longer for, is what makes the alert fire only when something is actually wrong, and the 14.4 threshold and sub-minute detection both fall straight out of the error budget.
• On overload, shed at the door with a 429 that names the offender, reject new series before existing ones, and never queue. And run a dead-man’s switch, because every other alert is blind to a pipeline that has gone silent.

Further reading

• Pelkonen et al., Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015): the delta-of-delta and XOR compression scheme.
• Fabian Reinartz, Writing a Time Series Database from Scratch: the series/chunk/block/postings model behind the Prometheus TSDB.
• The Site Reliability Workbook, “Alerting on SLOs”: multiwindow, multi-burn-rate alerting.
• Prometheus documentation, “Storage” and “Querying”: scrape model, retention, and the query API in practice.
• Grafana Mimir / Cortex / Thanos documentation: the horizontally scalable distributor-ingester-store-gateway architecture sketched above.

One line to remember: a metrics system is sized by active series, and active series is a product of independent label cardinalities, so the whole design is a fight to keep that product bounded.