How to design a metrics and alerting system

A metrics system is sized by active series, and active series is a product of independent label cardinalities, so the whole design is a fight to keep that product bounded.

Vocabulary

• Metric: named quantity a process reports (http_requests_total).
• Label: key/value pair to slice a metric (status="500").
• Series: metric name + one complete label set; change any label value and it is a different series.
• Sample: one (timestamp, value). Scrape: one sample per exposed series.
• Active series: still receiving samples, so must stay resident in memory.
• Cardinality: count of active series. It is a product of independently varying labels, not a sum. Adding one label with 1,000 values multiplies the total by 1,000.
• Functional dependency: one label fully determined by another (a host runs one service); overcounts by exactly the dependent label’s cardinality.

Three metric types

• Counter: only goes up; never read directly, ask for rate().
• Gauge: goes up and down (memory, queue depth).
• Histogram: many counters (one per latency bucket) + _sum + _count; 12 buckets = 14 series. Dominates the size estimate.

Sizing (cardinality first)

• Example fleet: 500 hosts, 50 endpoints, 10 status codes. Multiply only independent labels: 500 x 50 x 10 = 250,000 (host determines service, so drop the 20x).
• Cost per active series: ~1 KB live heap, 2 KB RSS (GC doubles it). Divide RSS budget by RSS figure.
• ~23 M series per 64 GB box (48 GB usable).

counters   500 x 50 x 10   = 250,000
histograms 500 x 50 x 14   = 350,000
host/proc  500 x 300       = 150,000
total active series        = 750,000
samples/s  750,000 / 30    =  25,000   (trivial; one machine)
memory     750,000 x 2 KB  =  1.5 GB   = 3% of one box

• Add unbounded user_id (200k values): 250,000 x 200,000 = 50 B series -> ~2,134 boxes, $18.7M/yr vs $8,760/yr.
• Rule: a label is legal only if its value set is bounded and does not grow with traffic. status, region, version fine; user_id, request_id, url belong in logs.

Storage and compression

• Data model: series (name + labels -> 64-bit ref) -> sample -> chunk (120 samples of ONE series, compressed) -> block (2 h, immutable) + inverted index (label=value -> sorted postings list).
• Locality (samples of one series adjacent) is what enables compression and cheap range queries.
• Gorilla: delta-of-delta timestamps (~1.35 bits) + XOR values (~9.84 bits) = 11.19 bits = 1.4 B/sample, 11.4x reduction from 16 B.
• Chunks are immutable/append-only; out-of-order writes go to a separate head or are rejected. Counters compress better than noisy gauges.
• Why not a key-value store: per-row overhead swamps the 24 B payload and nothing compresses against neighbours. ~1,500x write amplification (leveled) or 161x (size-tiered); 14,400 point lookups vs 120 chunk reads on a 120-series hour panel.

Retention and downsampling

• Downsampling replaces N raw samples with one point carrying 4 aggregates (min/max/sum/count).
• Tiers: raw 30 s / 15 d, 5 m / 90 d, 1 h / 400 d = ~194.5 GB per replica, 389 GB replicated.
• Downsampling does NOT save storage (5 m tier costs 2.4x more: 2.5x fewer samples but kept 6x longer). It buys query speed; store picks the tier via step_s.

Alerting

• Per-series thresholds fail: false trips scale with fleet size. 10,000 series at 3-sigma, every 15 s = 57,600 false pages/day.
• for does not fix it: noise is autocorrelated, so for: 5m only cuts pages ~12x; you needed 57,600x. It suppresses flapping within one alert, not the fleet-size term.
• Fix: alert on symptoms via SLO burn rate, not causes. Aggregate 10,000 series into ~20 SLO signals (500x fewer evals + sqrt(500) = 22.4x less noise).
• Error budget = 1 - SLO. Burn rate = observed error rate / budgeted. Threshold forced by policy: lose 2% in 1 h -> 0.02 x 720 / 1 = 14.4.
• Total outage burns at 1,000, so the 1 h rule fires in 14.4/1,000 h ≈ 52 s (the sub-minute target).
• Use a long window (precision) AND a short window (reset time within ~5 min).

Write path and gotchas

• Push vs pull: bandwidth is not the axis. Pull wins on liveness (up == 0), blast radius (sample_limit per target), backpressure (server sets rate), losing only reachability. Use both: pull for long-lived, push gateway for batch jobs.
• Interval quantizes: scrape at T resolves only features longer than 2T (Nyquist). Export counters + server-side rate() so spikes survive.
• On overload shed at the door with HTTP 429, never queue: a queue crosses the sender timeout (40 s head vs 30 s timeout by Little’s law), retries amplify arrivals -> congestion collapse. Recovery herd is ~20x steady (600 s backlog / 30 s catch-up).
• Shed selectively: per-tenant limit (fair share x2 = 75,000); reject new series before existing ones; 429 body names tenant, metric, and offending label.
• Two defences: per-tenant total limit (bounds damage) + per-label distinct-value limit (names the cause). Alert on d(series)/dt, not memory.
• Dead-man’s switch: one always-firing rule that pages when it stops arriving; the only thing that catches a silent pipeline.