How to design a real-time leaderboard

In this lesson, we’ll design a real-time leaderboard for 500 million players. A leaderboard orders players by score and answers two questions: the global top 10, and each player’s own rank (“you are number 4,182,996”). Both must update as scores change. By the end you’ll be able to see why ranking, not sorting, is the hard part, size each structure from the workload, and defend the whole design in an interview.

Here is the point everything hangs on: ranking is the hard part of a leaderboard, and sorting is not. Showing the top 10 is a small merge that costs microseconds. Telling one player their exact position is a count over the whole population, and that is where most of the work goes.

A few terms up front

Three of these recur throughout:

• Sorted set: a collection of members, each carrying a numeric score, kept permanently in score order. Three operations matter: insert-or-update a member’s score, read the top k, and ask for a given member’s position (its rank, where rank 1 is best).
• Redis: the in-memory store used here as the concrete reference. Its sorted-set type is a ZSET, and four commands appear repeatedly: ZADD writes a score, ZREVRANK reads a rank from the top, ZCOUNT counts members in a score range, and ZUNIONSTORE merges several sorted sets into one.
• Sharding: splitting one dataset across several machines because it no longer fits on, or can no longer be served by, one. Each slice is a shard.

What the product does

Three calls:

result carries events instead of a score because a client-supplied score is trivially forged; the cheating section explains why. Most of this lesson is about the middle row.

The one distinction everything turns on

The two questions are unrelated problems, and seeing why is the whole lesson in miniature. Top-10 is a merge of a few small lists and costs microseconds. “What is my rank” is a global aggregate over 500 million rows: a count of how many players score higher. No single machine holds enough of the population to compute that locally, which is exactly what makes it hard once the data is split across machines.

flowchart TD
    subgraph TK["top-k: decomposable"]
        A["each shard's local top 10 is enough<br/>merge 10 x 64 = 640 entries, take 10"]
    end
    subgraph RK["rank: not decomposable"]
        B["rank = 1 + count of players scoring higher<br/>local ranks do not sum<br/>needs a count over everybody"]
    end

Reaching for ZREVRANK answers only the easy half. The hard half is what happens when the sorted set no longer fits on one machine, and that is what forces every design decision below.

Requirements

The functional list is short.

• submit(player_id, match_result): the server computes the score; the client never sends one.
• top(board, k): the top k entries with score and rank.
• rank(board, player_id): the player’s rank and percentile.
• around(board, player_id, n): the n players immediately above and below.
• Boards are (game, region, window), where window is daily, weekly, monthly, or all-time.

The design pressure is in the non-functional targets, and especially the third:

Two terms in that table are load-bearing, so let’s pin them down. p99 is the 99th percentile of latency: the time 99 of 100 requests beat. We care about it more than the average because a player experiences one request at a time, and an average hides the slow ones a player will actually notice. A store is a cache when it can be thrown away and rebuilt from somewhere else, which is exactly the sorted set’s status here.

The requirement that shapes everything is that rank accuracy is not uniform. Naming it is what turns an impossible requirement (exact rank for everyone) into a tractable one, and the rest of the design builds on that reframing.

Back-of-envelope

Four given numbers drive everything: 500,000,000 registered players, 50,000,000 daily active (DAU), 10 submissions and 5 rank reads per active player per day. Every rate derives from the DAU, not the 500 M. (Estimation templates are in the estimation chapter.)

submissions/s   50,000,000 x 10 / 86,400   =  5,787    peak x3  =  17,361
rank reads/s    50,000,000 x 5  / 86,400    =  2,894    peak x3  =   8,682

The ratio is what to watch: 500 M submissions against 250 M rank reads is 2 writes per read. That is backwards from most systems, where reads dominate, and it forbids the obvious answer of precomputing a rank for every player: you would recompute more often than anyone reads it. Hold onto that ratio, because it kills a tempting design later.

The events the system handles, and the mistake that ruins each:

Data model: a sorted set, and what it costs per element

Redis stores a sorted set as two structures side by side. We’ll count the bytes of a single player’s entry first, because that one number sets every storage figure in the lesson.

A skiplist is an ordered linked list with express lanes. The idea: a plain linked list forces you to walk every node, so add sparse higher levels you can skip along. Every node sits in the bottom-level list; a random subset also appears one level up, a smaller subset above that, and so on. A search starts at the top and drops down, reaching any element in about log n steps instead of n. It gives us ordered traversal and rank.

A hash table maps a player id straight to a slot, giving O(1) “what is this player’s score”. It sits alongside the skiplist for point lookups.

Both structures hold a pointer to every element, so every player is paid for twice. That double bookkeeping is the main reason one entry costs about 100 bytes:

skiplist node (score 8 + member ptr 8 + backward ptr 8)            =  24
forward pointers and spans, average 1.33 levels x 16               =  21
member string (8-char id + sds header, rounded to a jemalloc bin)  =  16
hash-table entry (key + value + next ptrs)                         =  24
hash-table bucket, amortized at load factor 1                      =   8
allocator overhead                                                 =   7
                                                                     ---
                                                                     100

A span on each forward pointer is the count of nodes that pointer jumps over. It is what makes rank cheap: instead of walking and counting, we add up the spans along the search path. A member’s rank is that sum, so ZREVRANK is O(log n) and not a walk of the whole list.

The skiplist’s shape comes from one number, the promotion probability p. Each node is promoted to the next level with probability p, retried at each level, so heights are geometrically distributed. Redis uses p = 0.25, which fixes the average node height at 1 / (1 - p) = 1.33 levels (the 21 bytes of pointers above) and the search cost at roughly log_{1/p}(n) / p node touches. One p has to price both memory and cost, so keep them consistent: quoting 1.33 levels for memory but log2 n for cost mixes two different structures.

Multiply per-player cost by the population:

all-time board   500,000,000 x 100   =  50 GB

50 GB of RAM to store about 4 GB of actual payload (an 8-byte score-plus-id per player). That 12.5x overhead is what an in-memory ordered index costs, and what we buy with it is O(log n) rank: ten times the players costs a few more steps, not ten times the work. That is the whole reason we pay the tax.

What a rank query actually costs

O(log n) is the shape; the constant hiding inside it comes from p. With p = 0.25 and 500 million members, a rank touches about 53 nodes, and each touch is a pointer chase into scattered memory at roughly 100 ns:

node touches x 100 ns   =  53 x 100 ns   =  5.3 us per rank query

Almost all of the 5.3 us is memory stalls (cache and TLB misses), because walking pointers through 50 GB of scattered nodes is the worst case for both. The figure sometimes quoted, 2.9 us, comes from log2(500,000,000) = 29, which is the touch count of a p = 0.5 skiplist, a different structure from the one the 100-byte memory model was built for. Measured touch counts run about 1.8x log2 n at p = 0.25, so we use 53, not 29.

What actually runs out first, and it is not the CPU

To find the real ceiling, multiply peak rate by per-operation cost, which gives core-seconds per second (1.0 = one core saturated):

17,361 submissions/s x 5.3 us   =  0.092   ->  9.2% of one core at peak

So the single-threaded sorted set is nowhere near CPU-bound, despite “single-threaded” sounding like a ceiling. At 9.2% of a core, CPU has ten times the headroom we need. Four other things bind first:

Two of those rows hide a mechanism worth unpacking. BGSAVE snapshots to disk by forking: the OS gives the child a copy-on-write view of the parent’s memory, and pages stay shared until one side writes to a page. Under heavy writes, much of the data ends up duplicated, which is what doubles RSS (the physical memory the process holds) toward 100 GB. The alternative is an append-only file flushed once a second, which trades a second of possible loss for no fork. And with asynchronous replication, the primary acknowledges a write before the replica confirms it, so a window of already-acknowledged writes exists only on the primary and vanishes if it dies. That window is the 17 lost scores in the table, and it is why we need a separate record.

The sorted set is a cache, and rebuilding it takes 42 minutes

That last row about losing writes is the pivot for the rest of the lesson: the sorted set is a cache of an ordering, not the record of the scores. The append-only log is the record, and we can rebuild the sorted set from it by replaying every score as a ZADD. Pipelined, Redis absorbs about 200,000 of those per second, so a full rebuild is a division:

500,000,000 / 200,000   =  2,500 s   =  41.7 min

41.7 minutes is the recovery time objective (RTO), the target for how long the service may take to come back after losing its state. Forty-two minutes of blank leaderboard is the cost of treating the sorted set as the only copy, and it is the argument for keeping a recent snapshot: replaying a snapshot plus the log tail is minutes, while replaying all history is not.

Sharding the population

The one-key limit forces the last move: split the sorted set across S = 64 shards, each a complete sorted set over the players it owns:

per shard   50 GB / 64   =  0.78 GB      players/shard   500,000,000 / 64   =  7,812,500

Shard by player_id on a hash ring: map both players and machines onto a circle of hash values and give each player to the next machine clockwise, so adding a machine to N moves only 1/(N+1) of the members instead of reshuffling everybody (the consistent-hashing chapter).

Splitting is also what makes the single machine’s correctness worth naming, because we are about to give it up. On one machine, Redis runs commands one at a time against one key, so every submission is serialized and there is no read-modify-write race to reconcile. Correctness there is a free consequence of being on one machine, and every scaling step below spends some of it.

The hard part is rank, not top-k

Now we can see what sharding does to each query: it preserves the top-k query exactly and destroys the rank query completely.

Top-k survives. A member of the global top 10 is, by definition, in the top 10 of whatever shard holds it: if ten players on its own shard already beat it, at least ten players globally beat it. So each shard’s local top 10 is a sufficient statistic, everything the answer needs, and the rest of the shard can be ignored. Merge 10 x 64 = 640 entries and take 10: exact, one round of fan-out, microseconds.

Rank does not. A player’s rank is 1 + count of players scoring higher, a count over everybody. A shard can give the player’s rank within that shard, but those do not compose: summing 64 local ranks counts the player’s own shard-mates correctly and everyone else not at all. Rank is a global aggregate, and no local statistic reconstructs it. That leaves two real answers, and we’ll take each in turn.

Answer A: exact rank by distributed count

Here is the trick that saves us: local ranks do not sum, but local counts do. Each shard answers “how many of my players score above s” with ZCOUNT, and every higher-scoring player is counted exactly once by exactly one shard. Adding the 64 answers gives the exact global count. This pattern, same question to every shard then combine, is scatter-gather.

Each shard is 64x smaller, so a lookup on it touches about 42 nodes instead of 53. And because every rank read touches every shard once, the per-shard request rate equals the global rate no matter how many shards you add: capacity stays around 3.7% of a core per shard. So throughput is not the problem.

The problem is tail latency. A scatter-gather finishes only when its slowest branch does, so its latency is the maximum of S independent samples. Intuition first: even if each shard is almost always fast, asking 64 of them at once means the odds that none of them is slow get small. If any one call has a 1% chance of exceeding its own p99, the chance that at least one of S does is 1 - 0.99^S:

S = 64:    1 - 0.99^64    =  47%
S = 512:   1 - 0.99^512   =  99.4%

At 64 shards, 47% of rank queries pay some shard’s p99; at 512 shards, 99.4% do, so the fleet’s p99 becomes the query’s median. This is tail amplification, and it is the scaling wall, not throughput.

Hedged requests buy back most of it: once the p95 has elapsed with no answer, send a duplicate to a second replica and take whichever returns first. By construction only the slowest 5% of branches get duplicated, so it costs about 5% extra load. It works up to a few hundred shards. Past that, fan-out is finished, and we need a structure that answers rank without touching every shard.

Answer B: bucketed approximate rank

Instead of counting players one at a time, keep a histogram: cut the score line into B buckets and store only count[b], the number of players in each. A player’s approximate rank is the total count in the buckets above theirs. We know their bucket but not where inside it, so the worst-case error is exactly that bucket’s population. That single sentence is the whole accuracy story, and everything below is about shrinking that population.

Make the buckets equi-depth (each holding the same number of players, so its boundaries are the quantiles of the score distribution) instead of equi-width. Here is why it matters: game scores are brutally skewed, with most players clustered near zero. Equi-width buckets would pile a huge share of the population into the crowded middle bucket:

modal equi-width bucket   0.05 x 500,000,000   =  25,000,000 players
equi-depth bucket, B=1,000   500,000,000/1,000 =     500,000 players

That is a factor of 50 worse error, and it lands on exactly the players in the middle who most want an accurate percentile. Equi-depth fixes this by making every bucket hold N/B players, so the error is a flat N/B everywhere. Computing the boundaries is a nightly batch job: sample scores, read off the value at every 1/B of the way through, republish. We can get away with nightly because the distribution’s shape moves far more slowly than any individual score.

Sizing B from the accuracy target: the requirement is exact inside the top 10,000 and within 1% below that, so r_min, the worst rank where the 1% promise holds, is the head board’s boundary, 10,000. Worst-case error N/B must be within 1% of it, which pins down B:

N/B <= 0.01 x 10,000 = 100   ->   B >= 500,000,000 / 100 = 5,000,000

Five million buckets, 100 players each, 100-rank worst-case error. At 16 bytes per bucket that is 80 MB, replicated, cheap: a rounding error next to the 50 GB sorted set.

The constraint that actually binds is that a boundary is a score. Equi-depth boundaries are drawn from observed score values, so B can never exceed the number of distinct scores available to cut on. As the ties section shows, a naive 21-bit score field has only 2^21 - 1 = 2,097,151 values, which floors the error at 500,000,000 / 2,097,151 = 238 ranks no matter what B you write down. So the fix lives in the score field, not the histogram: a rebased 28-bit field gives 2^28 = 268,435,456 distinct scores, leaving 53x of room for B = 5,000,000. This is why the accuracy requirement and the tie-break encoding turn out to be the same decision.

Querying the histogram uses a Fenwick tree. The query needs a prefix sum (the total count in all buckets above one). The naive way, adding up buckets one by one, is O(B) and defeats the point. A Fenwick tree (binary indexed tree) instead stores partial sums at positions chosen by the binary representation of the index, so both a prefix sum and a single-bucket update cost O(log B). Let’s cost the whole rank query, not one call inside it: a rank makes four prefix walks plus a binary search over the boundary array, about 64 memory references, so 6.4 us per query, and keeping the tree current under peak writes costs about 12% of a core. That clears the 50 ms budget by four orders of magnitude, and it does it with one hop instead of the 47%-tail scatter-gather.

Here is the histogram, trimmed to the load-bearing parts, where the rank method returns both the approximate rank and its error bound (the bucket’s own population) so a caller always knows how much to trust the answer:

class BucketedRank:
    """Approximate global rank from a Fenwick tree over equi-depth
    score buckets. Error is bounded by one bucket's population, N/B."""

    def __init__(self, boundaries: list[int]):
        # k boundaries cut the line into k+1 buckets; a Fenwick tree over
        # m buckets needs m+1 slots (1-indexed). Get either +1 wrong and
        # the DROPPED bucket is the top one -- the champions everyone reads.
        self.boundaries = boundaries
        self.n_buckets = len(boundaries) + 1
        self.tree = [0] * (self.n_buckets + 1)

    def _bucket(self, score):                 # rightmost boundary <= score
        lo, hi = 0, len(self.boundaries)
        while lo < hi:
            mid = (lo + hi) // 2
            if self.boundaries[mid] <= score: lo = mid + 1
            else: hi = mid
        return lo

    def _add(self, i, delta):
        i += 1
        while i < len(self.tree):
            self.tree[i] += delta
            i += i & (-i)

    def _prefix(self, i):
        i += 1; total = 0
        while i > 0:
            total += self.tree[i]
            i -= i & (-i)
        return total

    def update(self, old_score, new_score):   # a move is a decrement + increment
        if old_score is not None:
            self._add(self._bucket(old_score), -1)
        self._add(self._bucket(new_score), +1)

    def rank(self, score):
        """Returns (approximate rank, worst-case absolute error)."""
        b = self._bucket(score)
        above = self._prefix(self.n_buckets - 1) - self._prefix(b)
        in_bucket = self._prefix(b) - (self._prefix(b - 1) if b else 0)
        return above + 1, in_bucket

The two + 1s in the constructor are the whole trick, and they compound. k boundaries make k + 1 buckets (one below the first boundary, one above the last), and a 1-indexed Fenwick tree over m buckets needs m + 1 slots. Size the array at len(boundaries) + 1 instead, and _add on the top bucket runs off the end and silently drops the increment, losing counts for exactly the highest-scoring players most likely to be looking.

Exact head, bucketed tail

One number resolves the exact-versus-approximate tension: the same absolute error means very different things at different ranks.

error 5,000 at rank 10           =  500x the rank      (nonsense for the champion)
error 5,000 at rank 10,000,000   =  0.0005            (invisible to the median player)

So the answer is not “exact” or “approximate” but exact at the head, bucketed in the tail. Keep the top 10,000 as their own exact sorted set (10,000 x 100 bytes = 1 MB, replicated to every read machine) and serve everyone below it from the histogram, where players ask for “top 2%” instead of an integer rank. We fall back to the scatter-gather only when a caller explicitly wants an exact deep rank (a profile page, a dispute), where a slow answer is acceptable but a wrong one is not.

flowchart TD
    R["rank(player) request"] --> H{"in top 10,000?"}
    H -->|yes| HEAD["exact head board<br/>1 MB local copy, exact"]
    H -->|no| E{"caller wants exact?"}
    E -->|no| HIST["bucket histogram<br/>one hop, 6.4 us, error 100 ranks"]
    E -->|yes| SG["scatter-gather<br/>64 x ZCOUNT, exact, slow tail"]

Ties and score updates

Start with the concrete case: two players reach 50,000 points. Who ranks higher? By default Redis orders equal scores lexicographically by member, so the tiebreak is “whoever has the alphabetically smaller player id”, which is deterministic, arbitrary, and will be noticed. Players expect first to reach the score ranks higher.

We fix it inside the score itself. A Redis score is an IEEE-754 double, which represents every integer up to 2^53 exactly and rounds above that. The idea is to spend those 53 bits on two fields at once, points in the high bits and an inverted timestamp in the low bits, so ordinary numeric comparison sorts first by points, then by time:

def composite(points, t_seconds):
    return points * 2**32 + (2**32 - 1 - t_seconds)

Multiplying points by 2^32 shifts them above the 32 timestamp bits so the fields never interfere. Storing (2^32 - 1) - t_seconds inverts the clock, so an earlier submission produces a larger composite and wins ties. The largest value this 21/32 layout can produce is 2,097,151 x 2^32 + (2^32 - 1) = 9,007,199,254,740,991, which is exactly 2^53 - 1, one below the double’s exact-integer ceiling. That is the check that matters: ordering is always exact, nothing rounds.

If 2.1 M points is too small a ceiling, rebase the clock to the season start instead of 1970. A 2^25-second clock covers 2^25 / 86,400 = 388 days and leaves 53 - 25 = 28 bits for points, i.e. 268 M distinct scores. This is not only a ceiling question. Recall from the histogram that the width of the score field is also the number of distinct scores, and the histogram cannot have more buckets than there are distinct boundaries to draw. 21 bits floors the bucketed error at 238 ranks; 28 bits gives B = 5,000,000 room to spare. So the rebased layout is what the accuracy requirement forces, not an exotic option.

The timestamp must come from one authority, not from whichever game server ran the match. Clocks that differ by 200 ms order simultaneous submissions by clock error, and a clock that steps backwards reorders already-published history. Use the score service’s own receive time or a monotonic sequence (the ID-generator chapter covers the backwards-clock case).

Finally, the write command matters as much as the ordering. An operation is idempotent when doing it twice leaves the same state as once. That is the property that makes a retry safe, and it matters because a network gives you at-least-once delivery, so retries will happen whether you plan for them or not.

ZADD GT is the design you want. When the product genuinely needs a cumulative board, you have re-created the exactly-once counting problem: deduplicate on match_id before the increment, or make the write an absolute set instead of an add (the ad-aggregation chapter).

Time-windowed boards, and the storage multiplier

Real products want daily, weekly, and monthly boards too. Each is a separate sorted set over a separate population, sized from the DAU (a daily board only holds players who played that day). The weekly and monthly multipliers of 3.5x and 8x are the distinct players seen over a week and a month, not 7x and 30x, because the same people come back day after day:

all-time   500,000,000 x 100   =  50 GB
daily       50,000,000 x 100   =   5 GB
weekly     3.5 x DAU x 100     =  17.5 GB
monthly    8   x DAU x 100     =  40 GB

Materializing history the obvious way, keeping 30 dailies, 8 weeklies, and 12 monthlies all live, costs 5x30 + 17.5x8 + 40x12 + 50 = 820 GB, a 16.4x multiplier over the all-time board, and about 620 GB of it is history almost nobody reads. The fix leans on one fact: a weekly board is derivable from its dailies, so only the current window needs to be live. That gives 30 dailies + current week + current month + all-time = 257.5 GB, a 3.2x reduction.

We derive the closed windows with ZUNIONSTORE, whose cost is O(N) + O(M log M): N is the total input elements, M the result size, and the sort term dominates. Merging 7 dailies is about 7 x 50M = 350M element merges plus sorting a 175M-element result, roughly 5.1 billion operations, about 86 minutes, not the 350 seconds the merge term alone suggests. That is fine as a 03:00 job and fatal in a request, but 86 minutes is still a scheduling constraint, so roll incrementally: ZUNIONSTORE week week today folds a single day into a standing weekly, keeping both N and M at about one board’s worth.

One flag makes or breaks this: use AGGREGATE MAX, never the default SUM. When ZUNIONSTORE finds a member in several inputs it combines their scores, and summing composites is wrong twice over: it carries timestamps up into the points field, inventing a score nobody achieved, and it blows the 2^53 exactness budget (seven maximal composites sum well past 2^53 and round). Concretely, a player who scored 600 twice would sum to a phantom 1,201 and beat a player who scored 1,000 once. MAX keeps each player’s best single-day composite intact, score and tiebreak both.

The policy, then:

• dailies are the primitive everything else is built from;
• weeklies and monthlies roll nightly and incrementally with AGGREGATE MAX;
• closed windows are snapshotted to disk and dropped from memory (set a TTL on each daily key so Redis expires it, instead of a cleanup job that can fall behind);
• the all-time board never expires.

One correctness note before we move on: a daily board’s boundary is a timezone. Auckland and Los Angeles do not share a Tuesday. Either the board is explicitly on UTC and the UI says so, or it is per-region and each region is a different key. A per-player timezone makes two players non-comparable, and something you cannot order is not a leaderboard.

Cheating: server-authoritative scoring

The score must be computed by a server you control, from match events. This is a hard requirement, not a hardening measure. Here is why there is no middle ground: a score computed on the player’s device is an integer supplied by an adversary who owns the debugger, the memory, and the binary. Match events (inputs, timings, outcomes) are the raw record the server can replay under the game’s own rules to derive the score itself.

Signing the score on the client does not help, and it is worth seeing why before you reach for it. The signing key ships in the binary and is extracted in an afternoon, and a signed forgery is worse than an unsigned one, because it carries a valid signature that defeats your own anomaly detection. The same goes for obfuscation and checksums: they raise the cost of the first exploit and zero the cost of every copy after it.

What actually works, in layers:

1. Server simulation. The client submits events; the server replays them under the game rules. Expensive, and the only real defense.
2. Plausibility bounds. Reject any submission implying more than the game’s theoretical maximum rate. Cheap, catches the naive majority.
3. Rate-limit submissions with a token bucket: a counter refilling at a fixed rate up to a cap, spending one token per request (the rate-limiter chapter). Derive the parameters from honest behavior, not capacity. A legitimate player submits 10/day; a bucket refilling once per 60 s allows at most 86,400 / 60 = 1,440/day, so a compromised account gets 144x the honest rate and no more. The cap of 5 lets a player finish five quick matches back to back without being throttled. Aggregate capacity is a non-issue at 17,361 writes/s; the bucket exists only to bound one account’s blast radius.
4. Anomaly quarantine, not rejection. Flagging scores above a cohort’s p99.99 flags one in 10,000, which is 500,000,000 x 0.0001 = 50,000/day, far past human review, so the flag cannot mean “reject”. It means the score is accepted and recorded but withheld from the public board pending an automated replay check. Rejecting outright punishes the genuinely exceptional player, and that is a worse failure than briefly delaying a cheater.
5. Idempotency as integrity. Deduplicating on match_id stops replay attacks, and ZADD GT makes a replayed high score a no-op even if the dedup misses. A replayed high score is indistinguishable from a legitimate one at the storage layer, so two independent mechanisms is the right amount.

Final design

flowchart TB
    subgraph WRITE["write path"]
        G(["game servers<br/>authoritative simulation"]) --> RL["rate limiter<br/>token bucket per account"]
        RL --> SS["score service<br/>dedup on match_id, ZADD GT"]
        SS --> LOG[("append-only score log<br/>system of record")]
        SS --> Z[("ZSET shards, S = 64<br/>hash ring on player_id<br/>0.78 GB each")]
        SS --> HB[("bucket histogram<br/>Fenwick tree, 80 MB, replicated")]
        Z --> HEAD[("exact head board<br/>top 10,000, 1 MB, replicated")]
        LOG --> RB["rebuild job<br/>RTO 41.7 min"] --> Z
    end

    subgraph READ["read path"]
        Q(["read API"]) --> C["1 s TTL cache<br/>+ single flight"]
        C --> HEAD
        C --> HB
        C -->|"exact deep rank"| FO["scatter-gather<br/>64 x ZCOUNT"] --> Z
    end

Reading the write path top to bottom: the score service writes to three places, the append-only log (the system of record, from which everything rebuilds), the 64 ZSET shards (hash-ringed on player_id, serving top-k and the exact scatter-gather), and the bucket histogram (approximate rank out of 80 MB). The head board is the top 10,000 kept exact and replicated. Every read first hits a 1-second TTL cache with single flight (when many identical requests miss at once, only one runs and the rest wait on its result), which is what tames the top-10 hot key, since a single key has one owner and only caching, not sharding, can absorb its traffic.

Bottlenecks and scaling

At each population, something different runs out first, and the fix is whatever answers that specific limit. The most useful row is the first: do not build most of this lesson until you have to.

Failure modes

Alternatives rejected

Conclusion

• A leaderboard is two unrelated problems. Top-10 is a decomposable selection that shards perfectly (merge each shard’s local top 10). Rank is a global aggregate that does not shard, and it is where all the work goes.
• An in-memory sorted set costs about 100 bytes per element (50 GB for 500 M players) and buys O(log n) rank. CPU is not the constraint; RAM, the persistence fork, and failover loss are. The sorted set is a cache rebuilt from an append-only log, at a 42-minute RTO.
• Exact distributed rank via scatter-gather ZCOUNT is correct but dies to tail amplification: 1 - 0.99^S reaches 47% at 64 shards. The default path is a Fenwick tree over equi-depth buckets: 80 MB, one hop, 6.4 us, error bounded by one bucket’s population.
• Accuracy must vary with rank: exact in a 1 MB head board of the top 10,000, approximate below it. And a bucket boundary is a score, so the histogram’s accuracy is capped by the width of the score field, which ties the tie-break encoding and the accuracy target into one decision.
• Pack points and an inverted timestamp into the 53 exact bits of a double so comparison sorts by score then time. Use ZADD GT for idempotent retries and ZUNIONSTORE ... AGGREGATE MAX for windowed boards.
• The score is server-authoritative or it is nothing. Everything else (plausibility bounds, token buckets, anomaly quarantine) is defense in depth around that one rule.

The single assumption the whole design rests on is that scores are mutable and a change must be visible within a second. Relax it to “rank may be a day old” and the sorted set is unnecessary: a nightly precomputed rank column is 145 seconds of compute and the correct answer instead.

One line to remember: top-k shards, rank does not, so you buy back exact rank only where a player can tell the difference.

Further reading

• Redis documentation on sorted sets (ZADD, ZRANGE, ZCOUNT, ZUNIONSTORE): the exact command semantics used throughout.
• William Pugh, “Skip Lists: A Probabilistic Alternative to Balanced Trees” (1990): the structure behind ZSET ordering and the role of p.
• Peter Fenwick, “A New Data Structure for Cumulative Frequency Tables” (1994): the binary indexed tree used for the histogram.
• Jeffrey Dean and Luiz André Barroso, “The Tail at Scale” (CACM, 2013): tail amplification and hedged requests, the reason scatter-gather stops scaling.