What this is
In this lesson, we’ll design the “friends near me” map you have seen in Snap Maps or Find My Friends, and the first move is to notice it is not the problem it looks like. It looks like a map search over moving pins. It is really a presence system (the machinery that tracks who is online right now) with a coordinate attached to each online user. That one reframing removes most of the storage you would otherwise reach for, because a location is worthless a minute after it is taken, so nothing about it is worth keeping durably. By the end you’ll be able to size the working set, justify why there is no database on the hot path, decide push versus pull from the write:read ratio, and defend skipping the spatial index in an interview.
Everything downstream follows from that single fact: where a location lives, who gets told about it and when, why the geospatial index the problem seems to demand is usually the wrong tool, and how a user goes invisible in a bounded amount of time.
What goes in and what comes out
in a stream of location fixes, one per publishing user every ~30 s
example: {user_id: 42, lat: 40.7580, lng: -73.9855, accuracy: 12, ts: ...}
out for each viewer with a map open, a stream of dots for the friends who
share with them, each carrying a distance and an age
example: [{friend_id: 91, lat, lng, distance_m: 1840, age_s: 12}, ...]
Two terms to fix now. A fix is one position reading from a device: a GPS sample with a timestamp. age_s is how many seconds ago that reading was taken. As the TTL section shows, age_s is the most important field in the response, because a dot on a map is only as useful as it is recent.
Which problem this actually is
This resembles the proximity-service chapter with the points moving, but it inherits its shape from the chat-system chapter’s presence system with a payload attached. The proximity service builds a read-heavy index over restaurants that never move; presence keeps tens of millions of phones connected and tracks which are still alive. Three differences push this problem toward the second:
| Proximity service | Nearby friends | |
|---|---|---|
| Load | Read-heavy over a corpus that barely changes | Write-heavy: every phone reports a new position every 30 s |
| Data lifetime | An address is true for a decade | A fix is worthless within a minute |
| Recipients | Chosen by radius | Chosen by social graph — your friend list decides who sees you |
The middle row is the design’s hinge: when data expires in a minute, durability stops being a virtue and starts being a cost. The spatial index is not the hard part either. As the geospatial deep dive shows, you may not need one at all. First, let’s turn those three differences into hard requirements.
Requirements
The functional list is ordinary. The non-functional constraints are what remove the storage layer, so spend your attention there.
Functional
- A user publishes location while sharing is on and the app is running.
- A user sees friends who share with them, with distance, on a map.
- “Nearby” highlights friends inside a radius: 5 km is the default.
- Sharing is per-friend, revocable, and instantly effective.
- Invisible mode: stop publishing, and disappear from everyone’s map.
Non-functional — these decide the design
A few terms first. TTL (time to live) is an expiry stamped on a stored value; after that many seconds the store deletes it on its own. A write-ahead log is the durable append-only file a database writes before acknowledging a change, so a crash can be replayed. It is what makes a write survive a power cut, and it is exactly what this design refuses to build. A quorum is the rule that a write counts as done only once enough replicas acknowledge it.
Each requirement forces a mechanical consequence, something this design either builds or refuses to build:
| Requirement | Consequence |
|---|---|
| Freshness beats durability | Location lives in memory with a TTL. No write-ahead log for a dot |
| A stale dot is a correctness bug | The TTL is a product requirement, not a cache-tuning knob |
| Battery and NAT bound the update rate | Same ceiling as a heartbeat: send something every 120–240 s at most |
| Privacy is enforced server-side | Every fanout edge is authorized at send time, never at render time |
| Loss of a location is survivable | No replication, no quorum. A fresh fix re-arrives in 30 s |
NAT (network address translation) is the box between a phone and the internet that rewrites addresses; it forgets an idle connection after a few minutes, which is why a client must send something periodically whether or not it has news.
The whole design follows from one rule: losing a location is fine, showing a stale one is not. That is the opposite of every storage-oriented design, which is why none of that machinery appears here. Before we trust that instinct, let’s price the storage we are refusing to build.
Back of the envelope: why not a database
Population figures line up with the chat-system chapter: 500 M daily active users, 20 M concurrent connections, half of them sharing → 10 M concurrent publishers. At one fix every 30 s that is 10,000,000 / 30 ≈ 333,333 fixes/s on average, and about 833,333/s at the 2.5x busy-hour peak.
Two rates, spent on different things, stated once so no later section has to argue it:
Fleets, core counts and device counts are sized on the 833,333/s peak. Bytes-per-day and cost-per-year figures use the 333,333/s average. Hardware must survive the worst second; a storage bill accumulates over a year.
One record is 28 bytes: user_id 8, lat/lng 4 each (int32 microdegrees, ~11 cm resolution), recorded_at 8, accuracy 2, flags 2.
The ratio that settles it
The working set (one current position per publisher, live in memory) is 10 M × 28 B = 280 MB, which fits in the RAM of one ordinary machine.
Storing every fix durably instead: 333,333 × 28 B ≈ 9.3 MB/s, which is ~806 GB/day, ~294 TB/year, or ~883 TB once you keep three replicas.
Divide the two, because the ratio is the whole argument: a year of history is about a million working sets (294 TB / 280 MB ≈ 1,051,199). Note the unit: this is bytes accumulated per year over bytes live now, so it is a rate, not a dimensionless multiple. You would write 294 TB a year to preserve data that is wrong 30 seconds after it lands. That is the central finding.
The device-count argument, and why it is weak
A commonly quoted version argues from disks instead of bytes and lands on 81 devices with replication. It does not hold up, and knowing why matters more than the figure, because the interviewer who quotes 81 wants to see you catch the error.
That version divides the average load, at 100% utilization, by 100,000,000 / 8,192 ≈ 12,207 page writes per device: a queue-depth-1 latency number misread as device capacity. A real NVMe drive is fed many requests at once and does ~250,000 8 KB writes/s. Sizing against the peak at 60% utilization gives 833,333 / (250,000 × 0.6) ≈ 6 devices, or 18 with replication, not 81.
And even 18 assumes one random page write per fix. Real LSM stores (Cassandra, RocksDB) buffer writes and flush them as large sequential files. At ~10x write amplification the peak is 833,333 × 28 B × 10 ≈ 233 MB/s, about 23% of one NVMe’s sequential bandwidth. The device leg collapses.
So the disk-count objection dies, but two arguments against durability survive:
- The ratio: 294 TB/year against a 280 MB working set, for data with a 30-second shelf life.
- The coupling (the stronger point): a durable write puts a replicated commit on the critical path of a fix that will be superseded before anyone reads it. You pay latency and add a component that can fail, to preserve a value nobody will read.
Bandwidth is a non-issue
Peak ingest is 833,333 × 28 B × 8 ≈ 187 Mbps, about 19% of a single 1 Gbps network card, for the entire input from ten million phones. So the bytes were never the problem; the durability was. That settled, let’s write down the three stores and watch only one of them survive.
API sketch
WS is a WebSocket: a connection that stays open and lets both sides send at any time. POST/GET are ordinary HTTP.
WS /v1/location client -> {lat, lng, accuracy, ts, moving}
server -> {friend_id, lat, lng, distance_m, age_s}[]
POST /v1/sharing {friend_id, mode: precise | city | off}
POST /v1/visibility {mode: on | ghost | off}
GET /v1/friends/nearby?radius_m=5000
-> {friends: [{id, distance_m, age_s, precision}]}
GET /v1/sharing/audit -> {viewer_id, last_seen_at}[]
The first two lines are one socket carrying traffic both ways. Three commitments in that shape each remove a class of client bug:
age_sis returned with every dot, so the client cannot draw a stale position as a fresh one: it is handed the staleness directly.precisionis returned, so a city-level dot cannot be rendered as a street-level pin.- One socket carries both directions. Publishing and receiving share the same connection; a second socket would double a fleet already at 20 M connections and buy nothing.
Data model
Three stores, and only one is durable.
location_live in memory, sharded by user_id, TTL 60 s
user_id -> {lat, lng, recorded_at, accuracy, flags}
sharing_edges durable, sharded by owner_id
owner_id, viewer_id, mode, created_at the authorization set
subscriptions in memory, sharded by publisher_id, TTL 120 s
publisher_id -> {viewer_id: gateway_id} who is watching right now
Sharded by means rows are split across machines by hashing that column, so each machine owns a disjoint slice of the keys.
location_live: one slot per publisher holding their current position. The only thing moving at 333,333/s, and the only thing with no durability.sharing_edges: who may see whom, at what precision. It changes a few times per user per year, so it is a small, heavily cached relational table.subscriptions: who currently has a map open on whom, and which gateway to route dots to. Rebuilt whenever someone opens the app, so it needs no durability either.
Shard location_live on user_id, not on cell
The tempting alternative is to give each shard a region of the map, as a proximity service does. Here it is a mistake, and the reason is that users move and cells do not. Work the number: a precision-6 cell is ~610 m tall, and a driver at 30 m/s crosses one every ~20.3 s. With ~2 M publishers driving at any moment, that is 2,000,000 / 20.3 ≈ 98,522 shard-to-shard migrations per second: pure overhead that produces no answer for anyone, existing only to keep the shard map consistent with the physical world.
Sharding on user_id makes the same update a single in-place write to a fixed owner. A user’s home shard never changes no matter where they go, so there is nothing to migrate. Use a consistent-hashing ring on user_id so adding a machine moves only a 1/n slice of the keys. With the stores decided, let’s assemble them into a request path.
High-level architecture
A fix comes in from a phone, gets authorized, gets stored, gets published, and comes back out through the gateway. Only the two shaded stores own data; everything else is stateless.
flowchart LR
M["Mobile clients<br/>20M open sockets"] --> GW["Gateway tier<br/>100 boxes, 200k sockets each"]
GW --> IN["Location ingest<br/>authorize, rate-limit, TTL"]
IN --> LS["location_live store<br/>in RAM, TTL 60s, sharded by user_id"]
IN --> PS["Pub/sub bus<br/>topic = publisher_id"]
PS --> GW
SUB["Subscription registry<br/>who is watching whom"] --> PS
GW --> SUB
SE[("sharing_edges<br/>durable authorization set")] --> IN
style LS fill:#1d3557,color:#fff
style SE fill:#1d3557,color:#fff
- Gateway tier: the machines terminating the sockets. Inherited from the chat-system chapter: 100 boxes at 200 k connections each, a count set by blast radius (how many users one failure takes down), not by how much RAM a socket needs. Treated here as already paid for.
- Location ingest: authorizes the fix, applies the rate limit, writes it with a TTL. Stateless.
location_live: the single in-memory slot per user. It is the only copy of a location anywhere, which makes it authoritative and not a cache, however much the TTL makes it look like one. A cache has something to fall back to; this does not.- Pub/sub bus: publish/subscribe, where the topic name is the publisher’s user id, so “subscribing to Alice” means subscribing to topic
alice_id. - Subscription registry: which viewers currently watch which publishers, and on which gateway. Losing it costs one round trip to rebuild.
sharing_edges: the one durable store, holding who may see whom.
The two data stores are the whole thesis as a picture: everything else forwards and forgets. In one sentence, a fix enters through the socket it will leave through, is authorized against a durable edge set, lands in a TTL’d memory cell, and is republished only to viewers currently looking. That last clause is what the next section prices.
Deep dive 1: push, pull, and the ratio that inverts
The news-feed chapter derives the general rule for whether a producer’s update is copied to consumers up front (push, fan-out-on-write) or fetched on demand (pull, fan-out-on-read). For one producer with F consumers, push costs F × w per write and pull costs F × r per read, so the comparison is w / r: the producer’s write rate against the consumer’s read rate. The follower count F cancels.
Here the two rates invert the usual answer, and that inversion is the whole insight. An average sharing user publishes ~2% of the day (10 M of 500 M publishing at any instant), which is ~1,728 s, or 1,728 / 30 ≈ 57.6 fixes a day. They open the map about 3 times. So w / r = 57.6 / 3 ≈ 19.2, nearly twenty writes per read, which points against push. A news-feed author, by contrast, writes 0.2 posts against 5 reads a day, and push wins by 25x. Same rule, opposite inputs.
The resolution: subscribe-on-view
Pure pull is cheap here (a viewer’s friend set is small, and each lookup in an in-memory map is ~100 ns), but it forces the client to poll every few seconds to feel live, which is push with worse latency. The answer is the one presence systems use: push, but only along edges someone is actively looking at. Opening the map subscribes you to your friends’ topics; closing it lets the subscription lapse.
flowchart TD
O["Viewer opens map"] --> S["Subscribe to each live friend's topic"]
F["Friend publishes a fix"] --> W{"Anyone subscribed<br/>to this publisher?"}
W -->|"yes"| D["Push dot to those viewers only"]
W -->|"no"| X["Store in location_live, send to nobody"]
C["Viewer closes map"] --> L["Subscription lapses via TTL"]
A given friend is publishing only if connected and sharing (20M/500M × 0.5 = 0.02), so of a 200-friend list about 4 are live on any open map. Naive push (every fix to every live friend) is 333,333 × 4 ≈ 1.33 M sends/s. Under subscribe-on-view, only ~520,000 people have a map open at any instant, giving ~2.08 M live subscriptions spread across 10 M publishers: an average of ~0.208 watchers per publisher, so 333,333 × 0.208 ≈ 69,444 sends/s.
That is a 19.2x reduction, exactly the write:read ratio, and not a coincidence: naive push pays publishers × F_live, subscribe-on-view pays viewers × F_live, and dividing cancels F_live to leave publishers / viewers. Subscribe-on-view converts a write-heavy fanout into a read-heavy one at the same exchange rate that made push wrong. Egress after the cut is ~16 Mbps average, ~39 Mbps at peak: trivial. That fanout number assumes “nearby” returns a useful list; the next section checks whether it even does.
Deep dive 2: how nearby is “nearby”
How many friends will “nearby” actually show? The answer is small enough to change the product, not just the system.
Model it in two steps. Assume 30% of your friends are in your metro. Model the metro as a 20 km disc with friends spread evenly, so the chance one is within 5 km is the area ratio π·5² / π·20² = 25/400 = 0.0625. Combined: P(within 5 km) = 0.30 × 0.0625 = 0.01875. Times the 4 live friends: 0.075 expected friends both live and within 5 km.
Treating the count as Poisson (rare independent events, known average), the chance of seeing zero is e^-0.075 ≈ 0.928, so 93% of sessions show an empty “nearby” list. That one number forces three design decisions:
- The default view cannot be “nearby.” It must be all sharing friends sorted by distance (about 4 dots, mostly far) with nearby as a highlight. A feature empty 93% of the time is not a default.
- The read path is trivial; the write path is everything. With 0.075 relevant results per query there is no ranking, no pagination, no candidate generation to build.
- “Recently nearby” is what makes it useful. Widen the time window, not the radius: a friend who was within 5 km in the last hour is the signal. That is a different structure: a small per-user ring buffer (a fixed-size array that overwrites its oldest entry) holding cell ids, not live dots.
Sizing that buffer: one entry per fix, one hour of history. 3,600/30 = 120 entries at 12 bytes each (8 B cell id + 4 B timestamp) is 1,440 B/user, so 10 M × 1,440 B ≈ 14.4 GB for everyone. So fourteen gigabytes buys an hour of “recently nearby,” still an in-memory structure with a TTL and no new storage tier. Since the answer set is tiny, the next question is whether we even need a spatial index to find it.
Deep dive 3: the geospatial index you probably do not need
A viewer’s answer set is friends ∩ nearby. Intersection does not care which side you start from, but the two orders cost wildly different amounts, and the reason is asymmetry: the friend set is bounded at ~200 while the nearby set is unbounded: however many people happen to be in a 5 km circle.
flowchart TD
A["Answer set = friends ∩ nearby"]
A --> FF["Friend-first<br/>200 hash lookups + 200 distance tests<br/>= 20 microseconds, exact set"]
A --> CF["Cell-first<br/>ask index for everyone in the 5km disc,<br/>then drop non-friends"]
CF --> S["~3,927 strangers fetched<br/>and distance-tested to find <= 4 friends"]
FF --> V["Use this: touches only your 200 friends"]
S --> P["Avoid: ~20x the work, and it pulls<br/>stranger positions into the service"]
Cell-first’s cost depends on publisher density. Derive it for the densest realistic place instead of borrowing the proximity chapter’s 50 businesses/km² (a different population). With 10% product penetration, 1.6 M Manhattan residents give 160,000 DAU; 4% connected and half sharing leaves ~3,200 concurrent publishers over 59.1 km²: about 54/km², call it 50. Over a 5 km disc (π·5² ≈ 78.54 km²) that is ~3,927 strangers fetched and discarded to find at most 4 friends: 3,927 / 200 ≈ 20x the work, plus it puts thousands of stranger positions into a service that has no right to them.
The rule:
Use a spatial index when the candidate set is unbounded. Use the relationship when it is bounded. A proximity service has no relationship to filter on, so it must index space. Nearby friends has a 200-element filter sitting right there.
The index comes back in two cases: “people near you” (an unbounded candidate set, a different product with a different privacy posture), and very large graphs: at F = 5,000 the friend-first scan is still only ~500 µs, but at F = 100,000 it is ~10 ms and you index instead. The crossover is set by F, not by geography. The index gone, the freshness of each dot is now the only thing standing between the design and a wrong answer, so we price the TTL next.
Deep dive 4: TTL, staleness, and adaptive intervals
A location stops being true the instant it is taken; what it has is an error that grows with time and speed. That turns the TTL into a distance budget, and the arithmetic is direct.
Declare a dot dead after two missed updates. At a 30 s interval a dot is deleted 30–60 s after the last fix, the same [h, 2h] window the chat-system chapter derives for heartbeats. Take the worst case, 60 s, times speed:
walking 1.4 m/s x 60 = 84 m
cycling 5 m/s x 60 = 300 m
driving 30 m/s x 60 = 1,800 m
The spread is the point: from one 60 s TTL a dot is 84 m wrong for a pedestrian but 1.8 km wrong for a driver, and one interval cannot serve both.
You cannot just update faster. Bounding a driver’s error at 50 m needs a 50 / 30 ≈ 1.7 s interval. With 2 M drivers that alone is 2,000,000 / 1.7 ≈ 1.18 M fixes/s (more than triple the whole current system) for 4.3x the total ingest, to serve one fifth of the population. Not worth it.
Send velocity instead, and let the client predict. Add vx, vy (4 bytes, a 14% record increase) so the client can dead-reckon: advance the last position along the last velocity as time passes, instead of leaving the dot frozen. Dead reckoning is exact on a straight road and wrong the moment it turns, so pair it with an event-driven send: transmit immediately when heading changes by more than 30°, and treat the timer as a floor. Blending urban turns (~every 40 s) and highway turns (~every 167 s) at 80/20 gives ~0.0212 events/s vs the timer’s 0.0333/s: a 1.57x cut for movers.
At the other end, stationary phones. GPS jitter makes a parked phone report a fresh position every 30 s, all noise. A displacement gate suppresses the send until the device actually moves, falling back to a bare keepalive (a payload-less packet proving the connection is alive) every 180 s, enough to keep a NAT entry alive. With 70% stationary, total ops drop from 333,333 to 7M/180 + 3M/30 ≈ 138,889, a 2.40x cut (2.86x in bytes, since a keepalive is 12 B, not 28).
Do not multiply the two savings. Both are measured against the same 333,333/s baseline, so 1.57 × 2.40 is meaningless. Compose them by applying each to the population it governs (movers send on events, everyone else keepalives): 3M × 0.0212 + 7M/180 ≈ 102,489, a 3.25x cut. Both tests run on the phone before anything is sent, so they cost the server nothing. The principle: the update rate should be a function of the world, not of a clock. The same TTL we just sized for freshness turns out to be the mechanism that enforces privacy, which is where we go next.
Deep dive 5: privacy, and why the TTL is the mechanism
Four properties, each enforced somewhere specific, none needing a new component.
Authorize at fanout, not at render. The sharing_edges check runs in the ingest path, when the server decides who to publish a fix to. A viewer’s client never receives a coordinate it is not entitled to, so there is nothing on the device to un-hide. A modified client, a patched app, a socket sniffer all gain nothing, because the bytes were never sent. That is the difference between a privacy feature and a privacy setting: a setting asks the client to behave, a feature makes misbehaving impossible.
Going invisible in bounded time. Invisible mode stops publishing, drops the subscription entries, and deletes the location_live cell. If that last delete fails, every dot already on a viewer’s screen still disappears within 60 s, because each dot carries age_s and the client refuses to render anything past the TTL. So the TTL that exists for freshness is also the privacy backstop: a system with no TTL has no bounded time-to-invisible, only a promise that a delete went through.
Reduce precision by truncating on the server. “City-level” sharing replaces the coordinate with the centre of a coarse geohash cell before publishing, so the client never sees the precise value.
| Mode | Cell | Cell size | What a viewer learns |
|---|---|---|---|
| precise | none | exact | Street address |
| neighbourhood | geohash 6 | 1,221 × 610 m | Which part of town |
| city | geohash 4 | 39 × 19.5 km | Which city |
| off | none | none | Nothing, and no “last seen” |
Truncate, do not add noise. This is the one place the obvious alternative is broken. Noise is averageable: a viewer who collects 100 fixes of a stationary user averages them and the random offsets cancel, so the true position emerges. Truncation is idempotent: the cell centre is the same value every time, so a hundred observations tell a viewer exactly what one did.
Make observation observable. GET /v1/sharing/audit returns who has seen your location and when: one row per (viewer, publisher) pair per day, and the only mechanism that lets a user verify the other three properties instead of trusting them. Privacy settled, the last question is what has to happen to every one of the 833,333 fixes as it lands.
Deep dive 6: the ingest path and the queue of depth one
Three things happen to every fix, and each has an obvious default that is wrong here.
Authorization: move the check off the hot path. One sharing_edges lookup per fix is 333,333 reads/s against a table whose rows change a few times per user per year. Move the check to subscription creation and cache the answer in the subscription entry: at ~17,361 map opens/s × 4 subscriptions that is ~69,444 checks/s, a 4.8x cut, against a cached row. One catch: revocation must not wait for the cache to expire, so it deletes the matching subscription entries directly, making the exposure window one propagation hop (milliseconds) instead of the 120 s TTL.
Rate limiting is not optional. Nothing forces a client to honour the 30 s interval; a buggy or modified client can publish at 100 Hz. Just 1% of clients doing that is 100,000 × 100 = 10 M updates/s: 30x the whole designed load, so a bad app release becomes an outage. A per-user token bucket (a counter refilled at a fixed rate, one token per request, bursts capped at bucket size; see the rate-limiter chapter) at one update per 5 s with a burst of 5 caps it, and costs nothing because the counter lives in the same shard as the location.
Backpressure: the only correct queue depth is one. The normal answer to overload is to buffer and catch up. Do not, because fixes are not events: they are successive estimates of a single value, so an older fix carries nothing a newer one lacks. A 10-deep queue at 30 s spacing means the head item is 300 s old, five TTLs past its own expiry. The correct structure is a single slot per user that a newer fix overwrites, which makes backpressure a no-op: under load you lose intermediate positions and keep the latest, exactly the degradation the product wants.
CPU is a rounding error. At ~2.3 µs per fix (parse, token bucket, slot write), the peak demands 833,333 × 2.3 µs ≈ 1.92 core-seconds/s, so 1.92 / 0.6 ≈ 3.2 cores at 60% utilization, a little over three cores for the entire application-level ingest of ten million publishers. (Quoting “under one core” uses the average load at 100% busy, the same mistake as before.) The ingest tier is not sized by this work at all; it is sized by the 20 M sockets in front of it, which the chat system already paid for. With every stage priced, we can collect the bottlenecks and see which one is actually large.
Bottlenecks and scaling
| Bottleneck | Number | Fix |
|---|---|---|
| Connection tier | 20 M sockets → 100 boxes at 200 k each (set by blast radius, per the chat-system chapter) | Cited, not re-derived. This is the fleet |
| Ingest CPU | 833,333/s at peak = 3.2 cores at 60% | Shard on user_id; the work is embarrassingly parallel |
location_live memory | 10 M × 200 B (record plus hash-map overhead) = 2.0 GB | One node per shard, no replication — a lost shard refills in 30 s |
| Subscription churn | 520 k viewers opening and closing | 120 s TTL and let it lapse; never require an explicit unsubscribe |
| Fanout | 69,444/s average, 173,610 at peak | Pub/sub keyed on publisher_id, colocated with the location shard |
| Dense venue | 50,000 publishers in one cell | Irrelevant: nothing is keyed by cell, so they hash uniformly across shards — the payoff for sharding on user_id |
The one genuinely large number is 20 million sockets, and it is inherited, not created. Everything this chapter adds (2 GB of state, ~187 Mbps in and ~39 Mbps out at peak, about three cores of ingest) is small. Because the state is small and disposable, the failures worth naming are not crashes but wrong answers shown with a straight face.
Failure modes
When the output is a dot on a map, the failure that matters is the system misleading a user while appearing healthy. There is no data-loss row and no consistency row: every failure here is about showing something wrong, not losing something. That is what a design with no durable hot path looks like.
| Failure | Symptom | Mitigation |
|---|---|---|
| Client clock skew | Dots from the future, or expiring instantly | Age is computed from server receive time; the client’s ts is advisory |
| GPS jitter while stationary | A parked user “walks” in circles | Displacement gate plus the reported accuracy radius |
| Gateway loss | 100–500 k clients reconnect at once | Jittered backoff (each client waits a random delay so they do not reconnect together); locations refill from the next fix |
location_live shard loss | Those users vanish from maps for up to 30 s | Accepted. No replication for a store whose contents expire in 60 s |
| Stale subscription | Pushes to a viewer who closed the map | 120 s TTL, and the gateway drops sends for closed views |
| Revocation race | A fix in flight when sharing is revoked | Revocation deletes subscription entries directly, so the window is one propagation hop, not the TTL |
| Tunnel or indoors | Last position freezes at a portal entrance | Show age_s, never interpolate past the TTL, mark the dot last-known |
Alternatives rejected
Each row names the number that kills the alternative.
| Alternative | Why not |
|---|---|
| Durable write per fix | 294 TB/year for a 280 MB working set, plus a replicated commit on the path of a value with a 30 s shelf life. (Not device count: that leg is 18 at worst, under one with any batching store) |
| Fan out on write to all friends | The write:read ratio is 19.2 the wrong way |
| Pure client polling over HTTP | Reopens a connection 500 k times a second and defeats the socket fleet you already pay for |
| Geospatial index as the primary lookup | The friend set is bounded at 200; a 20 µs scan beats an index that returns thousands of strangers |
Shard location_live by cell | 98,522 cross-shard migrations/s from moving users |
| Fixed update interval for everyone | 4.3x the ingest to bound a driver’s error, or 1.8 km of error to avoid it |
| Client-side privacy filtering | A modified client sees everyone. Authorize at fanout instead |
| Coordinate jitter for coarse sharing | Averageable over repeated observation. Truncate to a cell instead |
| Kafka as the location transport | Ordering and durability guarantees you are paying for and explicitly do not want |
Which assumptions hold the design up
Every design rests on assumptions, and the useful question is which ones, if false, make the design invalid (a box appears or disappears) instead of merely suboptimal (the machine count inside a box changes). Six here are load-bearing:
| Assumption | What it holds up | What replaces the design if false |
|---|---|---|
| A location is worthless within ~a minute | The absence of the whole storage layer — no WAL, no replication, no quorum, just a TTL’d memory slot | A time-series store with 294 TB/year of ingest, partitioning, compaction, retention — a different design with a durable write on the critical path |
| Losing a location is survivable | No replication of location_live; a lost shard just blanks its users for 30 s | Replicas, a quorum rule and failover; “one node per shard” becomes three plus a coordinator |
| Recipients are a bounded friend list, not a radius | The rejection of the spatial index, and the whole cell-lookup path | For “people near you” the candidate set is unbounded, the k-ring returns, a cell index appears, and the privacy posture changes |
| Publishers vastly outnumber concurrent viewers | Subscribe-on-view, and therefore the subscription registry | If everyone watched constantly, publishers/viewers → 1, the 19.2x saving vanishes, and unconditional fanout is correct |
| Privacy must survive a modified client | Authorization at fanout, which puts sharing_edges on the ingest path | Trust the client, broadcast everything, filter client-side; the audit endpoint has nothing to report |
| Clients already hold a socket paid for by chat | Treating the 100-box gateway tier as inherited | This chapter must justify 20 M connections on its own, and defend it against HTTP polling here |
The friend count of 200 is explicitly not load-bearing, even though it appears in the fanout arithmetic, the nearby estimate and the index decision. It has ~500x headroom (the design only flips at F = 100,000), and it cancels on both sides of the push-versus-pull comparison, so it cannot affect the largest decision even in principle. The rates (500 M DAU, 30 s interval, 2.5x peak, 28 B records) all scale linearly and move machine counts, not boxes.
Conclusion
- A nearby-friends feature is a presence system with a coordinate, not a geospatial search. That reframing removes the storage layer.
- Freshness beats durability: a location lives in one in-memory TTL’d slot, with no write-ahead log, replication, or quorum. A year of durable history would be ~294 TB against a 280 MB working set, to preserve data wrong in 30 seconds.
- Shard on
user_id, not on cell, so moving users never migrate between shards (which would cost ~98,522 migrations/s). - Subscribe-on-view turns a write-heavy fanout into a read-heavy one, a 19.2x cut equal to the write:read ratio that made naive push wrong.
- Skip the spatial index while the recipient set is a bounded friend list: a 200-entry scan is 20 µs and returns exactly the right set, where a cell lookup fetches thousands of strangers. Index only when the set is unbounded.
- The TTL is doing three jobs at once: freshness, the deletion of stale dots, and a bounded (60 s) time-to-invisible that survives a failed delete.
- Adaptive, world-driven update rates (dead reckoning, event-driven sends, a displacement gate) cut ingest ~3.25x on the client, for free, but the savings do not multiply, because they share a baseline.
One line to remember: when data is worthless in a minute, freshness beats durability, and every hard choice here falls out of taking that seriously.
Further reading
- Proximity service: derives every geohash cell size cited here and the k-ring lookup.
- Chat system: owns the 20 M-socket connection tier and the heartbeat interval this chapter reuses.
- News feed: owns the push/pull cost model this chapter inverts.
- Google Maps: takes the same location firehose and turns it into traffic.