How to design a nearby-friends feature

Nearby-friends is a presence system with a coordinate attached, not a geospatial search: a location is worthless a minute after it is taken, so nothing about it is stored durably.

Core terms

• Fix: one position reading from a device (GPS sample + timestamp).
• age_s: seconds since a fix was taken; returned with every dot, the most important response field.
• TTL: expiry stamped on a stored value; the store deletes it after that many seconds.
• Working set: one current position per publisher, live in memory.
• Default “nearby” radius: 5 km.

Scale numbers

• 500 M DAU, 20 M concurrent sockets, 10 M concurrent publishers.
• Ingest: 10M / 30s ≈ 333,333 fixes/s average, ~833,333/s at 2.5x peak.
• Record: 28 bytes (user_id 8, lat/lng 4 each as int32 microdegrees, recorded_at 8, accuracy 2, flags 2).
• Working set: 10M × 28B = 280 MB (fits one machine’s RAM).
• Sizing rule: fleets/cores on the 833,333/s peak; bytes-per-day/cost on the 333,333/s average.

Why no database

• Durable history: ~9.3 MB/s → ~294 TB/year (~883 TB with 3 replicas) to preserve data wrong in 30 s.
• Ratio settles it: 294 TB/year against a 280 MB working set (~1M working sets/year).
• Real killer is coupling: a replicated commit on the critical path of a fix nobody will read.
• Bandwidth is a non-issue: peak ingest ~187 Mbps, ~19% of one 1 Gbps NIC.
• The “81 devices” disk argument is wrong (queue-depth-1 misread as capacity); real answer is ~6 devices, 18 replicated, and it collapses further with any LSM batching store.

The three stores

• Shard location_live on user_id, not cell: cell-sharding costs ~98,522 shard migrations/s from moving users. user_id home shard never changes.

Push vs pull: subscribe-on-view

• Rule (from news-feed): compare w / r; follower count F cancels.
• Here w / r = 57.6 fixes / 3 opens ≈ 19.2 per day → points against push.
• Subscribe-on-view: opening the map subscribes you to friends’ topics; closing lets it lapse via TTL. Push only along edges someone is watching.
• Cuts fanout from ~1.33 M sends/s (naive) to ~69,444/s: a 19.2x reduction = the write:read ratio.

Nearby is nearly always empty

• P(within 5 km) = 0.30 (in metro) × 0.0625 (area ratio 5²/20²) = 0.01875; × 4 live friends = 0.075 expected nearby friends.
• Poisson: e^-0.075 ≈ 0.928 → 93% of sessions show an empty list.
• Default view = all sharing friends sorted by distance, nearby as a highlight. Write path is everything; read path is trivial.
• “Recently nearby” widens the time window, not radius: per-user ring buffer of 120 cell ids (1 hr) ≈ 14.4 GB total.

Skip the spatial index

• Answer set = friends ∩ nearby; friend set bounded (~200), nearby set unbounded.
• Friend-first: 200 hash lookups + 200 distance tests ≈ 20 µs, exact set. Use this.
• Cell-first: ~3,927 strangers fetched to find ≤4 friends ≈ 20x the work, and pulls stranger positions into the service.
• Rule: index space only when the candidate set is unbounded. Index kicks in at large F (~100,000 → ~10 ms scan), set by F not geography.

TTL, staleness, adaptive rates

• Dot dead after two missed updates: at 30 s interval, deleted 30–60 s after last fix.
• 60 s TTL as a distance budget: walking 84 m, cycling 300 m, driving 1,800 m wrong.
• Can’t just update faster: bounding a driver at 50 m needs ~1.7 s interval → 4.3x total ingest for 1/5 of users.
• Send velocity (vx, vy, +14% record) → client dead-reckons; event-driven send on >30° heading change, timer as floor.
• Displacement gate suppresses jittery stationary sends; keepalive (payload-less) every 180 s holds the NAT entry.
• Savings do not multiply (shared baseline); composed properly ~3.25x cut, all on the client, free to the server.

Privacy (TTL is the mechanism)

• Authorize at fanout, not render: sharing_edges checked in ingest; entitled bytes are the only bytes sent. A modified client gains nothing.
• Invisible in bounded time: stop publishing, drop subscriptions, delete cell; if delete fails, dots vanish within 60 s because clients refuse to render past the TTL.
• Reduce precision by server-side truncation, never noise: noise is averageable, truncation is idempotent.
• Audit endpoint (GET /v1/sharing/audit) makes observation observable.

Ingest path gotchas

• Authorization: move off hot path — cache the check in the subscription entry (~69,444 checks/s vs 333,333). Revocation deletes subscription entries directly (window = one hop, not 120 s).
• Rate limiting is mandatory: per-user token bucket (1 update/5 s, burst 5); 1% of clients at 100 Hz = 10 M/s, 30x designed load.
• Backpressure: queue depth of one. Fixes are successive estimates of one value; a newer fix overwrites the slot. Never buffer — a 10-deep queue means the head is 300 s (5 TTLs) stale.
• CPU is a rounding error: ~2.3 µs/fix → ~3.2 cores at peak. Tier is sized by the 20 M sockets (inherited from chat), not this work.

Load-bearing assumptions

• A location is worthless within ~a minute → removes the whole storage layer.
• Losing a location is survivable → no replication; a lost shard blanks users for 30 s.
• Recipients are a bounded friend list, not a radius → no spatial index.
• Publishers vastly outnumber concurrent viewers → subscribe-on-view.
• Privacy must survive a modified client → authorize at fanout.
• Sockets already paid for by chat → 100-box gateway tier inherited.
• The friend count 200 is not load-bearing (~500x headroom, cancels in push/pull).