How to design a cloud file-storage service

A file-sync service (Dropbox, Drive, iCloud) is two stores that must agree but cannot share a transaction: mutable authoritative metadata in a database, immutable content-addressed chunks in object storage.

The core split

• Metadata: small, hot, strongly consistent (linearizable per file). 0.02% of bytes.
• Object storage: huge, cold, immutable, eventually consistent. 99.98% of bytes. A blob is named by its own hash, so a replica has the right bytes or not that name at all, never wrong bytes.
• A chunk is one contiguous piece of a file cut by a boundary rule, stored and transferred independently. A file is a list of chunks.
• Load-bearing fact: no transaction spans the two stores, so a crash can land between the two writes. Almost every hard property follows from this.

Back of the envelope

• 50 M users, 10 M DAU, 10 GB quota at 20% used = 100 PB corpus.
• 100 M changes/day = 1,000 writes/s (3,000 peak). Metadata tier is easy.
• Naive re-upload = 209 TB/day (16.7 Gbps sustained, 50.1 peak). The hard part.
• 10% of changes (large 20 MB files) carry 96% of bytes. Optimize only the large-file path; a 100 KB file is smaller than one chunk and has nothing to diff.
• Metadata: 284 B/row × 47.8 B files = 13.6 TB + 4 TB chunk map = 17.6 TB, 0.018% of corpus. Shard on owner_id.

Content-defined chunking (CDC)

• Fixed-block boundary is a function of offset; CDC boundary is a function of content.
• Write amplification = bytes transferred / bytes changed. 1 is perfect.
• A 1 KB insert at the head: fixed blocks relabel every block → re-upload 100% (50,000×). CDC boundaries move with their content → re-upload 2 of 50 chunks = 4%. 25× better.
• Mechanism: slide a w-byte window, keep a rolling hash (mix new byte in, old byte out, constant time), cut when low b bits are zero. Cuts land ~every 2^b bytes.
• 1 MB chunk derived: 48-byte window, 2^19 mask (524,288 B), 512 KiB floor, 4 MiB cap → mean 524,288 + 524,288 = 1,048,576 = 1 MiB.
• Floor stops tiny chunks; cap stops runaway chunks on long zero runs.
• Why 1 MB not 64 KB: index size and PUT rate scale as 1/chunk_size. At 64 KB the map is 62.5 TB and PUTs go 650/s → 10,156/s, while delta saving saturates (an edit is already 2 chunks).

Conflicts

• Last-write-wins (LWW) keeps the later write, discards the other: 0.1% of changes conflict = 100,000/day = 36.5 M edits/year destroyed silently. Never in a file store.
• Detect with an 8-byte compare-and-swap on head_rev (commit only if parent is still head), not a version vector (960 B for 20 users × 3 devices, grows forever). One shard owns each file, so there is a single serialization point; CAS is 120× smaller.
• Commit is optimistic concurrency: read head, upload, commit; 409 returns current head and the client reconciles. Nothing is locked during the (minutes-long) upload.
• Resolve by conflict copy: keep both, rename with device + timestamp. Costs 76.3 TB/year (0.076%), loses nothing. Suppress when both have identical content hashes.

Notification: cursor, never the tree

• A cursor is a per-user monotonic seq into the append-only journal (owner_id, seq, file_id, op, rev). Client sends last seq, gets everything after. Resumable, idempotent, ordered within a user.
• Full-tree sync = 272 KB vs delta 1.5 KB = 181×; fleet-wide it would be a third of all traffic.
• Wake devices with push (WebSocket): ~5,556 connects/s vs long-poll 333,333/s vs short-poll 666,667 QPS (99.85% empty, 15 s latency).
• Socket tier: 20 M sockets, sized 40-200 boxes by file descriptors, TLS CPU, blast radius, not RAM.
• Reconnect storm: dropping all sockets = 4 M connects/s. Fix with full jitter over 10 min → 33,333/s. Beyond the 30-day journal window the cursor 410s → full resync (rate-limit it).

Storage, dedup, and its side channel

• Deduplication stores one physical copy of a chunk many accounts hold. 25% dup rate → corpus 100 PB → 75 PB.
• Erasure coding RS(10,4): 10 data + 4 parity, survives any 4 losses at 1.4× vs 3× for RF 3. 75 PB → 105 PB (vs 225 PB). Repair reads 10× the lost bytes, so replicate last 30 days, code the tail.
• Client-side dedup is a membership oracle: 4 MB new chunk = 3.2 s, existing = 0.05 s = 64× timing signal. One probe confirms a specific leaked file’s presence.
• Fix: server-side dedup only (client always uploads, server discards dup). Gives up 15% bandwidth, keeps all 35 PB storage saving, closes the channel.

Two stores, one commit + GC

• Only one write order survives a crash between the two writes:

• Orphans: 100,000/day, 36.5 TB/yr (0.037%). Retries write same hash to same key, so they never compound.
• Deleting: never reference counting (needs a transaction across two differently-sharded stores; a dropped increment deletes user data). Use mark and sweep, weekly, 4 TB at 1 GB/s = 1.1 machine-hours.
• Mandatory grace period (24 h): never sweep a chunk younger than the longest upload + retries, or GC deletes a file mid-upload.

Write path:
 client --1. probe hashes--> metadata API
        --2. PUT missing chunks--> object store (content-addressed)
        --3. commit CAS(parent_rev)--> metadata DB + journal
Read path:
 journal --> notify "cursor moved" --> other devices
 devices --GET /delta?cursor--> API, then fetch chunks via CDN

Gotchas

• A shared folder is an edge (shares row), not a copy. Duplicating rows makes a 500-person share do 500 writes/change (1,000/s tier → 500,000/s).
• A hot chunk (viral file in 2 M accounts) cannot be split by sharding (one hash → one place); put a CDN in front and replicate it.
• Validate the chunk list server-side: a commit naming a missing chunk must fail, not create a broken revision.
• Four load-bearing assumptions: skewed byte distribution, edits include inserts, one shard per file, nothing silently lost. Break any and the design is invalid, not just suboptimal.