Designing a File Storage & Sync Platform

Design a system where users upload files from any device, access them from anywhere, see changes sync automatically across devices, and share files with fine-grained permissions.

This problem decomposes into several distinct engineering challenges: handling large file uploads reliably, storing petabytes of data durably, keeping multiple devices in sync, resolving conflicts when two devices edit the same file, and controlling access to shared content. Each challenge requires a different architectural solution.

'File operations' means more than uploading and downloading content. It includes create, rename, move, delete/restore, share/revoke, and creating a new version — each of which must be tracked, synchronized across devices, and made recoverable.

Functional requirements define what the system does. Non-functional requirements and scale estimation shape every architectural decision.

Functional Requirements

Upload files (including very large files, with resumability). Download files from any device. Sync changes automatically across all user devices with incremental transfers, including offline edits and conflict handling. Share files with access control (viewer/editor) and revocation. Maintain file version history and support restoring deleted files or old versions. Support folder hierarchy and operations: create, rename, move, delete.

Non-Functional Requirements

Durability — extremely high durability, with a recovery/backup strategy appropriate to the product, rather than an absolute 'never lost' guarantee. Availability — access survives individual component failures. Integrity — bytes downloaded match bytes uploaded. Sync freshness — bounded, not instant, propagation across devices. Security — authorization on every file operation. Bandwidth efficiency — sync transfers only what changed where practical. Recovery objectives — defined RPO/RTO for disaster scenarios. Scalable metadata operations — namespace and sharing queries stay fast as data grows.

Scale Estimation

500M registered users, 100M DAU. ~200M uploads/day (~2,300/sec). Average file size 5 MB, up to 50 GB maximum. ~1 PB new data per day. Downloads at 5:1 read-to-write ratio (~1B/day). 100M devices syncing changes. These are illustrative design assumptions used to justify architectural decisions in this walkthrough — not characteristics of any specific commercial product, and not universal for all file-storage systems.

Before designing the internals, define the contracts clients depend on.

The simplest possible design: one API server, one database, local disk storage.

A small file uploads in a single HTTP request. The API writes metadata to the database and bytes to disk. For small files and low traffic, this works fine. But upload a 2 GB file: the API server receives the entire file body, holding a long-lived connection and application bandwidth/memory just to relay bytes it does nothing else with. At 71% progress, the connection drops. The entire 2 GB must be re-uploaded from scratch.

Proxying large file bytes through the application tier is undesirable for several concrete reasons: it consumes application bandwidth proportional to file size, it holds long-lived connections that pressure horizontal scaling, retries become expensive because a dropped connection loses all progress, and it couples data transfer availability to control-plane availability — if the application tier is degraded, uploads fail even though object storage itself might be healthy.

The foundational architectural decision in a file storage system: file bytes and file metadata are fundamentally different workloads.

Object storage (S3-like) is purpose-built for large binary blobs: durable across multiple availability zones, cheap per gigabyte, horizontally scalable. The metadata database stores everything about the file except the bytes: file_id, name, owner, folder, version, content_hash, timestamps, permissions, storage_key, lifecycle state.

The consistency gap

Because metadata and file bytes live in two separate systems, there is no simple atomic transaction spanning both. A file's lifecycle state exists precisely to manage this gap: CREATED → UPLOADING → VERIFYING/COMPLETING → AVAILABLE, with FAILED / EXPIRED / DELETING / DELETED as the other paths a record can take. Only once storage completion is verified does the metadata version get published as AVAILABLE — this is what prevents a partially-uploaded object from ever appearing as a valid, downloadable file.

For the exact reconciliation mechanics (how the system detects and recovers from metadata/object divergence), see Architecture Deep Dive → Metadata/Object Consistency.

Even with metadata separated, the application server is still receiving all file bytes and forwarding them to object storage. It does zero useful work on those bytes.

The solution: in the normal large-file path, the File Service does not proxy bytes. The client calls the File Service to initiate an upload. The service authenticates the user, chooses a server-controlled object key, creates the file record, and generates a temporary presigned URL to object storage. The client uploads bytes directly to object storage. The application tier now handles only orchestration and authorization instead of relaying every byte.

Security requirements

A presigned URL is generally a bearer capability: anyone who obtains it may be able to use it until it expires, subject to its signature/policy constraints — it is not cryptographically bound to the original end-user's identity unless the design explicitly adds another binding mechanism. This shapes the required controls: short expiry, HTTPS only, a server-generated (never client-supplied) storage key, least-privilege operation scoping, expected content-length/type/checksum constraints where the provider supports them, never logging or sharing signed URLs unnecessarily, and always authorizing before issuance.

Direct upload solved the proxy problem. But a 10 GB file as one PUT request still fails completely if the connection drops.

The client splits the file into chunks — the right chunk size depends on provider limits, file size, network conditions, client memory, and desired parallelism, not a fixed universal number (see Trade-offs & Decisions → Chunk/Part Size Selection). The upload service initiates a multipart session and returns per-chunk presigned URLs. Each chunk uploads independently and in parallel. If one chunk fails, the others already stored are unaffected — only the failed chunk retries.

Three related-but-distinct concepts are worth separating clearly: the application chunk the client splits the file into, the object-storage multipart part the provider actually stores (usually the same thing, but not guaranteed to map 1:1 in every implementation), and a sync/delta block used later for change detection — these serve different purposes and shouldn't be casually conflated.

When all chunks arrive, the client submits its recorded parts to the completion endpoint (idempotent — a retried completion is safe). The upload service reconciles what storage actually has (not just what the client's local counter claims) before instructing object storage to assemble the final object, then updates the metadata database: lifecycle state moves toward AVAILABLE only after integrity is verified. For the full session-state-machine and reconciliation mechanics, see Architecture Deep Dive → Multipart Upload State Machine.

Downloads follow the same principle: the File Service never proxies file bytes in the normal path.

The client requests a download. The File Service checks authorization, then returns a presigned download URL. The client fetches bytes directly from object storage. For popular, shareable content, a CDN can cache bytes at edge nodes for lower latency — but private-file CDN delivery is not automatic. It requires its own authorization design: a signed CDN URL or cookie, a private origin, and a cache-key/access policy that cannot leak one user's private object to another.

HTTP range requests are an important large-file capability: resuming an interrupted download from the last received byte, fetching only a portion of a large file, and streaming/preview use cases without downloading the whole object first.

Multiple devices must stay in sync automatically and efficiently — this is a reconciliation protocol, not just 'notice a change and copy the new bytes.'

Local change discovery

Each device's sync client watches the local filesystem (inotify/FSEvents/ReadDirectoryChanges-style APIs) for create/update/rename/move/delete events, with a periodic full scan/reconciliation as a safety net in case a filesystem event is missed.

Remote change discovery

A push notification is a wake-up hint, not the source of truth. The device fetches durable, ordered changes using GET /sync/changes?cursor=..., and persists its cursor only after successfully applying those changes.

Local sync state and applying an update

Each device persists a small local sync database per file: file_id, local path, server version it last synced, a content fingerprint, its last acknowledged cursor, and any pending local operation. When submitting an edit, the client includes the base_version it edited from. The server atomically checks whether that base is still current — if yes, it commits a new version; if the server has already moved past that base, this is a divergence (see Conflict handling below), not a silent overwrite.

Delta transfer, correctly framed

A whole-file content hash tells the system that a file's content changed — it does not, by itself, identify which bytes changed. Genuine delta transfer requires block-level signatures or a rolling-checksum/content-defined-chunking approach; otherwise the system falls back to transferring the whole new file. For compressed, encrypted, or otherwise binary-dense formats, a small logical edit can change much of the byte representation, so delta transfer isn't always a net win. See Architecture Deep Dive → Delta Sync Algorithms for the mechanics.

Metadata operations and offline edits

Sync propagates more than content updates — rename, move, delete, restore, and sharing/permission changes all flow through the same durable change log. Devices queue local changes made while offline and reconcile them once connectivity returns, following the same base-version check as an online edit.

Every edit creates a new version. The metadata database records version lineage (each version's parent). Object storage retains version objects per retention policy. Users can view history and restore prior versions. Version lineage — not timestamps — is what lets the system detect a genuine conflict.

When two devices edit the same file offline, both submit a version claiming the same parent. The sync service detects this fork and, by default, preserves both as conflict copies rather than silently discarding one — reducing the risk of silently overwriting a user's work, though overall durability still depends on the underlying storage and backup strategy. The user reviews and merges. For the full mechanics, see Architecture Deep Dive → Conflict Detection and Optimistic Concurrency.

Sharing introduces authorization as a separate concern from file storage.

A dedicated auth service manages the permissions table. When Alice shares a file with Bob, the auth service records the relationship. Every file operation goes through an ACL check, and a scoped, time-limited signed capability is only generated after that check passes — direct byte access after authorization is the design goal, not something being avoided.

Revocation nuance

Once a download URL has been issued, revoking the metadata ACL does not instantly invalidate that already-issued capability — it remains usable until it expires. Short-lived signed URLs, never issuing long-lived public URLs for private files, and re-authorizing on every new URL request are the primary mitigations; a stricter proxy/token-validation path is needed only if immediate revocation is a hard requirement.

Folder sharing

Sharing a folder introduces inherited permissions across its subtree. Permissions may be explicit on a specific file, inherited from a parent folder, or overridden depending on product semantics — a full enterprise IAM model is out of scope here, but the inheritance/override distinction matters for correctness.

Every failure has a known recovery path. Nothing is silently lost.

Upload interrupted

Chunked upload means only the failed chunk retries. The client reconciles which parts the storage layer actually has — not just its local record — before resuming from where it left off.

Object storage unavailable

Retry with bounded exponential backoff and jitter. Metadata remains accessible, so users can browse their files even if uploads/downloads are temporarily degraded — but multi-AZ durability does not mean data can never be lost, and retries must not be unbounded.

Metadata DB unavailable

Queueing writes in an in-memory application buffer can silently lose acknowledged mutations if the process dies. Prefer failing over to a healthy standby where configured, or safely rejecting/retrying mutations. Authorization should fail closed if current permission state cannot be safely determined, rather than trusting a possibly-stale replica.

Concurrent modification

Version preconditions and lineage detect the conflict — a write is only accepted if its base version is still current. Both versions are preserved as conflict copies rather than one silently overwriting the other; see Architecture Deep Dive → Conflict Detection and Optimistic Concurrency and Failure & Scale Scenarios → Concurrent Modification for the full mechanics.

Every component exists because we saw the problem it solves.

Upload path: Client → File Service (auth + metadata + presigned URLs) → Client uploads chunks directly to Object Storage → completion callback verifies integrity → File Service marks AVAILABLE.

Download path: Client → File Service (auth + ACL check) → presigned download URL → Client fetches from Object Storage or CDN.

Sync path: Device change → durable change log / sync event stream → other devices replay via cursor → conflicts detected by version lineage → conflict copies preserved.

Supporting infrastructure: a background job queue for thumbnailing/indexing/notifications/scanning, versioned object keys for immutable per-version storage, upload-session reconciliation and expiry cleanup, an optional CDN for cacheable content, and a cleanup/lifecycle worker for orphan and expired-version garbage collection.