File Storage & Sync: Architecture Deep Dive

The metadata database is the control plane of the entire system. Its schema determines what operations are fast, what queries are possible, and how correctly the system tracks state that spans devices.

Running example used throughout this Deep Dive: a user edits report.docx on a laptop while a phone is offline. Both devices reference the same file_id; only the version pointer, cursor and sync state differ per device.

Why stable identity matters

file_id is assigned once and never changes. Rename and move are metadata mutations — they update filename or parent_folder_id on the existing record — not a copy of the file bytes to a new object. This is what lets a sync client recognize 'this is still the same file, just renamed' instead of treating a rename as a delete-plus-create.

Folder operations

Rename and move update the folder's own row and leave every child's parent_folder_id untouched (a folder move does not require touching every descendant file). Deleting a folder subtree is typically handled as a tombstone on the folder plus asynchronous cascading cleanup, not a synchronous recursive delete of every child in the request path. Listing children is a query by parent_folder_id, paginated. If the product requires unique sibling names, enforce a uniqueness constraint such as (parent_folder_id, normalized_name) rather than relying on full-path uniqueness.

Metadata DB and object storage are separate systems. There is no ordinary atomic transaction spanning both — this gap is one of the central correctness problems in the whole design.

Because a single transaction cannot cover 'write bytes to object storage' and 'commit metadata row' atomically, the system must be built assuming these two writes can become momentarily inconsistent, and must reconcile rather than pretend the gap does not exist.

The state-machine pattern

1. Create metadata/session in a pre-available lifecycle state. 2. Upload object bytes/parts to storage. 3. Verify storage completion (existence, size, checksum). 4. Atomically publish the metadata version pointer as AVAILABLE only after verification succeeds. 5. Clean up incomplete/orphaned state later via a background reconciliation job.

Two directions of inconsistency

Presigned URLs are the mechanism that lets clients talk directly to object storage without the application server touching bytes — but they carry security properties that are easy to overstate.

A presigned URL is a temporary, scoped credential. The File Service generates it using its own storage credentials, embedding the operation (PUT or GET), the specific storage key, and an expiration timestamp. The client uses it directly; the application server never sees the file bytes.

Bearer-capability semantics

A presigned URL is generally a bearer capability: whoever holds the URL can use it until it expires, subject to its signature and policy constraints. It is not cryptographically bound to the original requester's identity unless the design explicitly adds another binding mechanism (such as requiring the client to also present a session token, or checking client IP/UA against issuance context in a stricter deployment).

Practical security controls

Short expiry (minutes, not hours). HTTPS only. Server-generated storage key (never accept a client-supplied path). Least-privilege operation scoping — a PUT-scoped URL should not be usable for GET. Expected content-length/content-type/checksum constraints where the storage provider supports them. Authorization performed before URL issuance, not after. Never log or share signed URLs unnecessarily — they behave like credentials in transit. CORS configuration matters for browser clients that upload directly to storage.

Revocation limitation

Once a URL has been issued, revoking the underlying metadata permission does not retroactively invalidate a URL that is still within its validity window. Short TTLs and object/version-scoped policies are the primary mitigation, not a promise of instant revocation — see Failure & Scale Scenarios → Permission Revoked While Signed URL Exists for the full scenario.

Multipart upload is a protocol between the client, the upload service, and object storage. Understanding the session lifecycle — not just the happy path — is essential.

Session states

INITIATED → UPLOADING → COMPLETING → AVAILABLE, with ABORTED / EXPIRED / FAILED as terminal alternatives. Initiation should be idempotent (a retried initiate request for the same client-supplied idempotency key returns the existing session rather than creating a duplicate one). Each session has clear ownership (which user/device/upload_id it belongs to) so that a client cannot accidentally complete or abort another session.

Lifecycle in detail

1. Client calls File Service to initiate an upload (idempotency key supplied). 2. File Service calls object storage's CreateMultipartUpload → returns an upload_id. 3. File Service issues one presigned URL per part (or per batch, refreshed on demand). 4. Client uploads parts, in parallel where useful, retrying only failed parts. 5. Where the storage provider supports it, the client or service can list already-uploaded parts to reconcile state after a reconnect — the application's own chunks_received counter is not necessarily authoritative if clients upload directly to storage, since a part could complete at the storage layer without the application ever hearing about it. 6. Client calls the completion endpoint with the part list; completion itself must be idempotent, since network retries can deliver it more than once. 7. On success, the service publishes the metadata version as AVAILABLE (see Metadata/Object Consistency).

ETag is not universally MD5

Not every ETag is an MD5 hash or a general-purpose content-integrity checksum. ETag semantics vary by storage provider, encryption mode, and multipart assembly behavior — for multipart objects in particular, the ETag is frequently not a simple hash of the final content. Where end-to-end integrity verification is required, use explicit checksum APIs/headers rather than relying on ETag equality.

Cleanup

Sessions that are never completed should expire on a lifecycle policy (e.g., after 24–48 hours of inactivity), triggering an abort call to storage to release the uploaded-but-unassembled parts. See Failure & Scale → Orphaned Multipart Upload Explosion for what happens when this cleanup is neglected at scale.

Integrity has several distinct layers, and conflating them leads to false confidence.

The verification flow

Client computes an expected checksum before upload → uploads bytes/parts → storage validates where the API supports it → the completion step records and/or verifies the final object's integrity against the expected value before the metadata version is published as AVAILABLE.

Large-file download deserves the same resumability thinking as upload.

HTTP Range requests let a client fetch a specific byte range of an object. This supports: resuming an interrupted download from the last received byte, fetching only a portion of a large file (e.g., video/document preview), and downloading multiple ranges in parallel for throughput on large objects.

A client resuming a partial download should re-verify integrity of the completed download against the expected whole-object checksum once all ranges are assembled locally — range delivery does not by itself guarantee the client's final reassembled file is correct if earlier ranges were corrupted in a way transport integrity didn't catch.

Each device's sync client is itself a state machine, not just a background thread that copies bytes.

Continuing the running example: report.docx has file_id F1. The laptop is SYNCED at version v2. The user edits it locally, and the client transitions to LOCAL_DIRTY. Meanwhile the phone is offline and unaware of any of this.

Local client state per file

Stable remote file_id, local path, base/server version the local copy was derived from, local content fingerprint, sync status, any pending local mutation (create/edit/rename/move/delete), and the last acknowledged server cursor.

Possible states

When the phone later reconnects, it fetches durable changes since its last cursor (see Sync Change Log & Cursor), discovers the laptop's new version, and — if it also has local pending edits from while it was offline — transitions to CONFLICT rather than silently overwriting either side.

The server-side change log is the durable source of truth that sync clients reconcile against; push notifications are only a hint to go check it.

Change log entry

Each entry conceptually identifies: a monotonically ordered cursor within its scope (e.g., per user or per shard), the file/entity ID it concerns, the operation type (create/update/rename/move/delete/restore/share-change), the resulting version, and any metadata needed to reconcile (e.g., parent_version_id).

Cursor semantics

A cursor represents 'all changes up to X have been durably processed by this device.' It must only be persisted locally after the corresponding changes have actually been applied — persisting it earlier risks silently skipping a change if the client crashes mid-apply. Duplicate delivery of the same change must be safe to reprocess (see Ordering & Idempotency). Push notifications are a wake-up hint that something changed; the client still calls GET /sync/changes?cursor=... to fetch the durable, ordered truth rather than trusting the push payload alone.

Cursor retention and full resync

If a device stays offline longer than the server retains change history, incremental replay from its old cursor may no longer be possible. The fallback is: fetch a full metadata namespace snapshot, compare it against local state, reconcile any pending local operations queued while offline, and establish a fresh cursor to resume incremental sync from that point forward. See Failure & Scale → Client Offline Beyond Change-Log Retention.

Partitioning alone does not guarantee correctness — it only bounds where ordering can be reasoned about.

Partitioning a user's sync stream to one partition can provide ordered processing within that partition, when the underlying log/queue technology's ordering semantics actually support it for that partition key. It does not, by itself, mean a user's entire namespace is processed through one globally serialized execution path, and it does not eliminate the need for idempotency.

Retries are inevitable in a distributed sync protocol: a client may resend a completion call, a push notification may be delivered twice, or two devices may submit overlapping operations concurrently. Every mutating sync operation needs an operation ID (for retry-safety) and a version precondition (for correctness against concurrent writers) — ordering within a partition reduces surprises, but does not replace these checks.

A whole-file hash tells you that content changed — not which bytes changed. Delta transfer requires a different mechanism entirely.

Whole-file replacement

Simplest approach: if the hash differs, upload/download the entire new file. Good default for small files, or formats where binary deltas are ineffective (compressed, encrypted, or otherwise densely-packed binary formats, where a small logical edit can change a large fraction of the byte representation).

Fixed-size block delta

Split the file into fixed-size blocks and compare per-block hashes; only changed blocks transfer. Weakness: an insertion near the start of the file shifts every subsequent block boundary, so a single-character insert can appear to change every following block even though the actual edit was tiny.

Rolling checksum / content-defined chunking

Uses a rolling (weak) checksum plus a strong hash, conceptually similar to rsync's algorithm, to find matching regions even when byte offsets have shifted. This can dramatically reduce upload size for large files with localized edits, at the cost of additional CPU for chunk-boundary detection and additional metadata to track chunk references.

Conflicts are detected by comparing version lineage, not timestamps or arrival order.

The write rule

A client submits a mutation based on the version it last saw — its base_version (e.g., v5). The server accepts the mutation as the next current version only if the file's current version is still v5. If the server has already advanced to v6 (because another device committed first), the incoming mutation is divergent: the client based its edit on a version that is no longer current.

Resolution strategies

Conflict copy: preserve both versions, one as 'current' (typically first to commit) and the other saved separately for the user to review/merge. Last-write-wins: the later write simply becomes current, discarding the other — a product policy choice, not a correctness requirement. Three-way merge: for mergeable text formats, use the common ancestor (the shared parent version) to merge non-overlapping changes automatically; not applicable to binary formats, and out of scope for collaborative real-time editing (CRDT/OT), which belongs to a different problem space.

Sync is not only about content changes. Rename, move and delete are metadata mutations that must appear in the same durable change stream.

Rename and move

Because file_id is stable, a rename or move only updates filename or parent_folder_id — the version lineage and content are untouched. A sync client that sees a rename event for a file_id it already has locally should rename its local copy in place rather than treating it as a new file.

Delete

Prefer tombstone/soft-delete semantics over immediately purging the metadata row. This lets an offline device that reconnects later learn 'this file was deleted' instead of concluding 'I've never heard of this file, I should re-upload my local copy as new' — which would silently resurrect deleted content.

Delete/edit race

A device is offline when a file is deleted elsewhere, and later reconnects with a local edit already in flight. Product policy must decide the outcome explicitly — for example: restore the file as a conflict copy, reject the edit as stale, or create a recovered file — but the underlying mechanics (stable file_id, tombstone visibility, version checks) are what make any of those policies safe to implement.

Committed versions should be simple and safe to store: one immutable object per committed file version.

The file's current metadata always points to its current version_id. Older versions remain addressable independently, subject to a retention policy (see Trade-offs & Decisions → Version Retention Policy). Deleting the current metadata pointer or soft-deleting a file should not immediately destroy the underlying recoverable bytes unless the retention policy specifically requires immediate hard deletion — this separation is what makes 'restore a deleted file' and 'restore a prior version' both possible.

Deduplication is an optional storage optimization, not a foundational requirement of this design.

Deduplication also complicates deletion: a deduplicated object needs reference counting across every owner referencing it, and garbage collection must never delete an object while any reference remains.

Authorization touches every file operation, and its failure modes matter as much as its happy path.

Permission sources: ownership, direct file share, folder-inherited permission, with product-defined override rules for how conflicting inherited vs. direct permissions resolve. Access levels are typically viewer/editor (and owner).

Revocation nuance

Revoking an ACL updates the metadata record immediately, but it cannot retroactively invalidate a presigned URL that was already issued and is still within its expiry window — see Presigned URL Security. Where immediate revocation is a hard requirement, the design needs shorter-lived URLs, a stricter token-validation/proxy path for sensitive content, or explicit invalidation of any permission cache.

Fail-closed authorization

If the current permission state cannot be safely determined (e.g., during a metadata outage), the system should fail closed — reject the operation — rather than serve a stale cached 'allowed' answer that might now be wrong.

'Shared with me' indexing

Looking up 'files shared with me' by scanning every file's ACL would require a cross-shard fan-out if metadata is sharded by owner. A recipient-oriented secondary index (recipient_user_id → shared file IDs) avoids this by materializing the reverse relationship, at the cost of keeping it in sync with the primary ACL table.

'Shard by user_id, each user's files live on one shard' is a reasonable starting point, but not a universal answer.

Work that happens after a file becomes AVAILABLE should not block the upload path — but 'not blocking' is not the same as 'fire-and-forget.'

Background jobs after AVAILABLE may include: thumbnail/preview generation, search indexing, notification delivery to collaborators, and — where required — malware/virus scanning. Each has different correctness semantics.

If malware scanning is a required gate, use explicit states such as UPLOADED → SCANNING → AVAILABLE, with quarantine (not silent availability) on scan failure or scanner unavailability.

CDN delivery for private files needs its own explicit authorization design — an ordinary presigned object-storage URL does not automatically become CDN-backed.

Immutable, versioned object keys/URLs (a new key per version rather than overwriting one mutable URL) improve cacheability and sidestep most cache-invalidation problems: a new version simply has a new URL, and old cached responses for the old URL remain harmlessly stale-but-correct for the old version.

For private files, the CDN edge must enforce authorization before serving from cache — typically via a signed CDN URL or signed cookie, checked against a private origin, so that one user's cached private object cannot be served to another user who guesses or reuses a similar-looking URL. ACL is checked before the signed capability is ever issued, same as for direct object-storage downloads.

Revocation under CDN caching

Revoking access to a cached private response has the same limitation as presigned URLs generally — an already-issued, unexpired signed CDN capability can still be honored until it expires. Careful TTL policy and versioned keys are the primary mitigations.

Objects become orphaned for several distinct reasons, and cleaning them up incorrectly is one of the most dangerous classes of bug in this system.

Causes of orphaned objects: an abandoned multipart upload, an object upload that succeeded but whose metadata commit failed, an expired old version past its retention policy, a permanently deleted file past its grace period, or a failed background transformation that left a partial derived object behind.

The core safety rule

Never delete a blob solely because one transient metadata lookup says it is unreferenced. Use grace periods before hard deletion, durable deletion-state tracking (not a single point-in-time query), reference checks that account for deduplication if used, and idempotent deletion (a retried delete should not error or double-count).

Object durability and disaster-recovery policy are related but distinct concerns, and this section stays intentionally bounded for a foundations design.

A storage provider's durability objective describes the probability of losing a given object over time under normal operation — it does not by itself define your recovery point/time objectives for a regional outage, an operator error, or a bad deployment that corrupts metadata.

What a bounded DR design covers

Metadata backups and point-in-time recovery, independent of object storage's own durability. A replicated change log where cross-region durability of the sync stream is required. Explicit RPO (how much recent data can be lost) and RTO (how long recovery may take) targets appropriate to the product. Regional failure handling — DNS/routing failover, and whether the system degrades to read-only rather than accepting writes that a failed-over region might not have seen. Optional cross-region object replication, chosen based on requirements and cost, not assumed by default. Care to prevent split-brain metadata writes during failover — accepting writes in two regions simultaneously after a network partition is a correctness hazard, not just an availability one.