File Storage & Sync: Failure & Scale Scenarios

A client is uploading a 5 GB file in chunks. After chunk 47 of 50, the network drops.

Failure

The connection for chunk 48 is interrupted. Chunks 1–47 are already stored in object storage with confirmed acknowledgment; chunks 49–50 were never sent.

User impact

The client shows the upload as paused. When the network returns, it must reconcile which parts actually landed before resuming.

Detection & Recovery

The client (or service, where the storage API supports listing uploaded parts) reconciles application-recorded state against what storage actually has — the application's own chunk-received counter is not necessarily authoritative if clients upload directly to storage. It resumes by uploading only the missing/failed parts, then calls the idempotent completion endpoint.

Prevention

Durable session/part state and a client that always reconciles-before-resume rather than trusting its last known local state blindly.

A part arrives at object storage but the data is corrupted in transit (bit flip, truncation) or fails validation.

Failure

A part fails an available integrity check at upload time, or the assembled object fails whole-object verification at completion.

Detection

Where the storage API supports per-part checksums, a mismatch is caught immediately and only that part needs retry. Where only a final whole-object hash is available, a mismatch tells you the object is wrong — not which part caused it, unless per-part state also exists.

Safe behavior

A file must not be marked AVAILABLE after a failed integrity check. Reject completion and surface which parts (if known) need to be retried.

Recovery

Client retries the affected part(s); if only a whole-object mismatch is available with no per-part signal, the safer fallback is re-uploading the affected parts based on other available session state, or in the worst case the whole object.

The client's completion request times out from the client's perspective (though it may have succeeded server-side). The client retries. The service receives the same completion call twice.

Risk

Without idempotency, the second completion could attempt to assemble the object twice, or create a duplicate metadata record.

Safe behavior

The completion endpoint is idempotent, keyed by the upload session ID: if the session is already DONE, it returns the same success response without side effects. A deterministic file_id from the initial create call is not by itself sufficient to make duplicate creates safe — the create endpoint needs its own idempotency key or explicit reuse semantics, since two independent create calls could otherwise be treated as two different files even with similar inputs.

A client pauses an upload for an extended period (device sleeps). When it resumes, something about the previously-issued access no longer works.

Differentiate the actual cause

These are distinct failure modes that need distinct handling: the presigned URL itself expired (short TTL passed); the upload session expired (longer-lived session state was cleaned up); the underlying credentials/policy changed (e.g., a key rotation or permission change invalidated the signing context); or the upload was already completed or aborted by another path (e.g., a background expiration job).

Recovery

The client detects an authorization/expiry error, checks session status, and — if the session is still valid — requests fresh presigned URLs for the remaining unconfirmed parts only. Already-confirmed parts keep their recorded state. If the session itself has expired, the client must re-initiate.

Object storage returns errors for some or all requests during an outage.

Impact

Uploads and downloads fail while the outage persists. Metadata operations may be unaffected, letting users still browse file listings and manage shares even though byte transfer is degraded.

Safe behavior

Retries must be bounded, with backoff and jitter — not unbounded retry storms. The UI should show a degraded state rather than a silent hang.

Recovery

In-progress multipart uploads do not resume automatically purely because storage recovers — the client or service still needs to detect recovery and re-drive the session/part reconciliation described in Upload Interrupted.

The primary metadata database becomes unavailable.

Impact

New uploads cannot be initiated, authorization checks that require current state cannot be answered reliably, and sync cannot commit new changes. File bytes already in object storage remain safe, and already-issued presigned URLs continue to work until they expire.

Safe behavior

A stale read replica may not reflect a very recent permission revocation — using it to authorize sensitive operations during an outage is a risk, not a safe fallback; authorization-sensitive paths should fail closed rather than trust a possibly-stale replica. An in-memory application write queue is not a safe durable recovery mechanism — it can silently lose acknowledged mutations if the application process dies before flushing. Automatic failover time is deployment-specific and should not be described with a fixed number without qualification.

Recovery

Prefer failing over to a healthy primary/standby where the architecture supports it, or safely rejecting/retrying mutations (with clear client-facing errors) rather than silently accepting writes into an unreliable buffer.

Two devices edit the same file while one is offline, or two writers race even while both are online.

Detection

The server applies a compare-and-set check: the incoming mutation's base_version must equal the current version for it to be accepted as the next version. If the current version has already advanced, the incoming write is divergent — a conflict, detected via version lineage rather than timestamps.

Delete/update race

A device offline before a file was deleted elsewhere later submits an edit based on a version that no longer has a live parent. Product policy decides the outcome (restore as conflict copy, reject as stale, or create a recovered file) but the underlying stable ID and version-lineage check is what makes any of those choices implementable safely.

At large scale, each component hits different, coupled scaling limits.

Metadata DB

Read replicas absorb read-heavy traffic; at extreme scale, sharding (by owner, tenant, or file_id — see Trade-offs & Decisions → Owner Sharding vs File-ID/Tenant Sharding) becomes necessary, and 'shared with me' queries need a dedicated index to avoid cross-shard fan-out.

Upload/sync services

Generally stateless and horizontally scalable, but sync connection count, change-log write throughput, and notification fan-out can still become bottlenecks independent of raw CPU.

Object storage

Managed object storage does not mean the architecture has no capacity, request-rate, bandwidth/egress, or key-distribution considerations under all workloads — hot key prefixes, request-rate limits, and cost at high egress volume are real constraints to design around, not assumptions to ignore.

A central consistency failure: object bytes are fully and correctly stored, but the metadata transaction that should publish the file as AVAILABLE fails or times out.

Result

Either an orphaned object with no corresponding AVAILABLE metadata record, or a metadata record stuck in an intermediate lifecycle state with an underlying object that is actually fine.

Recovery

Completion must be idempotent so a retried completion call is safe. A reconciliation job periodically verifies storage-object/session state against pending metadata records: if the object is confirmed intact and the session is still within a valid window, it safely commits the metadata as AVAILABLE; if the session has expired with no valid metadata commit, the object is garbage-collected after a grace period.

Metadata claims a file version is AVAILABLE, but the corresponding object cannot be found in storage.

Potential causes

A bad lifecycle transition (marked AVAILABLE before verification), an accidental deletion, a replication/recovery issue on the storage side, or an incorrect storage-key pointer recorded in metadata.

Safe behavior

Never silently return an empty or corrupted response as if it were the real file. Raise an alert, and where the architecture supports it, attempt an alternate replica, prior version, or backup rather than failing the user request with no explanation.

Recovery

Mark the affected version for reconciliation, investigate the root cause (which lifecycle transition path let this happen), and prevent recurrence by tightening the verify-before-publish step in Metadata/Object Consistency.

A device has been offline longer than the server retains change-log history, so its stored cursor can no longer be replayed incrementally.

Impact

The device cannot simply 'catch up' by asking for changes since its old cursor — that history no longer exists.

Recovery

Fetch a full namespace metadata snapshot, compare it against local on-device state, reconcile any local operations that were queued while offline (which may themselves conflict with what changed remotely), and establish a fresh cursor to resume incremental sync going forward.

Push notifications and change-log entries may be delivered more than once, delayed, or observed out of the order they were produced.

Risk

A naive client that applies whatever it receives, in receipt order, without version checks can apply an older change after a newer one, or double-apply the same change.

Safe behavior

Version IDs and operation IDs make replay safe — reprocessing the same operation ID is a no-op, and applying an update whose target version no longer matches current state is detected and rejected rather than blindly applied. Cursor semantics (persist only after successful apply) prevent skipping changes on crash-restart.

Concrete races that fall out of multi-device, sometimes-offline synchronization.

Examples

A laptop renames a file while a phone (offline) is editing it. One device deletes a file while another edits it offline. A folder is moved while a child file's upload is still completing.

Safe behavior

Stable file_id keeps identity intact across a rename regardless of what else is happening. Version checks and tombstones (rather than immediate hard deletes) let a device that later reconnects discover what actually happened instead of guessing.

Recovery

Explicit product policy resolves each race (e.g., 'edit against a deleted file becomes a conflict-recovered file'), but the mechanics — stable IDs, version preconditions, tombstone visibility — are what make any chosen policy implementable without silent data loss.

The push/notification layer that wakes up online devices becomes unavailable.

Impact

Devices stop receiving low-latency 'something changed' hints.

Safe behavior

Push is explicitly an optimization on top of the durable change log, not the source of truth — clients must not permanently miss changes just because push failed.

Recovery

Fallback to cursor-based polling on reconnect/periodic schedule ensures devices still catch up, just with higher latency than push would have provided, until the notification service recovers.

One tenant, user, or folder has an unusually large number of children or shared recipients — millions, not thousands.

Risk

Naive implementations that synchronously enumerate or recurse over a full folder subtree, or fan out a share operation to every recipient inline, become slow or fail outright at this scale.

Safe behavior

Paginated listings rather than returning entire folder contents in one response. Metadata partitioning that avoids putting one enormous tenant's entire workload on a single shard. Asynchronous fan-out/indexing for operations like 'notify everyone this folder was shared with' rather than doing it synchronously in the request path. Per-tenant limits or fairness controls so one hot user cannot starve others sharing the same infrastructure.

An owner revokes a user's access to a file, but that user already obtained a presigned download URL that has not yet expired.

The capability window

Revoking the metadata ACL does not retroactively invalidate a URL that is already issued and still within its validity window — this is an inherent property of bearer-capability URLs, not a bug to be patched away casually.

Mitigations

Short URL TTLs bound the exposure window. Where immediate revocation is a hard product requirement, use edge/token validation that checks current permission state on each access rather than trusting a static signature alone, or rotate/version the underlying access strategy so revocation can invalidate a whole class of prior URLs at once.

A background job — thumbnailing, indexing, or malware scanning — fails or the worker pool handling it becomes unavailable.

If the job is a security gate

When malware scanning is required before a file can be downloaded or shared, a scanner failure must not silently leave the file in an available/downloadable state — it should remain in a scanning/quarantined state until scanning actually succeeds.

If the job is optional

Thumbnail generation or search indexing failing should degrade gracefully — the file remains usable, just missing a preview or lagging in search results — rather than blocking the user.

A large number of multipart upload sessions get abandoned — client crashes, app closes mid-upload, or users simply give up — without ever completing or aborting.

Impact

Each abandoned session can leave uploaded-but-unassembled parts consuming storage and, in some architectures, contributing to account/billing overhead, even though no usable file ever results.

Safe behavior

Lifecycle expiration policy on sessions (e.g., auto-abort after a bounded period of inactivity), explicit abort calls to release provider-side part storage, and background GC that sweeps expired sessions.

Prevention

Track metrics on abandoned-session age, count, and bytes consumed so this failure mode is visible operationally rather than discovered only when storage cost spikes.

A bug in reference counting or cleanup logic causes garbage collection to delete bytes that were actually still needed.

Risk

This is one of the most severe possible failures in the system — unlike most other failures here, it can cause genuine, permanent data loss for a user who did nothing wrong.

Safe behavior

Soft delete with a grace period before any hard deletion. Idempotent GC operations, so a retried GC pass cannot double-delete or miscount references. Regular audit/reconciliation comparing what GC believes is unreferenced against independent evidence.

Recovery

Backups and version retention (see Architecture Deep Dive → Version Storage and Retention) act as the last line of defense if a GC bug slips past the grace period — this is precisely why 'never hard-delete immediately' is a safety rule, not just a nicety.

An entire region hosting metadata and/or object storage becomes unavailable.

Impact

Depending on architecture, this can range from a temporary full outage (single-region design) to a partial degradation (if cross-region replication and failover are in place).

Safe behavior

Metadata failover (if configured) must avoid split-brain writes — two regions both accepting writes after a network partition is a correctness hazard, not just an availability one. DNS/routing failover should be tested, not assumed to work correctly on the actual day. The system may need to degrade to a read-only mode rather than accept writes it cannot safely reconcile later.

Recovery

Recovery time and acceptable data loss are governed by the RPO/RTO targets set in Architecture Deep Dive → Disaster Recovery and Multi-Region — a regional failure is the scenario that plan exists for.

Triage map