File Storage & Sync: Trade-offs & Decisions

Client → Application → Object Storage. The application server receives the entire file body and writes it to storage. Simple to implement, and the application can validate, transform, or inspect bytes in transit — this is a legitimate reason to choose proxying for specific requirements. Trade-off: it consumes application bandwidth and connection/resource capacity proportional to file size, couples data transfer to control-plane availability, and adds horizontal-scaling pressure as upload volume grows.

Client → Application (auth + presigned URL) → Client → Object Storage. The application only orchestrates: it authenticates, creates metadata, generates a presigned URL, and returns it. The client sends bytes directly to storage, which removes byte-proxying load from the application tier. Trade-off: presigned URL management adds complexity (expiration, scoping, security) — see Architecture Deep Dive → Presigned URL Security.

Store file bytes as BLOBs alongside metadata. Can be a reasonable choice for small files in some systems — one transaction, strong consistency, no second system to coordinate. For this design's workload (large files, up to tens of GB, at petabyte-per-day ingest), relational databases are not well suited: multi-GB binary objects degrade query performance, bloat backups, and prevent independently scaling byte storage from metadata storage.

Store bytes in a dedicated object store, purpose-built for large binaries with independent capacity scaling, a durability/cost model suited to bulk storage, direct/range access, and lifecycle policies. Trade-off: two systems to coordinate — metadata and bytes can become inconsistent if not handled carefully (see Architecture Deep Dive → Metadata/Object Consistency).

Fast retry on failure (little data to re-upload), but higher per-chunk coordination overhead — more presigned URLs, more HTTP round-trips, more part-state tracking. At very small sizes and very large files, coordination overhead can dominate.

Fewer chunks, less coordination overhead, but a failed chunk costs more to re-upload, and client memory/buffer pressure per chunk is higher.

Provider minimum/maximum part-size rules, maximum part-count ceiling, total file size, network conditions, client memory constraints, desired parallelism, and retry cost. Adaptive part sizing — adjusting chunk size based on observed connection quality — is a further optimization some clients implement.

Each device periodically calls GET /sync/changes. Simple, stateless, works behind any firewall. Trade-off: latency is bounded by the poll interval, and a large device fleet polling frequently generates substantial request volume — most of it returning no new changes.

The sync service maintains a persistent connection to online devices and pushes a change notification when something happens. This can reduce notification latency and wasted empty polling requests. Trade-off: requires infrastructure for persistent connections, reconnection logic, and fallback for devices that can't maintain them (restrictive firewalls, backgrounded mobile apps).

Push for online devices as a low-latency hint; polling as a fallback and safety net. Both paths converge on the same durable, cursor-based change log as the source of truth.

Simple; the most recent write becomes current with no user intervention needed. Trade-off: it silently discards the other write. Whether that's acceptable is a product policy decision, not a distributed-systems requirement — 'first/last arrival becomes main' is a choice the product makes, not something the protocol mandates.

Both versions are saved; the non-current one is exposed with a distinguishing name for the user to review and merge. Reduces the risk of silently overwriting one branch. Trade-off: requires user action, and ignored conflict copies can accumulate.

Using the common ancestor version to auto-merge non-overlapping changes — works for some text formats, not for binary files, and is more complex to implement correctly.

Transaction support well suited to permission changes and version-precondition checks, native JOINs useful for sharing queries, and mature tooling for schema evolution. Trade-off: horizontal write scaling requires deliberate sharding, which adds operational complexity.

Designed for horizontal write scaling and high throughput. Trade-off: typically no native JOINs, so sharing/relationship queries become application-level fan-out or require secondary indexes; consistency options vary by system and must be chosen deliberately for authorization-sensitive paths.

Maximum recoverability — any version can be restored at any time, with a complete audit trail. Trade-off: storage cost grows without bound for frequently-edited files.

Expire versions older than a window, or beyond a count, or once a per-owner storage quota is reached. Controllable storage cost. Trade-off: versions outside the retained window are not recoverable, unless a legal/compliance hold exception applies.

Simplest to implement and reason about. The right default for small files, or any format where binary deltas provide little benefit (compressed/encrypted/binary-dense formats).

Cuts transfer size for large files with in-place edits, but boundary shifts from insertions can defeat the savings.

Finds matching regions despite shifted offsets, providing the best transfer savings for large files with localized edits — at the cost of CPU and chunk-index metadata.

Simpler — the provider handles part storage and final object assembly. No additional application-level chunk metadata to maintain.

Can enable cross-version chunk reuse, deduplication, and finer-grained delta semantics, but greatly increases metadata, reference-counting, and garbage-collection complexity — effectively building a mini content-addressed storage system on top of object storage.

Intuitive for simple systems, but rename/move become identity changes — any reference by path breaks, and reconstructing 'is this the same file after a rename' from path history is fragile.

file_id is assigned once and never changes; filename and parent_folder_id are just metadata fields that can be updated cheaply. This is what allows sync clients to recognize a rename as a metadata change on the same file, not a delete-plus-create.

Good locality for 'my files' queries. Weakness: a single very large or very active user/tenant can become a disproportionately hot or oversized shard, and 'shared with me' queries cross ownership boundaries.

Natural fit for multi-tenant products; same large-tenant hotspot risk as user sharding.

Better load distribution across shards, at the cost of weaker locality for 'list my files' queries, which then need a secondary index.

Scales well — the application only issues a short-lived scoped URL after an authorization check; it does not carry byte-transfer load.

Gives the application fine-grained real-time control — it can enforce immediate revocation, audit every byte served, or transform content on the way out. Trade-off: reintroduces the bandwidth/scaling cost that direct upload was designed to avoid.

Lower cost and operational complexity; metadata consistency reasoning stays simple. Trade-off: a regional outage affects the whole system, with recovery bounded by backup/restore rather than live failover.

Improves regional resilience and can tighten RPO/RTO, at meaningfully higher cost and operational complexity — replicated change logs, cross-region metadata consistency handling, and care to prevent split-brain writes during a partition.