Designing a Video-Sharing & Streaming Platform

We are designing a platform where creators upload videos, the platform processes them into streamable form, and viewers watch them smoothly on any device and any network, anywhere in the world.

The core loop looks simple: a creator uploads a video file, the platform makes it available, and viewers press play. Underneath that simplicity is one of the most demanding workloads in software engineering — ingesting large binary files reliably, transforming them into multiple streamable variants, and then serving that content at massive, wildly uneven, globally distributed scale.

Two workloads, not one

The single most important observation about a video platform is that it contains two fundamentally different workloads. The first is upload and processing: comparatively low in volume, but each unit of work is large and computationally expensive. The second is delivery: an extremely high-volume, read-heavy, globally distributed streaming workload. Designing both with the same infrastructure is the most common mistake in this problem space.

Scope boundaries

In scope: video-on-demand (VOD) upload and playback — chunked/resumable upload, asynchronous transcoding into multiple renditions, thumbnail generation, metadata storage, segmented/adaptive streaming, CDN delivery, search, view counting, playback authorization for non-public content, and editing/deleting videos. Likes/comments can remain part of the picture but shouldn't consume disproportionate design depth. Out of scope: live streaming, DRM internals, recommendation algorithms, monetization/ads, and content-moderation internals — each is a substantial design problem in its own right.

Functional requirements describe what the system does; non-functional requirements describe how it must behave under scale and failure — and for a video platform, the second list is arguably more important.

Functional

Upload video with the ability to pause, resume, and retry an interrupted upload. Process/transcode uploaded video, and publish it only once playable outputs are actually ready. Generate thumbnails and extract metadata. Support multiple renditions. Stream video with adaptive quality, including seeking to any point. Enforce playback authorization for non-public (unlisted/private) content. Search videos. View and edit metadata/privacy. Delete a video. Track playback/view telemetry. Likes/comments may remain in scope, but they should not consume disproportionate design depth relative to the upload/processing/playback problems above.

Non-functional

It's tempting to state a fixed universal ordering like 'playback availability > processing reliability > cost efficiency,' but that's a product-specific choice, not a law. What is broadly true is that playback tends to be latency-sensitive (a slow or broken play button is immediately visible to a viewer) while processing tends to be delay-tolerant (a creator waiting an extra few minutes for 'processing…' to clear is a much smaller cost) — but the actual priority ordering, and how much each is worth trading against cost, depends on the product's own SLOs.

Scale estimation here is less about precision and more about establishing that uploads and views tend to live on very different orders of magnitude — and about having the right formulas ready, not memorized numbers.

For a YouTube-like platform, it's a reasonable illustrative interview assumption — not a documented fact about all video platforms — that views vastly outnumber uploads, often by two to three orders of magnitude, and that a single upload can be several gigabytes. Any specific figures like '5 million uploads/day,' '5 billion views/day,' a '1,000:1' ratio, or 'petabytes per day' of storage growth should be treated as illustrative assumptions you'd state and justify in an interview, not as properties every video platform necessarily has.

Capacity estimate (illustrative)

These are illustrative planning numbers, not measurements of any real system; the 5 million uploads/day, 5 billion views/day and 1,000:1 ratio are the example figures named above, chosen only to show the formulas in use. Assumptions: average original 500 MB; average source length 6 minutes; a ladder of 6 renditions, with derived renditions, thumbnails and manifests totalling 1.5× the original bytes (so total stored is 2.5× the original: 1× original + 1.5× derived); one transcoding core encodes at 2× real time per rendition (0.5 core-minute per rendition-minute); average watch time 300 s per view at an average delivered bitrate of 3 Mbps; peak-to-average factor 2. Day = 86,400 s.

What this implies for the design: playback bytes exceed upload bytes by more than two orders of magnitude, so the CDN and origin protection, not the upload path, set the delivery architecture; even a 99% edge byte-hit ratio would leave about 0.5 Tbps (≈ 5.6 PB/day) of origin egress, which is why a shield tier and cache-key discipline matter. The upload side is the opposite: about 58 requests/s is modest, but 62,500 busy cores and 2.3 EB/year of storage make processing cost, backpressure and storage lifecycle the real concerns. The Failure & Scale Scenarios page reuses these numbers for its upload-surge and viral-video traces.

Start with the simplest thing that could work: one client, one app server, one database, one file store. No CDN, no queue, no transcoding pipeline.

At small scale — a handful of creators, modest traffic — this works fine. The application server accepts uploads, writes metadata to the database, stores the video file, and serves it back on request. There is no architectural reason to reach for object storage, a CDN, or asynchronous processing yet.

Something saturates first — but not by a universal rule

As creators and viewers grow, some part of this simple system runs out of capacity first. It's a common illustrative case that the application tier saturates first — CPU and connections handling uploads, metadata lookups, and playback requests, while the database still has headroom — but this isn't guaranteed. Depending on the actual workload, network egress, storage throughput, database connections/locking, CPU, or connection limits could each become the first bottleneck instead. The honest answer is: measure it, don't assume it.

A standard response to app-tier saturation specifically is horizontal scaling: put a load balancer in front of several stateless application server instances, so any instance can handle any request. This removes the single point of failure and generally improves capacity as instances are added — at the cost of requiring the app tier to hold no per-request state locally.

Large file uploads over unreliable networks are the first real problem, and they demand several separate decisions: how to upload reliably, where the bytes should go, how to know an upload is genuinely complete, and how to keep the credential that authorizes it safe.

Resumable, chunked upload and a session lifecycle

A single-request upload of a multi-gigabyte file has no recovery story — if the connection drops partway through, the only option is to start over from zero. The fix is to split the file into parts, upload them independently with per-part acknowledgment, and retry only the part that failed. This requires an upload session with its own lifecycle — a representative shape is INITIATED → UPLOADING → FINALIZING → FINALIZED → PROCESSING — tracked under a stable upload_id owned by the uploading creator.

Direct-to-storage upload

Routing every part through the application server consumes app-tier bandwidth and connection capacity for pure data transfer — work that has nothing to do with business logic. The better pattern: the app server issues a short-lived, scoped upload credential, and the client uploads parts directly to object storage. The app server is only involved in orchestration and metadata, never in moving the actual bytes.

Video binaries — originals, transcoded renditions, segments, and thumbnails — belong in object storage, never in the primary application database.

A relational database is optimized for structured rows, indexes, and transactions — not multi-gigabyte binary blobs. Object storage is purpose-built for exactly this: durable, highly scalable, and dramatically cheaper per gigabyte than block storage attached to a database server.

An uploaded video cannot usually be streamed directly — it needs transcoding, thumbnail generation, and metadata extraction, and none of that should block the upload response.

Trying to process video synchronously — inside the same request that accepted the upload — means the request stays open for as long as transcoding takes, which can be tens of seconds to minutes. Worse, transcoding is CPU-intensive, so a handful of concurrent uploads can saturate the exact servers responsible for accepting new uploads and serving metadata.

The fix: once the original file is durably finalized, acknowledge the upload and durably schedule a processing job. But 'finalize the metadata write, then enqueue a job' is two different systems (a database and a queue/broker) that cannot be assumed to succeed or fail together — a naive 'single logical transaction' framing glosses over a real dual-write gap. The safer baseline is a transactional outbox or durable job record: write the finalized metadata and a job row in the same database transaction, then have a separate dispatcher publish that job to the processing queue, tolerating the possibility that a publish gets duplicated. See Architecture Deep Dive -> Durable Processing Handoff for the exact mechanics.

One uploaded original becomes several output renditions, because no single format serves every device and network condition well — and 'rendition' means more than just a resolution number.

A representation/rendition can differ by resolution, bitrate, codec, frame rate, audio codec/bitrate, and HDR/SDR profile — resolution alone is an incomplete description. See Architecture Deep Dive -> Codec, Container, Bitrate & Resolution for a concise breakdown of these distinct properties.

A representative rendition ladder might include something like 1080p, 720p, 480p, and 360p — but this is an example, not a fixed universal ladder. The right ladder depends on the source's own quality (there's no benefit to a rendition above what the source supports, and the platform should not upscale merely to fill a ladder rung), the device/network population being served, codec support across that population, content characteristics, and cost.

Producing more renditions generally increases compute and storage cost, but not by a fixed linear multiplier — codec choice, resolution, source duration, hardware acceleration, shared decode work, and available parallelism all affect the real cost of each additional rung. Because transcoding workers are stateless and pull from a shared queue, the worker fleet can scale with demand. Failed jobs are redelivered and retried — but a redone transcode must never simply overwrite a viewer-visible output in place; see Safe Output Publication in the Processing DAG discussion below.

Processing as a DAG, not one job

Rather than one opaque 'transcode the video' job, model processing as a staged pipeline: probe → transcode video representations → transcode/package audio → segment/package → thumbnail → validate → publish. Several stages — most notably transcoding each representation — can run in parallel. Track state at least per stage, and ideally per representation, so a failure in one rendition doesn't require guessing about the others or redoing work that already succeeded. See Architecture Deep Dive -> Processing DAG and Media Probe.

Serving one large file per rendition is clumsy. Slicing every rendition into short, time-aligned segments — and packaging them with a manifest — enables smooth buffering, fast seeking, and dynamic quality switching.

Each rendition is cut into segments, aligned in time (and at compatible keyframe boundaries) across renditions, and referenced by a manifest — commonly packaged using a protocol like HLS or MPEG-DASH. A master manifest lists the available representations; each representation has its own media playlist listing its segment URLs in order. A player fetches the segments it needs, buffers ahead, and can jump to any timestamp by requesting the corresponding segment — all without downloading the whole file. Segment duration is typically a few seconds, chosen as a trade-off between adaptation speed and request overhead — treat any specific duration as an illustrative example, not a fixed rule; low-latency streaming uses different chunking strategies and is out of scope here.

Because every rendition shares the same segment/keyframe boundaries, the player can switch quality between segments without a visible restart: request the next segment at a lower bitrate when bandwidth drops, and climb back up when it recovers. This is adaptive bitrate streaming (ABR) — the client, not a fixed server-side assumption, decides the sustainable quality based on measured throughput, buffer health, and device constraints.

Serving every viewer directly from origin storage does not scale — bandwidth and request load concentrate on one place while viewers are scattered across the world.

A CDN caches segments at edge locations physically close to viewers. Most streaming requests are then served from a nearby edge rather than traveling back to origin, which reduces origin load, cuts latency, and lets the system absorb far more concurrent viewers than origin bandwidth alone could support.

Cache hits and misses

The first viewer to request a given segment at a given edge causes a cache miss. A well-designed hierarchy doesn't send that miss straight to origin object storage — it first checks an intermediate regional/shield cache tier, which may already have the object from serving a different edge's earlier miss. Only a genuine miss at that tier reaches origin. See Architecture Deep Dive -> CDN Cache Hierarchy and Origin Protection.

Hot videos, and why unrestricted origin fallback is not automatically safe

Popularity on a video platform is extremely uneven — a single video can dominate a large share of total traffic. Because that content gets cached across many edges quickly, the CDN absorbs most of the surge — but the moment right after publication, before anything is cached anywhere, is a genuine cache-stampede risk: many edges missing near-simultaneously. Request coalescing at each edge, plus an origin shield to collapse cross-edge fan-in, plus origin-side rate limiting, are what keep this from becoming an origin overload. It is not a safe assumption that 'the worst case is falling back directly to origin' — for a large platform, unrestricted fallback under a real spike can overload a finite origin. A sound failure policy instead includes options like an alternate edge/PoP, a shield layer, safe stale-serving, bounded (rate-limited) origin fallback, and load shedding. See Architecture Deep Dive -> Origin Protection and Failure & Scale -> CDN Cache-Miss Storm.

Video metadata is deliberately decoupled from video binaries, and search is deliberately decoupled from the metadata database — and a video's metadata existing is not the same thing as a video being playable or discoverable.

Title, description, creator, thumbnail reference, duration, available renditions, visibility, and creation time live in a database optimized for structured application queries. But metadata needs real lifecycle states, not just fields: has the original been durably uploaded, is it currently processing, does a validated playable output set exist, does policy currently permit discovery/playback, and is it currently blocked/private/deleted. A video should not become playable merely because a metadata row exists for it — playability depends on the processing pipeline reaching a validated, published state (see Architecture Deep Dive -> Metadata Lifecycle and Safe Output Publication).

As the catalog grows and search needs ranking and full-text matching, querying the primary database directly becomes both slower and disruptive to transactional metadata traffic. The fix is a dedicated search index, kept in sync via asynchronous indexing whenever metadata changes — this module treats the indexing/ranking mechanics themselves as belonging to the Search & Autocomplete module rather than re-teaching them here. The index should reflect discoverability, not merely existence: a video that's unpublished, blocked, or deleted should not still be indexed. And a privacy-restricting change or a deletion deserves faster propagation to the index than an ordinary title/tag edit — the cost of a stale index entry is different in kind, not just degree.

A synchronous increment on a single database row for every view does not survive a popular video's view velocity — but before optimizing the mechanism, define what actually counts as a view.

There's no single universal technical definition of a 'view' — it's a product policy decision. An illustrative policy might be 'count a view once playback crosses the platform-defined watch threshold,' but the specific threshold (a fixed duration? a percentage watched? something else?) is configurable product policy, not a hard-coded engineering constant.

For a hot video, many concurrent viewers can generate many write attempts per second against the exact same row — severe lock contention that has nothing to do with the actual usefulness of the number being incremented. The fix is asynchronous aggregation: publish a view event per view, batch many of them together over a short window in aggregation workers, de-duplicate by a stable event identity to guard against retries or duplicate deliveries, and periodically flush one batched update per video instead of many individual ones.

Beyond video delivery, metadata reads, search results, and thumbnails all benefit from caching — and the metadata store itself may eventually need to scale further, once a specific workload actually demands it.

Frequently-read, rarely-changing data — a popular video's title, thumbnail, and stats — is a good caching candidate. A cache-aside pattern in front of the metadata database absorbs much of the repeated read load, with TTL-based expiration and explicit invalidation on writes as the two defenses against staleness.

As the catalog and interaction volume grow, there is no single universal scaling progression that applies to every workload — vertical scaling, read replicas, and partitioning are all real options, but which one (if any) is needed depends on which specific bottleneck (write throughput, data volume, or query latency) has actually been measured. Video metadata partitions naturally by video ID for point lookups, though other access patterns (like listing a creator's videos) may need a different index or partitioning strategy.

A CDN solves delivery latency for cached content. It does not, by itself, solve origin availability, upload locality, or disaster recovery — and multi-region is not simply 'each region gets its own copy of everything.'

With a single origin region, a creator on the other side of the world pays real upload latency and cross-region transfer cost, and a regional outage can become a global outage for anything not already cached. Regional origin infrastructure — its own upload/processing pipeline, its own metadata replica, its own storage — lets distant creators upload and process locally, and lets the platform survive the failure of a single region, but only if ownership and replication are actually designed for it.

Every asynchronous, distributed piece of this architecture must assume failure, and every expensive piece of it must justify its cost. This chapter is a summary — the full scenario-by-scenario treatment lives in Failure & Scale Scenarios.

A crashed transcoding worker must not lose its job — the queue redelivers unacknowledged work, and safe, versioned publication means a duplicate or overlapping retry can never corrupt viewer-visible output. Object storage failures are mitigated by redundancy, though not every kind of partial or regional failure is automatically made transparent — see Failure & Scale -> Object Storage Failure for the distinctions that matter. A metadata database failure is absorbed by cached reads for already-popular content and resolved via replica promotion, which has its own lag/RPO considerations. A CDN edge failure should reroute to an alternate edge or shield tier rather than unconditionally falling back to origin. Search failing entirely should never affect playback of videos a viewer already knows how to reach.

Cost is a first-class constraint, and it has more dimensions than storage alone: storage, transcoding compute (CPU/GPU/accelerator), CDN egress, origin egress, cross-region transfer, cache miss rate, reprocessing, and long-tail rendition storage. Not every video needs the same treatment — cold, rarely-watched content can move to cheaper storage tiers, and lazily generating a rarely-requested rendition on first request is a legitimate cost lever, though it is a genuine trade-off (added first-view latency, added implementation complexity) rather than a universally recommended default. The best architecture is the one that satisfies the requirements at an appropriate, deliberately chosen level of complexity and cost — not the one with the most components.