Instagram Architecture Deep Dive

Photos and videos dominate storage, processing, and network bandwidth on a social platform. The upload path exists to get large binary files off the client and into durable storage as efficiently as possible — without turning ordinary application servers into a bandwidth bottleneck for every megabyte a user ever uploads.

Why application servers should not proxy media bytes

A naive design routes every uploaded byte through the application tier: client → application server → object storage. This works for small files, but a large video means the application server holds an open connection and consumes bandwidth and memory for the entire transfer. At scale, this ties fixed, expensive compute capacity directly to raw upload throughput — exactly the wrong resource to scale for pure byte movement.

Direct-to-object-storage upload

A more scalable pattern separates authorization from transfer. The conceptual flow: Client → API: request upload → API → Client: temporary upload target → Client → Object Storage: media bytes → Client/API → Post Service: upload complete. The application server authenticates the request, authorizes it, and issues a temporary, signed upload target (a pre-authorized destination with a short expiry). The client then transfers bytes directly to object storage — the application tier never sees the payload.

Resumable / chunked upload

Large video uploads on unreliable mobile networks should not restart from zero after a dropped connection. A chunked upload splits the file into parts, each acknowledged independently; if the connection drops at 80%, the client resumes from the last acknowledged chunk instead of re-uploading the whole file. This is a client/object-storage concern — the application tier is involved only at authorization and completion, not per chunk.

Completion, integrity, and validation

Once all bytes have arrived, the client (or object storage, depending on the provider) notifies the Post Service that the upload is complete. At this point the system should validate file size and type against expected bounds, verify an integrity checksum against what the client declared, and reject or quarantine anything that fails validation before it is treated as a real post. This finalization step is also where post metadata is created or updated and asynchronous processing is triggered.

Once raw media lands in object storage, it is not yet ready to serve. A dedicated, asynchronous pipeline turns a raw upload into the set of variants a feed and CDN can actually deliver.

Conceptual pipeline: raw upload → processing event → workers → validation → image/video transforms → variants → metadata update → READY. The upload's completion triggers a processing event on a queue; a pool of workers consumes these events, independent of the request/response path the client is waiting on.

Image and video transforms

For photos, this typically means resizing to several standard sizes, compressing, and generating a thumbnail. For video, it means transcoding into multiple resolutions and bitrates so playback can adapt to device and network conditions, plus generating a poster frame/thumbnail for preview. The goal here is conceptual: understanding that multiple renditions exist and why, not the specifics of any particular codec or transform library.

State model

A post's media moves through an explicit lifecycle: UPLOADING → PROCESSING → READY, with a failure path of PROCESSING → FAILED. A post only becomes eligible for normal feed distribution — fanout, appearing in candidate generation — once its media reaches READY. This state is what the rest of the system checks before treating a post as servable.

Retries, dead-letter handling, and idempotency

Processing can fail transiently — a worker crash, a temporary resource limit — and should be retried with a bounded number of attempts. Media that keeps failing after those attempts is moved to a dead-letter or quarantine path rather than retried forever, and the post is marked FAILED so it never silently blocks a user's upload flow indefinitely. Because retries can reprocess the same event more than once, processing should be idempotent: reprocessing the same raw upload should produce the same stable output variants rather than duplicate or conflicting ones.

Once media reaches READY, it must be delivered to hundreds of millions of readers efficiently. Object storage is the durable origin; a CDN sits in front of it to keep the vast majority of requests from ever reaching that origin.

Immutable, versioned object keys

Processed variants (thumbnail, medium, large, per-bitrate video renditions) are written under stable, deterministic, effectively immutable keys. Because the underlying bytes never change once written, both the CDN and any downstream cache can treat them as safely, indefinitely cacheable — a property that only holds because processing produces a new key rather than overwriting an old one.

Rendition selection

A client requests the rendition appropriate to its device and network conditions — a small thumbnail for a feed scroll, a larger image or a specific video bitrate when actually viewing a post. This keeps typical feed-scrolling bandwidth low while still making full-quality media available on demand.

Viral media and cache hierarchy

For an ordinary post, CDN edge caching alone handles load: a miss fills from origin, subsequent requests hit the edge. A viral post creates a different problem — a surge of near-simultaneous requests can arrive at many edge locations that have never seen the asset, producing simultaneous cache misses. A layered cache hierarchy (edge, plus a shared regional or shielding layer in front of origin) reduces how often a miss actually reaches origin. Request coalescing — collapsing concurrent misses for the same object into a single origin fetch — prevents a single hot object from generating origin requests proportional to reader count.

Deletion is three separate events

Deleting a post's media is not one atomic action. Application-level unavailability (the post no longer resolves for users) can and should take effect immediately. CDN/cache invalidation — purging cached copies at the edge — happens next, and may lag briefly depending on the CDN's invalidation mechanism. Deleting the actual underlying object in storage is a separate, often asynchronous step governed by retention requirements, and does not need to block the first two.

The social graph is read on nearly every feed generation and every fanout operation. Its storage and access patterns must serve both directions of the relationship efficiently.

Two access directions

The graph is used two distinct ways, and a design should name both explicitly. Following lookup — user → accounts user follows — is used for feed composition, pull-source lookups at read time, and relationship/ranking signals. Follower lookup — author → users following author — is used for fanout-on-write. These are different access patterns on the same underlying relationship, and conflating them is a common source of confused designs.

Storage shape

At its core this is an adjacency-list access pattern: given an account ID, return the accounts it follows or the accounts following it. This can be represented as a sharded relational table indexed by the account ID for each direction, partitioned by account ID so no single shard bears an unbounded number of relationships. A graph database is one option for this workload but is not automatically required — a single-hop adjacency-list access pattern like this can be served well by several different storage architectures. Follower and following counts themselves are best treated as derived counters rather than something computed by scanning the full relationship set on every request.

High-degree nodes

An account with a very large follower count has a correspondingly large adjacency list on the follower side. Reading that list for fanout must support pagination and streaming rather than fetching the entire list into memory at once — this is the storage-layer root cause of the write-amplification problem covered in Fanout Queue & Workers.

Follow / unfollow effects

A follow or unfollow is a graph edge update. It changes the outcome of future fanout immediately, but it does not retroactively rewrite feed entries that were already written under the old relationship state. Those existing entries become stale references, handled by read-time relationship/visibility filtering (see Post Lifecycle, Privacy & Visibility) and cleaned up asynchronously rather than synchronously at the moment of the graph change.

The fanout infrastructure — queue plus workers — is the asynchronous write path that turns a new post into feed entries for its followers. Its capacity determines how quickly followers see new content, and its correctness under retries determines whether that content shows up once.

When a post becomes eligible for distribution (its media, if any, has reached READY), the Post Service emits a durable fanout event: Post Service → durable fanout event. Fanout workers consume this event, read the poster's follower list from the social graph, and write a feed entry per follower into the feed store.

Follower batching

Workers must not fetch tens of millions of follower IDs into memory at once. Instead, the follower list is paged or streamed, work is grouped into bounded batches, each batch is processed independently, and progress is checkpointed. If a worker crashes partway through, it resumes from the last checkpointed batch rather than restarting the entire fanout from zero.

Idempotency

Because queues typically provide at-least-once delivery, the same fanout event can be processed more than once. Idempotent feed writes use a logical uniqueness key equivalent to (feed_owner_id, post_id) — writing the same entry twice is a no-op rather than a duplicate. Note that queue partitioning alone does not prevent duplicates: partitioning helps with ordering and work assignment, but it is the idempotent write, not the partition key, that makes retries safe.

Partial fanout, backpressure, and fairness

Fanout is eventually consistent by design: a post may be visible to some followers before others while a large fanout job is still in progress. Queue lag — how old the oldest unprocessed event is — is the key health signal. If workers fall behind, the queue absorbs the backlog rather than dropping events, but worker autoscaling is bounded by downstream feed-store write capacity, not just queue depth — adding workers faster than the feed store can absorb writes just moves the bottleneck. Fairness matters too: fanout for one very large author should not be allowed to monopolize the entire worker fleet and starve fanout for ordinary users' posts.

Neither pure fanout-on-write nor pure fanout-on-read works across the entire follower distribution. Instagram-style platforms need this reasoned through independently, even though the same hybrid pattern appears in other social-feed designs.

For normal, low-fanout authors, fanout-on-write is used: the write cost is small and bounded, and readers get a fast, precomputed feed. For extremely high-fanout authors, pushing every post to every follower's feed store is avoided; instead, their recent posts are retrieved directly during feed composition — a pull step layered onto an otherwise push-based feed.

Beyond a single follower-count threshold

Follower count is a useful first approximation for classifying an author as normal or high-fanout, and it is the right place to start for interview and educational purposes: a configurable follower-count threshold. But fanout cost in a mature system can also depend on posting frequency (a moderate-follower account that posts constantly still generates real load), the ratio of followers who are actually active, current queue pressure, and observed feed-store write capacity. A more mature classification can incorporate all of these signals adaptively rather than relying on a single static number.

Feed composition under hybrid fanout

A feed read becomes: precomputed candidates + pull candidates → filter → merge/deduplicate → rank → response. Precomputed candidates come from the feed store (built by fanout workers for normal authors); pull candidates are fetched live for high-fanout authors the user follows. Both sets are merged, deduplicated by stable post ID, filtered for eligibility, and ranked together as one feed — the reader should not be able to tell which path a given post came from.

Strategy transitions

An author's classification can change over time, and the transition needs explicit handling. Push → pull (an author crosses into high-fanout status): previously pushed feed entries can simply remain in place, future posts stop being pushed, and read-time composition begins pulling the author's new posts — deduplication against any already-pushed entries handles the overlap. Pull → push (an author drops back below the threshold, or a new account transitions the other way): future posts begin fanout-on-write again; backfilling historical posts into every follower's feed store is optional and usually unnecessary.

Ranking never operates over every post in the system. Before anything is scored, a bounded candidate pool has to be assembled — this is a distinct step from ranking itself.

Candidate sources

Candidates come from precomputed feed-store entries (posts already fanned out to this reader), recent posts pulled live from hot/high-fanout accounts the reader follows, and — optionally, if the platform's chosen scope includes recommended content — a recommendation source surfacing posts from accounts the reader doesn't follow. This page keeps the Following/Home feed conceptually separate from a full recommendation-system design; a recommendation source is mentioned here only as one possible candidate input, not built out in depth.

Pipeline

candidate retrieval → merge → dedup → eligibility filtering → feature hydration → ranking. Candidate retrieval collects raw candidates from each source above. Merge combines them into one working set. Deduplication removes repeats by stable post ID — the same post can legitimately appear from more than one source (for example, already pushed to the feed store and also pulled live during a strategy transition), and a stable ID is what makes deduplication reliable. Eligibility filtering removes posts the reader should not see at all (deleted, blocked, private-and-not-followed). Feature hydration retrieves the additional data ranking will need. Only after all of this does ranking run — over a bounded set, not the entire corpus.

Once a bounded candidate set exists, ranking decides what order the reader actually sees posts in. This is a distinct concern from generating the candidates in the first place.

Conceptual stages: candidates → filtering → feature fetch → scoring → post-ranking rules → top K. Filtering removes any remaining ineligible candidates. Feature fetch retrieves the signals scoring needs. Scoring assigns each candidate a value. Post-ranking rules adjust the scored list for product concerns. The final top K is what gets returned as the page of feed results.

Feature categories

Signals used for scoring generally fall into three categories: post features (recency, media type, early engagement velocity), author-reader relationship features (how often the reader interacts with this author, closeness of the relationship), and reader features (the reader's general interests and historical engagement patterns). These are generic feed-design signal categories, not a claim about any specific platform's proprietary ranking implementation.

Scoring progression

A design can reasonably progress from a chronological baseline (simplest, always current) to a heuristic score (a weighted combination of signals, still simple to reason about) to a learned ranking model (trained on historical engagement to predict relevance). A common practical pattern is a cheap first-stage ranker over the full candidate set followed by a more expensive model applied only to a smaller shortlist — this bounds the cost of the expensive model to a fraction of the traffic.

Post-ranking rules

After scoring, a handful of rules are commonly applied before finalizing the page: deduplication (in case scoring somehow surfaced the same post twice), diversity constraints (avoid the same author appearing many times in a row), author repetition limits, freshness constraints (surface at least some recent content even if an older post scores higher), and other product-specific rules layered on top of the raw model output.

A feed is conceptually infinite and continuously changing. Standard offset pagination breaks down against that combination, and the fix is different depending on whether the feed is chronological or ranked.

Why offset pagination fails here

Offset/limit pagination assumes a stable, unchanging ordered list. Against a feed, new posts arrive and shift what would have been at a given offset, rankings can change between page requests, and the result is duplicated or skipped items across pages. Deep offsets are also inefficient for typical implementations, since satisfying a large offset still means processing everything before it.

Cursor model for chronological feeds

For a chronological feed, a cursor can conceptually encode (timestamp, post_id) or an equivalent stable ordering key. The next page is requested as 'everything after this cursor,' which remains correct regardless of what has been inserted elsewhere in the feed since the first page was fetched.

Ranked feeds are genuinely harder

Ranked order does not correspond to a single stable sort key the way chronological order does, so a simple cursor is not automatically sufficient. Three practical approaches, with real trade-offs rather than one universal answer: a materialized ranked window (compute and temporarily store an ordered page set, paginate through that fixed window); a snapshot/session cursor (fix the ranking as of the start of a session and paginate against that snapshot, accepting that it grows stale as the session continues); or a best-effort live feed with deduplication (re-rank on each request but track already-seen post IDs to avoid showing the same post twice as ranking shifts).

Interactions are high-frequency writes with a very different scaling shape than posts and feeds — the challenge is less about fanning content out and more about not turning a single row into a bottleneck.

Like idempotency vs. aggregate count

A like has a logical relationship — this user liked this post — modeled with a uniqueness key equivalent to (user_id, post_id), so a retried like request is a no-op rather than a duplicate. This is a completely separate concern from the aggregate count displayed on the post. A viral post receiving a huge volume of likes should not require every single interaction to synchronously update one relational counter row — that row becomes a hotspot. Instead, use partitioned/sharded counters (splitting the increment load across multiple physical counters), a fast counter layer for immediate feedback, and asynchronous aggregation that periodically reconciles the fast counters into a durable total.

Unlike

Unlike is the reversal of the same logical relationship, with an eventual, asynchronous update to the aggregate count — it does not need to synchronously decrement the same hot counter path any more than a like needs to synchronously increment it.

Comments

Comments are deliberately kept out of the main feed query path: they have their own storage and service, require pagination at scale just like posts do, and are fetched on demand when a reader expands a post. The feed itself typically carries only a preview or a count, not the full comment thread.

A feed entry can become invalid long after it was written — a post gets deleted, an account goes private or is suspended, two users block each other, a follow relationship ends, or the underlying media fails or is removed. None of these require rewriting every feed that already references the post.

The governing idea: the feed store is a candidate index, not the ultimate source of truth. It tells you what might be shown; it does not guarantee what should still be shown by the time a reader actually requests their feed.

Serving path

1. Retrieve candidate IDs from the feed store and pull sources. 2. Validate and hydrate — fetch the current post/media state for each candidate. 3. Apply visibility and relationship rules — is the author's account still public relative to this reader, are they blocked, is the follow relationship still active. 4. Drop any candidate that fails validation or visibility. 5. Asynchronously clean up stale feed references so future reads don't repeatedly re-check the same dead entries.

Across every layer above, the same distinction keeps reappearing: durable source of truth, derived/rebuildable state, and cache. Naming which category each piece of data falls into is more valuable than debating which specific database technology should hold it.