URL Shortener — Architecture Deep Dive

The short code is the primary key of this system. Before choosing how to generate a code, decide how big the namespace needs to be — code length is a sizing decision, not a fixed constant.

Namespace math

Base62 uses the alphabet [0-9a-zA-Z] (62 symbols). A code of length k gives a keyspace of N = 62^k: 6 chars ≈ 56.8 billion, 7 chars ≈ 3.5 trillion, 8 chars ≈ 218 trillion. Longer codes trade a few extra characters of URL length for a much larger namespace and lower collision/retry pressure at high occupancy.

What actually determines code length

There is no universally correct starting length. Choose it from the combination of: expected number of active/retained mappings, desired occupancy (how full the namespace is allowed to get before retries rise), acceptable retry rate for random generation, space reserved for custom aliases, expected future growth, and whether enumeration/security concerns push toward a larger random keyspace. A design expecting 100M–1B retained mappings commonly lands on 7–8 characters, but that is a consequence of the assumptions, not a rule.

Custom aliases share the namespace

User-chosen aliases and generated codes occupy the same short_code column and the same uniqueness constraint. If aliases can be much longer than generated codes (e.g. up to 50 characters), the schema and length assumptions must accommodate both — see Complete Design → Data Model for the schema implication.

Random code generation needs a precise, correct model of collision probability — the existing intuition of 'collisions are basically impossible' or 'collisions are common' are both wrong without doing the math for the specific keyspace and occupancy.

Two different probabilities

For a keyspace of size N = 62^k with n codes already occupied, two distinct questions are often conflated:

For N = 62^7 ≈ 3.5×10^12 and n = 100,000,000 occupied codes, the per-attempt collision probability is roughly 100,000,000 / 3.5×10^12 ≈ 0.0029% — low enough that a bounded retry loop rarely needs more than one retry. The birthday-bound figure at the same n is higher because it accumulates probability across every prior insertion, so it is not 'the chance my next code collides.'

UNIQUE constraint as the correctness guard

Regardless of how low the computed probability is, correctness must never depend on probability alone. A UNIQUE constraint on short_code is the final arbiter: INSERT ... ON CONFLICT (short_code) DO NOTHING (or the equivalent) returns zero rows affected on a genuine collision, and the application generates a new random code and retries. This makes the system correct even if the probability estimate is wrong or occupancy grows faster than expected.

Bounded retry, not unbounded retry

The retry loop must be bounded (for example, a small fixed number of attempts) rather than looping indefinitely. If retries repeatedly fail — which would only happen at very high namespace occupancy — that is a signal to increase code length or apply a different generation scheme, not to keep retrying forever. See Failure & Scale Scenarios → Short-Code Collision / Namespace Pressure for what this looks like operationally as occupancy climbs.

A Snowflake-style generator produces decentralized, unique, roughly time-ordered integer IDs without a database round-trip. It is a genuinely useful option — but it is one option among several, not the mandatory answer once a system has multiple write servers.

Mechanics

A typical layout packs a 64-bit integer as: a timestamp component (millisecond precision, commonly 41 bits ≈ decades of range), a worker/node-ID component (uniquely assigned per generator instance), and a per-millisecond sequence counter. Encoding the result in Base62 produces a compact, URL-safe short code.

What it actually solves and what it costs

Base62-encoding a Snowflake ID does not make it unpredictable

This is an important nuance: encoding a Snowflake integer in Base62 changes its representation, not its information content. The encoded value still contains structured, time-ordered information — an attacker or curious user can still infer creation order and approximate creation time. If unpredictability/non-enumerability is an actual requirement, add a reversible permutation/obfuscation step on top of the Snowflake ID, or choose random identifiers instead. See Trade-offs → Random Codes vs Enumerable Codes for how to reason about this separately from uniqueness.

When it becomes attractive

Snowflake-style generation is worth its operational cost when the system genuinely needs decentralized unique-ID generation at high throughput without a database dependency for allocation — for example, many independent write-path nodes, multi-region generation, or extremely high creation throughput. It is not required merely because a design runs more than one write server; a central database sequence or bounded random-code retries can comfortably handle substantial concurrent write throughput (see Trade-offs → ID Generation Strategy).

The redirect status code is a product and caching decision with real infrastructure consequences — not a coin flip between '301 always caches' and '302 never caches.'

What each status communicates

301 Moved Permanently signals that the mapping is expected to be durable; 302 Found and 307 Temporary Redirect signal that the destination may change and the client should re-check with the server. 307 additionally preserves the original HTTP method, which matters if the original request was not a GET.

Caching is status plus headers plus client behavior

Whether a browser or intermediary actually caches a redirect depends on the status code combined with explicit HTTP caching headers (Cache-Control, Expires) and on client/proxy implementation behavior — it is not guaranteed purely by the status code. A 301 without caching headers may still be revalidated by some clients; a 302 served with explicit long-lived Cache-Control can be cached by clients that respect it. Avoid the oversimplification 'a 301 always caches in the browser, a 302 always hits the server' — treat the status as a signal of intended cacheability, and pair it with the caching headers that actually control behavior.

Why this matters for analytics

If a client or intermediary caches the redirect response, subsequent visits from that client may never reach the URL service again — meaning those clicks are invisible to server-side analytics. A design that promises exact click analytics should lean toward status/header combinations that keep repeat requests visible to the server (or explicitly document the undercount), while a design optimizing for load reduction on stable, immutable destinations can accept that repeat-visit visibility is intentionally reduced.

Immutable vs mutable destinations

If the destination behind a short code is truly immutable once created, a longer-lived caching signal is safe. If destinations can be updated after creation (editable links), aggressive client-side caching risks serving a stale destination after an update — the redirect semantics decision is coupled to whether the product allows mutation.

The full read path is a hierarchy, not a single cache: an optional process-local cache, a shared distributed cache, and the authoritative database — each layer exists to absorb load the layer beneath it cannot handle economically.

Layers and their role

Eviction-policy choice is workload-dependent

No single eviction policy (e.g. allkeys-lru) is universally correct. LRU-style eviction is a reasonable default; for highly skewed traffic where a frequency-aware policy would preserve very popular entries better than pure recency, an LFU-style policy may perform better. The right choice depends on the actual access distribution and the cache implementation's available policies — measure before committing to one.

Negative caching

Caching a short-lived 'not found' result for nonexistent or malformed codes protects the database from repeated lookups of codes that will never resolve — this matters most under a random-code scanning pattern (see Failure & Scale → Malicious Random-Code Scan / Abuse). The trade-off: a negative cache entry must not cause a newly created mapping or a just-claimed custom alias to appear falsely missing. Either invalidate the negative entry explicitly at creation time, or keep the negative TTL short enough that the false-miss window is acceptable.

Cold-start behavior

A cache that starts empty (fresh deploy, restart, new region) has a 100% miss rate until it warms from real traffic. This is expected, not a failure — but a large simultaneous cold start under high traffic can create a stampede against the database; see Cache Stampede below.

A cached mapping must never outlive the authoritative mapping's expiry. This is a correctness rule, not a performance tuning knob, and it is easy to get quietly wrong.

The bug: a blind fixed TTL

Assigning every cached mapping the same fixed TTL (for example, 24 hours) regardless of the mapping's own expires_at can cause an already-expired URL to keep redirecting successfully straight out of cache — the authoritative store correctly considers it expired, but the cache does not know that, and nothing prompted it to check.

The fix: bound cache TTL by remaining mapping lifetime

cache_ttl = min(configured_cache_ttl, expires_at - now). For a mapping with no expiry, a normal cache TTL applies as usual. For a mapping with a near-term expires_at, the cache entry must expire no later than the mapping itself does.

Serving-time guard as the real enforcement mechanism

Even with TTL-bounding, defense in depth matters: if expiry metadata is cached alongside the mapping, the read path should verify it has not logically expired before returning a redirect, rather than trusting cache presence alone as proof of validity. Background deletion of expired rows is storage hygiene — it is not the mechanism that enforces expiry; the authoritative read path must treat an expired row as expired even if it still physically exists (see Complete Design → Expiry & Cleanup).

A viral URL is usually safe once its mapping is cached — but a cache restart, mass eviction, cold regional deployment, or many entries expiring at once can make a large volume of requests miss the same code(s) simultaneously, converting a normal cache miss into a database thundering herd.

Why a single miss is fine but many simultaneous misses are not

One cache miss triggers one database read — cheap. Thousands of concurrent requests for the same freshly-cold code, arriving before the first miss has finished repopulating the cache, can each independently trigger their own database read for the identical row, multiplying load without adding any value.

Mitigations

Sharding a distributed cache spreads different keys across nodes — it does not spread traffic for a single key, because one key still maps to exactly one shard. A viral URL concentrating all of its reads onto one key is a different problem from overall cache capacity.

Why horizontal sharding alone does not help

A clustered cache (e.g. Redis Cluster) partitions the keyspace across many nodes/slots, which is effective for spreading load across many distinct keys. But every read for one specific hot key still lands on the single node/shard that owns it — clustering does not automatically replicate or spread traffic for that one key.

Mitigation options — not just one

Detecting a hot key operationally

Per-key or per-shard request-rate metrics (where the cache/proxy layer exposes them) let operators identify a hot key before it saturates a node, rather than discovering it only after latency degrades.

The database's job in this design is narrow but must be fast and correct: primary-key lookup by short code on every redirect, and reliable uniqueness-checked writes on creation.

Primary-key lookup and indexing

A primary-key (or unique) index on short_code gives logarithmic lookup cost independent of table size. As long as the working set of frequently-accessed index pages fits in the database's buffer cache, repeated lookups for popular codes are served from memory rather than disk. Avoid quoting a fixed number of I/Os or an exact SSD latency as an architectural fact — these depend on hardware, index size, buffer-cache pressure, and concurrency; use them only as illustrative examples if included at all.

Primary vs replica reads

Cache-miss reads are commonly routed to a read replica to isolate them from the write path, while creation/update/delete always go to the primary. This works well as long as replica staleness is acceptable for the read path — see Replica Lag Correctness below for when it is not.

Replica lag correctness

Replication to a read replica is commonly asynchronous, not synchronous — do not silently assume otherwise. If a newly created URL is eagerly populated into the cache immediately after the authoritative write, most immediate reads are served from cache and never touch the replica. But if that cache entry is evicted, never populated, or the cache itself is cold, a read routed to an asynchronously-replicated replica could momentarily not yet know about a mapping that was just committed on the primary — producing a false miss for a brand-new URL.

Reasonable approaches: read from the primary for a short window immediately following a write ('read-after-write' routing), use synchronous or semi-synchronous replication if the product genuinely cannot tolerate this window, rely on eager cache population after commit to make the window rare in practice, or accept a very small eventual-consistency window if the product tolerates it. See Trade-offs → Read from Primary vs Read Replica on Cache Miss and Failure & Scale → Read Replica Lag.

Connection handling and uniqueness/expiry filtering

Connection pooling (in-app or via a proxy such as a transaction-pooling pgbouncer-style layer) lets a modest number of real database connections serve many concurrent application requests. Uniqueness enforcement for short codes and custom aliases should rely on a database-level UNIQUE constraint rather than an application-level pre-check (see Failure & Scale → Custom Alias Race). Expiry/status filtering (WHERE expires_at IS NULL OR expires_at > now(), and any ACTIVE/DISABLED status check) belongs in the authoritative read query itself, not only in a background cleanup job.

Publishing an analytics event is still a dependency unless it is deliberately designed not to be one. 'Async' does not automatically mean 'free' — a slow or unavailable broker must not be allowed to degrade redirect latency or availability.

Event schema

A compact event typically carries: short code, timestamp, and whatever analytics fields the product actually requires (which may be transformed, truncated, hashed, aggregated, or omitted for privacy — raw IP/User-Agent/Referer retention should not be treated as a default architectural requirement; see Complete Design → Requirements). If exact deduplication matters, include a stable event_id so downstream consumers can detect and discard duplicates.

Producer policy — the part that protects redirect latency

The redirect path should publish with a bounded timeout, using an asynchronous/non-blocking producer where the client library supports it, and a clear, explicit policy for what happens when publishing fails or is slow: buffer briefly if the transport supports it, or drop/sample the event and let redirect serving continue regardless. Redirect availability is the priority; analytics durability is a secondary, explicitly-chosen requirement layered on top when the product needs it. If analytics truly must be lossless, that is a legitimate requirement — but it creates a stronger durability obligation and a different trade-off, not a free upgrade. See Failure & Scale → Analytics Broker Unavailable.

Consumption side

Consumer groups read from the event transport and batch writes into the analytics store (batching amortizes per-write overhead — a batch of events per write is far cheaper than one write per event for most append-optimized analytics stores). Track consumer lag as an explicit operational signal; retries and a dead-letter path handle events that repeatedly fail to process rather than blocking the whole consumer group indefinitely.

Exact vs approximate counting

At-least-once delivery can produce duplicate events. If exact click counts matter, deduplicate explicitly using the stable event_id (a storage engine's built-in deduplication behavior is not sufficient on its own without a defined key and merge timing to rely on). If approximate analytics is acceptable for the product, exact deduplication can be intentionally skipped to reduce cost and complexity — state which one the design has chosen rather than leaving it implicit.

ClickHouse and BigQuery are both suitable examples of an append-optimized analytics store for this workload — either illustrates the pattern; neither is the only correct choice.

Partitioning analytics events by short code gives clean per-URL ordering, but it also means a single viral URL concentrates its entire event volume onto one partition — the opposite of what a partitioning scheme is usually meant to achieve.

Key by short_code

Advantages: all events for one URL land on the same partition, preserving per-URL ordering; downstream per-key stream processing (e.g. a simple running-count consumer) stays simple because it never needs to merge state across partitions.

Disadvantages: a viral URL can hotspot a single partition, since every one of its events is forced onto the same consumer, regardless of how many partitions exist overall.

Spread events for hot URLs / no strict per-code ordering

Advantages: distributes throughput evenly across partitions and avoids one URL's popularity dominating a single consumer's capacity.

Disadvantages: aggregation must merge counts across partitions rather than relying on a single ordered stream per URL, and strict event ordering for that URL is no longer guaranteed.

Which one to choose

The right choice depends on what the analytics actually need. Most URL-shortener click analytics are commutative aggregates — total counts, counts over time, top referrers — which do not require strict per-URL ordering and tolerate merge-based aggregation well. Per-URL ordering is genuinely necessary only if downstream processing depends on the sequence of events for a given URL, not merely their sum. Per-URL ordering is not automatically required; verify against the actual analytics product requirement.