Redis Fundamentals

Redis is an in-memory data structure server — a network-attached process whose core command execution path is largely serialized through one main thread, keeping its dataset in RAM and exposing it through a small set of highly-optimized commands.

That sentence hides three ideas doing all the work. First, Redis is in-memory: your data lives in RAM, not on disk. This is what makes it dramatically faster than a traditional database — every read is a memory lookup, not a disk seek. Second, it's a data structure server: it doesn't just store bytes, it understands strings, lists, hashes, sets, sorted sets, streams, and more, and provides atomic operations on each. Third, its core command execution path is largely single-threaded: commands against the keyspace are not interleaved with each other, which makes individual commands atomic and ordering easier to reason about.

It helps to compare Redis to what it isn't. A relational database like Postgres is optimized for arbitrary queries over large disk-resident datasets — joins, aggregations, indexes on multiple columns. Redis is optimized for fast key-based access to small-to-medium in-memory datasets with rich value semantics. The two solve different problems and almost every serious application uses both.

The defining characteristics

Redis was created in 2009 by Salvatore Sanfilippo — originally to solve one very specific problem, then rapidly expanded to solve a whole class.

The original problem

Salvatore was building LLOOGG, a real-time web analytics service. Every visitor generated events. A traditional SQL database couldn't keep up with the write volume, and a plain memcached couldn't express the aggregations he needed (real-time counters per URL, sorted by traffic). He needed something with memcached's speed but with data structures.

The first version of Redis was a few hundred lines of C that spoke a simple text protocol and supported strings and lists. Within months it added hashes, sets, sorted sets, pub/sub, and persistence. Every addition was driven by a real problem, not by a roadmap.

Why it took over

Redis solved three problems most systems had ignored:

Latency. Web applications were increasingly bottlenecked on database round-trips. Every read that could be served from memory was a dramatic speedup. Redis made that trivial to adopt — swap one call, watch P99 drop.

Ephemeral state. Sessions, rate-limit counters, feature flags, ephemeral queues — none of these need to survive an outage but they need to be fast, coordinated, and shared across app servers. Databases were overkill; local memory didn't scale. Redis fit exactly.

Coordination. Distributed locks, atomic counters, uniqueness checks. Doing these correctly on a relational database means holding row-level locks and hoping. Doing them on Redis means one atomic command. This is why Redis appears in almost every microservices architecture — not as a cache, but as the coordination substrate.

“Redis is what you get when you take memcached seriously as a general-purpose primitive.”

— adapted from Salvatore Sanfilippo's early talks

Redis's architecture is deliberately minimal. There is one process. Inside it, one main event loop. That single loop is what serves your traffic.

A Redis server is a single Unix process. Inside that process, an event loop (built on epoll / kqueue) waits for network activity. When a client sends a command, the loop reads it, parses it, executes it against the in-memory dataset, writes the response, and moves on. That core command execution path happens on one thread.

Everything else the server does — persistence, replication, expiration sweeps, cluster gossip — happens either on background threads (I/O and file writes) or inside the event loop between commands. The command hot path stays fast because it never yields to anything.

The main event loop

The loop's structure is roughly: poll network → parse command → dispatch to handler → serialize response → write to socket → advance expiration cursor → repeat. Because it's single-threaded, Redis's throughput per core is determined by how fast this loop can cycle — which is why command implementations are obsessively optimized for cache locality and pointer chasing.

I/O threads (optional)

Since Redis 6, network reads and writes can be offloaded to I/O threads — but command execution still happens on the main thread. This lets Redis use more cores for network work and improve throughput on some multi-core workloads, without introducing the correctness hazards of parallel command execution.

Persistence and replication threads

RDB snapshots are written by a forked child process — Redis uses fork(2) to create a copy-on-write snapshot of memory, then the child streams that snapshot to disk while the parent continues serving traffic. AOF rewrites use the same fork trick. Replication is asynchronous: a full resync uses a forked child to produce the RDB, after which the primary streams its write commands to replicas.

It's not one trick. It's four decisions that compound.

1. RAM instead of disk

The biggest single reason. As a mental model, RAM access is orders of magnitude faster than random disk access. When your entire dataset fits in memory, you skip the slowest layer of the storage hierarchy entirely.

A single-row lookup in a general-purpose database may hit RAM if you're lucky and storage if you're not. In Redis, the common case is a direct in-memory lookup through a purpose-built structure. The exact gap depends on workload, hardware, and caching, but the memory-first design is the point.

2. Simple data structures optimized for the hot path

Redis's internal data structures are chosen for cache locality and simplicity, not generality. Small hashes use a compact linear layout (listpack) that fits in a couple of cache lines. Sorted sets use a skip list for O(log n) range queries with better cache behaviour than a red-black tree. Strings are stored inline for short values. Every one of these decisions saves cache misses.

3. Single-threaded command execution

Sounds like a bottleneck. It often isn't. A single core running tight, cache-friendly code with serialized command execution can be extremely fast, and it avoids dataset-lock contention on the hot path. Adding parallel command execution would complicate ordering and coordination, so Redis instead uses extra threads mainly for networking and background work while keeping command execution on the main thread.

4. Custom binary protocol (RESP)

Redis speaks its own protocol (RESP — REdis Serialization Protocol) that's designed for speed of parsing. Commands are length-prefixed byte arrays. No JSON. No schema. Parsing is straightforward and low-overhead compared with general-purpose text-heavy application protocols.

Redis's power isn't in what it stores — it's in the shapes it stores things as. Pick the right shape and complex problems become one command.

Fast does not mean free. Some Redis commands are O(1), but others scale with collection size or response size; large collections, huge responses, SCAN-style iteration, and more complex operations can still be expensive. In Redis, "fits in memory" and "cheap to operate on" are related, not identical.

Redis stores data in RAM, but you don't have to lose it on a restart. You get to pick your durability model.

No persistence

Redis will happily run with save "" and appendonly no — pure in-memory, nothing written to disk. Use when: the data is truly ephemeral (session tokens, rate limit counters, cache entries you can regenerate). A crash means losing everything, but that may be perfectly acceptable and it makes Redis marginally faster.

RDB — point-in-time snapshots

Redis periodically writes a full dump of the dataset to a .rdb file. The dump is triggered by rules like "if 10,000 keys changed in the last minute" or by a manual BGSAVE command. The dump uses fork(2) — the child process serializes memory to disk while the parent keeps serving traffic. Copy-on-write means the dataset is not copied up front, but the fork still copies page tables, so on large datasets (especially on virtualised hosts) it can briefly pause the main thread.

Trade-off: RDB gives you a fast recovery (loading a single file is much faster than replaying commands) and a compact backup format, but between snapshots you can lose everything that happened after the last save. If snapshots run every 5 minutes and Redis crashes, you lose up to 5 minutes of writes.

AOF — every command, logged

AOF (Append-Only File) records every write command as it's executed. On restart, Redis replays the log to reconstruct the dataset. Because commands are logged individually, the durability window is much smaller than RDB — configurable via appendfsync:

AOF files grow over time — every INCR adds a line. Redis periodically rewrites the AOF (again using fork(2)) to produce a compact equivalent — one SET per key instead of a thousand INCRs.

Which do you use?

The right answer for most production Redis instances that hold anything worth keeping: both. RDB gives you fast recovery and clean backups; AOF gives you a small durability window. On restart, Redis prefers AOF (more up-to-date). If AOF persistence is enabled, Redis loads the AOF because it is the most complete; the RDB file is mainly useful for backups and faster restarts.

Persistence protects against restarts. Replication protects against machines dying.

Redis replication is asynchronous. The primary appends every write to an internal replication buffer and streams the buffer to each connected replica. The primary does not wait for replicas to acknowledge — it responds to the client the moment the write is applied locally.

This makes writes fast (no round-trip to replicas) but means the last few milliseconds of writes can be lost if the primary crashes before the buffer drains. Redis is unapologetic about this trade-off — it prioritizes throughput and simplicity over synchronous replication.

How a replica catches up

When a replica first connects, it requests a full sync: the primary triggers an RDB snapshot, streams it to the replica, and then streams every command that happened during the snapshot. Once caught up, the replica switches to incremental sync — a running stream of commands. If the connection drops briefly, the replica tries partial resync using an offset into the primary's replication backlog. This is dramatically cheaper than a full sync.

Reading from replicas

By default, replicas are read-only and can serve GET, HGETALL, ZRANGE, etc. This lets you scale read traffic horizontally — one primary handling writes, N replicas handling reads. The catch is staleness: because replication is async, a replica might be a few ms (or a few seconds under load) behind the primary. If you SET foo bar and immediately GET foo from a replica, you might see the old value.

Replication gives you copies. Sentinel decides when to promote one.

Redis Sentinel is a small companion process that watches your Redis primary and replicas. Typically you run three or five Sentinels, each on a separate machine. They gossip among themselves, ping the Redis nodes, and vote when something goes wrong.

The failover sequence

Step 1 — detection. Sentinel A pings the primary. No response. Sentinel A marks the primary as subjectively down (SDOWN).

Step 2 — quorum. Sentinel A asks the other Sentinels what they see. If a configured majority (usually 2 of 3, or 3 of 5) agree the primary is unreachable, they promote the mark to objectively down (ODOWN).

Step 3 — leader election. The Sentinels elect a leader among themselves (Raft-style) to run the failover.

Step 4 — replica selection. The leader picks the best replica — highest replication offset (most caught up), lowest priority number if there's a tie.

Step 5 — promotion. The leader sends REPLICAOF NO ONE to the chosen replica, turning it into the new primary. It reconfigures the other replicas to follow the new primary. Sentinels notify clients via pub/sub — modern clients subscribe to the +switch-master channel and update their connection automatically.

Result: an unattended failover without manual intervention. The actual timing depends on Sentinel settings, topology, network conditions, and how quickly clients detect and reconnect to the new primary.

Sentinel keeps one primary alive. Cluster splits your data across many primaries.

Redis Cluster is the built-in horizontal sharding solution. Instead of one primary, you run several — each responsible for a subset of the keyspace. The keyspace is divided into 16 384 hash slots. Every key's slot is CRC16(key) mod 16384. Each primary owns a range of slots (e.g., primary 1 owns 0–5461, primary 2 owns 5462–10922, and so on).

How the client routes commands

The client keeps a local map of slot → node. When you send GET user:42, the client computes the slot (CRC16("user:42") mod 16384), looks up which node owns that slot, and sends the command directly. There's no proxy in the middle — routing is entirely on the client side.

If the map is stale (a resharding happened), the target node responds with a MOVED redirect telling the client the new owner. The client updates its map and retries. This lets Cluster reshard live without any coordination stall.

Multi-key operations

A single Redis command touching multiple keys can only work if all keys hash to the same slot. Otherwise Cluster returns CROSSSLOT. The workaround is hash tags: {user42}:sessions and {user42}:cart both hash based on user42, so both live on the same node and can be operated on atomically together. Use hash tags whenever you have multiple keys that logically belong together.

Replication inside a cluster

Each primary in the cluster has its own replicas. A cluster with 3 primaries and 3 replicas per primary is 12 nodes total (3 primaries + 3×3 replicas). Failover of a single primary is handled by the cluster itself — no separate Sentinels needed. The cluster runs its own gossip protocol on a dedicated port (usually the client port + 10000).

You now have the mental model. Here's the shape of it.

Redis is one process, one event loop, one main command thread, one giant hash table in RAM. Everything else — persistence, replication, sharding, high availability, background expiration, optional I/O threads — is a supporting cast around that hot path. The design is not accidental; every complexity has been pushed off the command path so the command path stays fast.

The data structures aren't sugar — they're the product. INCR for atomic counters, ZADD for leaderboards, SET NX EX for locks, XADD for streams. Reaching for the right structure is what makes Redis feel effortless. Modelling a problem as "which Redis data structure fits this shape?" is the mental habit that separates users from experts.

Persistence, replication, and clustering are independent knobs, not a stack. You can pick RDB or AOF or both, replicate for reads or availability, cluster for scale — mix and match to fit your workload. The right shape for a session store is different from the right shape for a real-time leaderboard, and Redis lets you configure both without changing servers.