etcd Engineering Insights

Raft is the consensus algorithm behind etcd's strong consistency guarantee. Understanding Raft means understanding why etcd behaves the way it does under failures.

Raft was designed around understandability as an explicit design goal, in contrast with Paxos, which is widely regarded as difficult to implement correctly even though it is formally sound. Raft divides consensus into three independent subproblems: leader election (picking one node to coordinate), log replication (the leader distributing entries to followers), and safety (ensuring the elected leader has all committed entries). Each subproblem has clear invariants that are easy to reason about and test. etcd was created at CoreOS and has been built on Raft since the beginning, so that understandability is paid back directly in a codebase that many contributors and operators have to reason about under failure.

Log matching safety

Raft's safety invariant: if two entries in different nodes' logs have the same index and term, those entries are identical, as are all entries before them. This (the Log Matching property) is guaranteed by the AppendEntries consistency check: a follower accepts new entries only if its log contains an entry matching the leader's preceding index and term. A separate rule, the election constraint, ensures the elected leader always has all committed entries: a candidate can only win if its log is at least as up-to-date as any node that votes for it, meaning its last log entry has either a higher term, or the same term and an index at least as large as the voter's last entry.

The hardest architectural decision isn't whether to use etcd — it's knowing when NOT to use it and what to use instead.

etcd vs Redis

Redis is optimized for throughput and low latency; etcd is optimized for coordination correctness. Redis Cluster distributes keys across shards and each shard is independently consistent, but the cluster as a whole doesn't provide the same linearizable, single-log view across all keys that etcd does — a meaningfully different consistency model, not simply 'less consistent.' For distributed locking, Redis's Redlock algorithm has been publicly debated for its behavior under clock skew and long GC/process pauses, while etcd's CAS transactions are backed by Raft's linearizable log. That difference matters more for locks protecting correctness-critical external actions than for best-effort use cases like rate limiting or cache-stampede protection, where Redis is a reasonable fit.

etcd vs ZooKeeper

ZooKeeper is the older distributed coordination service (used by Hadoop and, historically, Kafka). It uses ZAB (ZooKeeper Atomic Broadcast) for consensus, provides a hierarchical namespace (znodes), and exposes ephemeral nodes and watches. etcd covers similar conceptual ground — leases are roughly analogous to ephemeral nodes, Watch to ZK watches, CAS to ZK version-based conditionals — with a simpler operational footprint: a single Go binary, a gRPC API, and no JVM dependency. ZooKeeper remains in active production use in many existing systems; etcd's main practical advantage for new systems is operational simplicity and being Kubernetes' native backing store, not a categorical correctness advantage.

etcd vs Consul

Consul bundles service discovery, health checking, a KV store, and a service mesh in one product. etcd provides only the strongly consistent KV store and leaves discovery/health-checking/mesh features to other tools. Consul's Raft-backed KV layer is also strongly consistent, but Consul's broader feature set introduces more moving parts. In Kubernetes environments etcd is used because it's the built-in backing store behind kube-apiserver; outside Kubernetes, Consul can be a better fit when integrated health checking and DNS-based discovery outweigh the benefit of a smaller, single-purpose store.

etcd's storage engine is a custom MVCC layer on top of BoltDB. Understanding it explains both its performance characteristics and its operational limits.

bbolt is an embedded key-value store that organizes data in a B-tree structure on disk. It provides ACID transactions with MVCC semantics — readers operate against a consistent snapshot without blocking writers. etcd uses bbolt as its persistence layer and implements its own MVCC index on top: an in-memory index that maps each key to its revisions, while the key-value data itself is stored in bbolt keyed by revision; because revisions are indexed too, ranging over revisions with watchers is efficient.

The practical consequence: reads are fast B-tree lookups, and sequential scans of a prefix are efficient range scans. As the backend database grows, operation latency can be affected, which is why etcd's operational guidance treats keeping the database reasonably small as good practice rather than storing large volumes of data in it. Compaction logically removes old revisions; defragmentation is the separate step that returns the resulting free space to the filesystem, since bbolt doesn't reclaim freed pages automatically.

etcd handles coordination workloads, not high-throughput business data. Its write path has deliberate bottlenecks that reflect its design goals.

Every write goes through the Raft leader and requires a quorum round-trip plus an fsync before it commits. Actual sustained write throughput depends heavily on hardware, request size, and network topology, so there's no single number that applies universally — but the design intentionally trades raw throughput for durability and linearizability. The useful mental model isn't a specific writes/sec ceiling; it's that etcd optimizes for coordination correctness and predictable consensus behavior rather than high-volume application-data throughput.

These are representative failure scenarios — illustrative patterns worth recognizing, not claims about specific documented incidents. Each one teaches something specific about how etcd fails in practice.

etcd's data model is richer than a simple string→string map. Understanding revisions, leases, and the key space hierarchy is essential for building correct applications on top of it.

The cluster revision is a monotonically increasing counter over the key space. It starts at 1 when the store is created, and each atomic mutating operation on the key space (a put, a delete, or a transaction that writes) creates a new revision — reads, watch progress notifications, and operations that don't change the key space do not advance it. When you read a key, you get its value along with create_revision, mod_revision, and version, which together describe the key's history.

The key space is a flat byte-string namespace with no native directory structure — a hierarchy like /services/payment/instance-1 is purely a naming convention the application imposes. The only structure etcd itself understands is lexicographic key ordering, which is what makes prefix range scans efficient. This is simpler and more predictable than ZooKeeper's hierarchical znodes, but it means applications have to think deliberately about key naming.