etcd Fundamentals

etcd is a strongly consistent, distributed key-value store designed for reliability, not throughput. It stores small, critical coordination data with well-defined consistency guarantees rather than best-effort behavior.

The name is a portmanteau: /etc (the Unix directory for system configuration) and d (for distributed). That naming choice tells you exactly what etcd was designed for — it's the configuration and coordination backbone of a distributed system, just as /etc is the configuration directory of a single Linux machine.

etcd was created by CoreOS in 2013 to solve a specific problem: when running a cluster of machines, services need a reliable source of truth. Which node is the current leader? What's the current database connection string? Which feature flags are enabled? Storing this in application memory leads to drift. Storing it in a traditional database introduces a new single point of failure. etcd is built to be that store — highly available, strongly consistent, and operationally simple.

The guarantees that matter

Before etcd, every distributed system solved the same coordination problems differently and badly. etcd came from the pain of running real clusters at scale.

The configuration drift problem

Imagine 50 microservices, each holding its own copy of the database host, port, and feature flags. When you need to rotate the database host, you have to restart 50 services in the right order, cross your fingers, and hope they all pick up the change before any user hits an inconsistent service. In practice, some services get the old config, some get the new config, and debugging why half your checkout flow behaves differently from the other half costs you hours of on-call time.

etcd solves this with a single source of truth. Every service reads config from etcd on startup and watches for changes via the Watch API. When the database host rotates, you write one key and every watching service is notified quickly, without polling. The config can't drift because there's only one copy.

The leader election problem

Another classic distributed systems problem: you have five scheduler instances, but only one should execute the daily billing job. How do they agree on which one wins? Without a coordination service, you end up implementing your own mutex over a database row — which doesn't handle the case where the winning instance crashes mid-job and holds the row lock forever. etcd's leases handle this: the winning instance holds a key backed by a TTL lease, and if it crashes, the lease expires and the next instance can win. The lock auto-heals.

An etcd cluster is a group of member nodes running the same binary, coordinating via Raft to maintain a replicated key-value store with a single, well-defined history of writes.

Every member runs the same Go binary. There are no separate master and worker roles — each member is capable of being the leader, and leadership rotates via election when the current leader fails. The cluster presents a single logical view: for linearizable reads and all writes, clients get a result consistent with every write acknowledged before it, regardless of which member they contact, because every committed write has been replicated to a quorum before the leader acknowledged it.

The WAL and snapshots

etcd persists state in two forms: the write-ahead log (WAL) and periodic snapshots. The WAL records every operation in order — it's the source of truth for recovery. On restart, etcd replays the WAL from the most recent snapshot forward. Snapshots compact the log to avoid unbounded disk growth. The combination of WAL + snapshots gives etcd its durability guarantees without requiring synchronous disk flushes on every read.

The gRPC API

etcd exposes its entire API over gRPC with protobuf serialization. The API surface is intentionally small: KV (Put, Get, Delete, Txn), Lease (Grant, Revoke, KeepAlive, TimeToLive), Watch (Watch), Cluster (MemberList, MemberAdd), and Maintenance (Defragment, Snapshot, AlarmList). Every call is synchronous: a write returns only after it is committed to the Raft log and applied to the state machine, and reads are linearizable by default, so they also go through Raft consensus (serializable reads can be served by a single member and may be stale).

The etcd KV API is small but deep. Understanding revisions, MVCC, and prefix queries unlocks the full power of the store.

Revisions and MVCC

Every write to etcd advances a cluster-wide revision counter. The revision is a monotonically increasing integer that represents the cluster's history. When you put a key at revision 42, the key exists at revisions 42 and above (until it's deleted or overwritten). This is multi-version concurrency control (MVCC): etcd maintains previous versions of keys until compaction removes them. Clients can fetch the value of any key at any past revision with Get(key, WithRev(n)).

Prefix queries

etcd supports range queries over sorted keys. Since keys are lexicographically ordered, all keys under /services/ can be fetched with a single Get call: Get('/services/', WithPrefix()). This is how Kubernetes fetches all Pods in a namespace in one call — it reads /registry/pods/my-namespace/ with prefix. The range query is O(n) in the number of matching keys, backed by bbolt (BoltDB) for fast sorted reads.

The Watch API is etcd's change-notification mechanism — it tells clients what changed in the key space without polling. It's not a general-purpose event bus: it notifies on etcd state changes, not on arbitrary application events the way Kafka or a message queue would.

A Watch subscribes to changes on a key or prefix starting from a given revision. When a matching key changes, etcd pushes a WatchEvent to every watcher: the event type (PUT or DELETE), the new and old key-value pairs, and the revision. Within a single watch stream, events arrive ordered by revision.

A watcher can specify a start revision: Watch(key, WithRev(42)) receives events from revision 42 forward, as long as that history hasn't been compacted away. This makes reconnect-and-resume straightforward: save the last processed revision, and on reconnect start the watch from there. But the guarantee has a boundary — if the requested revision has already been compacted, the watch fails and the client must re-list current state instead of replaying from an arbitrarily old point. Consumers should also treat their own processing as idempotent, since retries and reconnects on the client side can replay an event the application already handled.

Leases are etcd's mechanism for ephemeral ownership. A key attached to a lease lives and dies with that lease — automatically.

A lease is a time-bounded token issued by etcd. The client that holds a lease must send periodic KeepAlive RPCs (heartbeats) to renew it. If the keepalives stop — because the client crashed, the network was severed, or the process was killed — the lease expires after its TTL. When a lease expires, etcd atomically deletes every key attached to it and pushes DELETE WatchEvents to any watchers of those keys.

The practical consequence is powerful: service health, session management, and distributed lock timeouts all reduce to lease TTLs. A service that wants to signal liveness attaches a health key to a lease and heartbeats. A client that crashes stops heartbeating, the lease expires in TTL seconds, and the health key disappears. No human intervention, no cleanup daemons, no stale registrations.

etcd transactions are atomic conditional writes — the coordination primitive that makes distributed locks, leader election, and safe config updates possible.

An etcd transaction (Txn) has three parts: a compare block (the conditions), a success block (operations to run if all conditions pass), and a failure block (operations to run if any condition fails). The entire transaction is applied atomically at a single revision — there's no possibility of interleaving between the compare and the write.

The compare block can test: the current value of a key, its create revision, its mod revision, or its version. The most common pattern is compare-and-swap: 'put this key if and only if its current value equals X'. This is optimistic locking over key-value pairs.

etcd is exceptional for coordination. It is a poor choice for everything else. Knowing the distinction is what separates a good architecture from an over-engineered one.

etcd is the right tool for

etcd is the wrong tool for

etcd is not a general-purpose database. It's optimized for small, infrequently-written coordination data rather than high-volume application data. The default storage quota is 2 GiB, and official guidance suggests 8 GiB as a reasonable upper bound for normal environments — etcd warns at startup if you configure something larger. Using etcd to store large blobs, analytics data, user records, or high-volume event streams works against its design and operational model.