etcd Production Engineering

etcd is I/O sensitive above everything else. Get the hardware right and the cluster stays stable. Get it wrong and disk latency becomes your most common source of instability.

Node count: odd numbers, sized to the failure tolerance you need

Run an odd number of etcd members — typically 3 or 5. Even numbers don't buy extra fault tolerance for the added replication cost: a 4-node cluster still needs 3 nodes for quorum, so it tolerates only 1 failure, the same as a 3-node cluster, while paying to replicate to one more member. The standard choices are 3 nodes (tolerates 1 failure, the default for most production clusters) and 5 nodes (tolerates 2 failures, useful when you need headroom during rolling maintenance).

Disk: fast, low-latency, ideally dedicated

etcd's WAL write path is synchronous — it calls fdatasync before acknowledging a write, so every commit depends on disk latency. On a slow or shared disk this shows up as missed heartbeats and unnecessary leader elections. Official etcd guidance treats this as a strong recommendation rather than a hard protocol rule: use fast, low-latency storage (SSD is typical in production), and avoid co-locating etcd's data directory with other heavy-I/O workloads where contention is likely. Whether you need a fully dedicated disk/NVMe device depends on your workload and what else shares the host — it isn't a universal requirement for every deployment.

Regular etcd snapshots are a baseline requirement for any cluster you cannot afford to rebuild. A corrupted etcd cluster without a recent backup means rebuilding your entire Kubernetes cluster from scratch.

etcd provides a built-in snapshot API: etcdctl snapshot save backup.db. This creates a consistent point-in-time snapshot of the entire key-value store, including all keys, leases, and the current revision. It does not require the cluster to be stopped — snapshots are taken online. Backup cadence should be driven by your recovery objectives, not a fixed universal schedule: decide how much data loss (RPO) and how much downtime (RTO) are acceptable, then choose a snapshot frequency and retention window that meet those targets, and store backups somewhere independent of the cluster itself.

Restoring is more involved: current etcd tooling separates the two concerns — etcdctl takes snapshots from a live cluster, while etcdutl snapshot restore builds new, independent data directories from a snapshot file (verify the exact commands for your deployed etcd version, since restore tooling has moved between etcdctl and etcdutl across releases). Restore creates a new logical cluster from the snapshot; it does not patch an existing one in place. Practice this procedure regularly in a non-production environment — the incident is the wrong time to be learning the restore workflow.

MVCC history accumulates until it's compacted; many deployments enable auto-compaction so this happens continuously rather than requiring a manual step. Without any compaction policy, the database grows unbounded.

etcd's MVCC model means every write is a new revision of a key, and old revisions are retained until compaction removes them. An etcd cluster running for a long time with no compaction policy accumulates old revisions and grows the database even if the current key space is tiny. Compaction removes history older than a given revision, reclaiming logical space. Defragmentation is a separate step that reclaims physical disk space after compaction, because bbolt (the underlying storage engine) doesn't automatically return freed pages to the OS.

Five Prometheus metrics tell you almost everything you need to know about etcd cluster health. Know them before you're on call.

etcd holds every secret in your Kubernetes cluster. Securing it is not optional.

etcd supports mutual TLS (mTLS) for both peer-to-peer communication (between etcd members) and client-to-server communication. In Kubernetes, the API Server uses a client certificate to authenticate to etcd, and etcd presents its own certificate that the API Server verifies. No other process should be able to connect to etcd directly.

Enable --client-cert-auth=true and --peer-client-cert-auth=true. Restrict etcd's listen address to the cluster-internal network — it should never be reachable from outside the cluster. In cloud environments, security group rules should allow port 2379 (client) and 2380 (peer) only from specific IP ranges.

In Kubernetes, etcd trouble doesn't mean the whole cluster goes down — it means the control plane can't reliably change state. Every operator needs to know how that failure manifests and how to recover.

When etcd loses quorum (more than (n-1)/2 members are down), kube-apiserver writes that require etcd fail. kubectl apply/create/delete commands hang or error. New Pods cannot be scheduled and controllers can't reconcile new state. Existing Pods generally keep running — kubelet operates on its already-known Pod spec independently of etcd availability — but no new workloads can start and no cluster state changes can be persisted. Exactly what reads still succeed depends on API server caching behavior, so don't assume the whole cluster is simply 'read-only' in a precise sense; writes are what's definitively blocked.

Recovery approach: first identify which members are down. If they can be restarted, restart them — etcd re-syncs from the remaining quorum via the Raft log. If a member's data is corrupted, remove it from the cluster and add a fresh member. Only if quorum is permanently unrecoverable (not just temporarily down) does restoring from the most recent snapshot become necessary.

Every etcd operator needs a practiced disaster recovery runbook. The moment of the incident is the wrong time to be reading documentation.

Scenario 1: Single member failure. The remaining 2/3 members maintain quorum. Writes continue. Simply restart the failed member — it re-syncs from the leader via the Raft log. If the member's data is corrupted, remove it with etcdctl member remove <ID>, provision a new instance, and add it with etcdctl member add.

Scenario 2: Multiple members down, quorum lost. First distinguish temporary from permanent: if the downed members can be restarted and rejoin with their data intact, quorum is restored without any data loss — this is the common case for things like a reboot, a brief outage, or a transient network partition. Only when members are genuinely unrecoverable (destroyed disks, permanently lost nodes) do you fall back to restoring from the latest snapshot onto fresh members, which recovers state up to the snapshot point and loses any writes after it.