etcd Interview Revision

Through the Raft consensus algorithm. Every write must be replicated to and persisted by a quorum of nodes before the leader acknowledges it. Linearizable reads route through a quorum check so they reflect every write acknowledged before them; etcd also supports serializable reads, which skip that check for lower latency at the cost of potentially returning slightly stale data.

Followers detect the missing heartbeat once the election timeout elapses (etcd's defaults are a 100ms heartbeat interval and a 1000ms election timeout, though real election duration depends on configuration and network/disk conditions). A follower becomes a candidate, increments the term, and requests votes; the first candidate to get a majority wins. Writes that require the leader fail or stall during the election window.

Watch creates a long-lived gRPC stream that delivers WatchEvents (PUT/DELETE) in revision order whenever a watched key or prefix changes, and can resume from a given revision as long as that history hasn't been compacted. Kubernetes controllers don't call etcd's Watch API directly — they watch Kubernetes API resources through kube-apiserver, which itself uses etcd's Watch underneath, with its own watch-cache in front.

Leases are TTL-bounded tokens. Keys attached to a lease are automatically deleted when the lease expires. They solve the problem of stale registrations: a client registers a key under a lease and heartbeats to keep it alive; if it crashes, the lease expires and the registration disappears without manual cleanup. Used for service discovery and distributed locks; a lease alone proves the client is maintaining it, not that the application is fully healthy.

Grant a lease (TTL ≈ acceptable lock-hold time). Txn: compare /lock/name's CreateRevision == 0 (doesn't exist); success: put /lock/name = my-id with the lease; failure: return the current holder. The compare-and-put is atomic, and the lease ensures the key is released if the holder crashes. Note this only guarantees exclusive ownership of the lock key — if the holder stalls past the lease TTL, another client can acquire the lock while the first resumes, so protecting the actual external action still needs its own safeguard (fencing token, idempotency check).

Quorum = floor(n/2) + 1. For 3 nodes, quorum is 2; for 5 nodes, quorum is 3. Adding one node to an odd-sized cluster (3→4) doesn't increase fault tolerance — 4 nodes still needs 3 for quorum, tolerating only 1 failure, the same as 3 nodes, but at a higher replication cost. That's why odd counts are the normal default rather than a hard protocol requirement.

Multi-version concurrency control: every mutating operation creates a new revision, and old revisions are retained until compaction removes them. Enables point-in-time reads (get a key at revision N), Watch replay from a past revision (bounded by compaction), and optimistic concurrency (CAS on mod_revision).

Because Raft's stability depends on timely disk and network I/O, not just process uptime. A spike in WAL fsync latency or peer round-trip time can cause missed heartbeats, which triggers unnecessary leader elections even though every etcd process is technically 'up.' This is why disk and network signals, not just process health, are the first things to check when a cluster is unstable.

When you need high-volume application data writes (use a database or cache built for that), large object storage (use blob storage or a general-purpose database), or when eventual consistency is an acceptable trade-off for simpler operations. etcd's strong consistency and fsync-per-commit durability model make it a poor fit for anything beyond coordination-scale write volume — the exact ceiling depends on hardware and workload, not a fixed number.

etcd is the sole state store for Kubernetes objects, but kube-apiserver is the boundary in front of it — clients and controllers interact with the API Server, which reads from and writes to etcd on their behalf. If etcd loses quorum, API Server writes that depend on it fail, but existing workloads generally keep running since kubelets operate on already-known Pod specs.

They're both key-value stores but for completely different use cases. Redis: throughput, caching, pub/sub. etcd: coordination, strong consistency, small data. Saying 'etcd is basically a distributed Redis' reveals a fundamental misunderstanding of their consistency models.

Saying a 3-node cluster can 'handle 2 failures' is wrong — it handles 1 failure. Quorum for 3 nodes is 2. After 2 nodes fail, only 1 remains — no majority. Know this cold: 3-node tolerates 1, 5-node tolerates 2, 7-node tolerates 3.

In Kubernetes, you never connect directly to etcd. Everything goes through the API Server. Saying 'a controller watches etcd' instead of 'a controller watches the API Server which reads from etcd' reveals you're describing architecture without operational experience.

etcd needs fast, low-latency storage, regular compaction, and a practiced backup/restore procedure. Describing it as 'just three etcd processes' without discussing disk sensitivity, compaction policy, or backup strategy signals you haven't operated it in production.

Use etcd for leader election (only one scheduler instance active at a time) and a distributed lock to prevent duplicate job execution. Each scheduler instance holds a lease; the one that wins the /election/scheduler key is active. Before running a job, it does CAS on /locks/job/{id} — but call out explicitly that winning the lock only guarantees exclusive ownership of the lock key, not at-most-once execution of the external job, unless the job itself is idempotent or the lock includes a fencing token the downstream system checks.

Services watch /config/{service}/ for changes. Config updates are atomic via etcd transactions — updating several config keys together in a single Txn means the cluster commits them as one atomic state transition at one revision, so no client observes a partially-updated set. Individual watchers still receive and process that update independently and at their own pace, so 'atomic commit' is not the same as 'every client reacts at the same instant.' Use revision-based optimistic locking to prevent stale overwrites.

A control plane needs to discover upstream services and push config updates to proxies in real time. Store service registrations and proxy configuration in etcd, and have the control-plane component watch etcd and stream updates to proxies (the xDS pattern used by Envoy-based control planes). This is the general shape of the problem; verify any specific product's current architecture (e.g., whether it talks to etcd directly or through another API layer) against its current documentation rather than assuming a fixed implementation.