Production Engineering

Every sizing decision starts with three numbers: peak MB/s in, peak MB/s out, and retention window. Everything else falls out.

Kafka's operational simplicity hides a real risk: because it looks like "just install brokers", teams tend to skip capacity planning entirely and let the cluster grow organically. This works for a while, and then it stops working — usually all at once, in the middle of an incident. The correct way to size a Kafka cluster is boring and mechanical.

Step 1 — Compute steady-state throughput

Start with producer ingress: ingress MB/s = producers × avg record size × records/s. This is the raw data rate hitting the leader brokers. Then multiply by the replication factor (usually 3) to get the total disk write rate across the cluster — because every record hits every replica's disk. Finally add consumer egress, which is ingress × fanout where fanout is the number of independent consumer groups reading the topic. A busy cluster typically has fanout of 3–10.

For a topic doing 100 MB/s of ingress with RF=3 and 5 consumer groups, the cluster is doing 300 MB/s of disk write and up to 500 MB/s of network read. That's the number your NICs and disks have to handle — not the 100 MB/s the application team reported.

Step 2 — Pick partitions per topic

Partitions are the unit of parallelism, but they're not free. Every partition costs per-broker memory (index, buffers), per-consumer poll overhead, and rebalance time. There is no universal MB/s-per-partition target — a reasonable per-partition throughput range depends on record size, consumer processing cost and hardware, so benchmark your own workload rather than importing someone else's number. What's true in general: too few partitions caps parallelism, and a single consumer thread per partition is still the ceiling for that partition's processing rate regardless of what the "ideal" range is.

The concrete partition count for any workload has to come from your own measured per-partition throughput and expected consumer parallelism, not a fixed range. Pick the higher end of your estimate if the topic might grow, because you cannot reduce partitions later — adding is fine, removing is impossible without recreating the topic and reprocessing.

Step 3 — Size broker storage

A useful starting model is raw retained bytes ≈ ingress bytes/s × retention seconds, then multiplied by replication factor, and with additional headroom added for segment overhead, compaction working space, and the natural sawtooth of retention-driven deletion. For a 100 MB/s topic with RF=3 and 7 days of retention, the raw replicated figure is 100 × 7 × 24 × 3600 × 3 ≈ 181 TB before headroom — treat any specific headroom percentage (including ones elsewhere in this guide) as an illustrative starting point to validate against your own disk-pressure and rebuild-time tolerance, not a fixed constant.

Prefer many mid-size disks over few giant ones. Kafka's sequential I/O parallelizes cleanly across independent spindles/SSDs. Two 4 TB NVMes will out-throughput one 8 TB NVMe every time.

Step 4 — Pick broker count

A floor of 3 brokers is the minimum for RF=3 to make sense; beyond that, broker count is driven by total throughput, storage, and how many leader partitions any single broker can host before controller/metadata overhead and failover time become a problem — that ceiling is version- and hardware-dependent, so verify it against your own cluster rather than a fixed partitions-per-broker number. Fewer, bigger brokers minimize blast radius but maximize failover cost; more, smaller brokers trade the other way.

Kafka producer configuration offers 60+ settings. Eight of them decide whether your producer is durable, fast, and safe. The other 52 are noise until you have a specific problem.

acks = all

The single most important durability knob. Wait for the full ISR before treating a send as successful. Any other setting is either a footgun (acks=1, silent loss on leader failover) or fire-and-forget metrics territory (acks=0). If you're publishing anything a business would notice was lost, this must be all.

enable.idempotence = true

The broker deduplicates retried sends using a Producer ID and a monotonic sequence number per partition — even if a network blip causes the producer to retry, the broker won't append a duplicate. This protects exactly the producer→partition write path; it does not by itself make a read-process-write pipeline exactly-once, and it does nothing for side effects outside Kafka (an external database write, an HTTP call). For those, you need Kafka transactions across the specific topics/partitions involved — and even transactions don't extend atomicity to non-Kafka systems. Turn enable.idempotence on by default; it has no real downside for a correctly configured producer.

max.in.flight.requests.per.connection = 5

Max concurrent unacked batches per connection. Must be ≤ 5 for idempotence to guarantee ordering — the broker's sequence-number cache holds the last 5 seen per PID+partition. Setting this to 1 gives you ordering with a slight throughput cost; setting it above 5 with idempotence enabled will refuse to start.

compression.type = zstd

The single largest throughput win in almost every real workload. zstd is a strong default for most workloads — meaningfully better compression ratio than snappy at a modest CPU cost — but the actual throughput and ratio gain is highly data-dependent (text vs. already-compressed binary payloads behave very differently), so benchmark against your own message shapes rather than assuming a fixed multiplier. Use snappy only if you must interop with clients pinned to older Kafka versions.

linger.ms and batch.size

These two work together. linger.ms is how long the producer waits to fill a batch before sending; batch.size is the maximum batch bytes. Higher linger = better batching = higher throughput and higher latency. A small linger.ms (single-digit to low tens of milliseconds) is a reasonable starting point for most request/response-adjacent workloads, but the right value depends on how latency-sensitive your producers are and how small your records are — tune it against your own traffic rather than treating one number as universal. If records are small, raising batch.size gives linger.ms more room to build a full batch.

buffer.memory

The total heap the producer will use for un-sent records. If a broker slows down, records back up in this buffer. When it's full, send() blocks (up to max.block.ms) and eventually throws. Monitor buffer-available-bytes — if it approaches zero, something downstream is stuck.

delivery.timeout.ms

The total budget from send() to delivered-or-failed, including all retries. Set it slightly higher than your application's SLA — beyond this, the producer will give up and surface a failure to your code.

The producer configs assume the broker is holding up its end. These are the server-side knobs that decide what your producer configs actually buy you.

min.insync.replicas = 2

With RF=3, this means acks=all writes require at least 2 in-sync replicas. If only 1 replica is in sync (the leader), writes are rejected with NotEnoughReplicasException. This trades availability for durability: you'd rather see writes fail loudly than silently be one broker failure away from data loss. This is the correct trade-off for anything you care about.

unclean.leader.election.enable = false

Never turn this on for anything you care about. When set to true, a follower that is not in the ISR — one that fell behind the leader — can be elected leader if all in-sync replicas die. That replica becomes the leader with its shorter log, so any records the previous leader had but that hadn't replicated are silently lost (they are truncated from the old replicas when they rejoin and follow the new leader). This is a data-loss switch dressed up as an availability feature.

auto.create.topics.enable = false

When enabled, any producer publishing to a topic that doesn't exist causes Kafka to create it — with default partition count and default replication factor, which are almost always wrong. Turn it off. Topic creation should be an explicit, reviewed operation, not a side effect of a typo in a producer's config.

log.segment.bytes = 1 GB

Kafka splits each partition into segment files of this size. When retention kicks in, whole segments are deleted — Kafka never rewrites within a segment. Smaller segments make deletion and compaction faster and more granular; larger segments reduce file-handle count. 1 GB is a common starting point in many production deployments, but the right size depends on retention granularity needs and file-handle budget on your brokers — treat it as a default to revisit, not a fixed rule.

num.replica.fetchers

The number of follower-fetcher threads per broker. The default (1) is far too low for any broker hosting a meaningful number of partition leaders. Increasing it (a small single-digit value is a common starting point) lets follower replication proceed on multiple threads — otherwise follower replication becomes a queue behind a single thread and your ISR shrinks under load.

The producer decides whether the data gets to Kafka. The consumer decides whether your application processes it correctly.

enable.auto.commit = false

Auto-commit is the single biggest source of both duplicate and loss bugs in Kafka consumers. It commits offsets on a timer, whether or not your handler actually finished processing. Turn it off and commit manually after processing succeeds. This is the difference between at-least-once (safe with idempotent handlers) and "whoever wrote this hopes their processing time is under 5 seconds always".

The poll-interval trap

max.poll.interval.ms is the maximum time between two poll() calls before the coordinator considers you dead and kicks you out of the group. Its default is 5 minutes. If your handler takes longer than that on any record, the group rebalances mid-processing — and the record you were working on gets reassigned to another consumer, which also takes 5 minutes, gets kicked, and so on. This is the classic rebalance storm.

The fix is not "raise max.poll.interval.ms to infinity" — because that means a genuinely stuck consumer never gets replaced. The fix is: size `max.poll.interval.ms` for your P99 processing time × safety factor, and shrink `max.poll.records` so a single poll doesn't accidentally hand you 500 records that all need heavy processing.

session.timeout.ms and heartbeat.interval.ms

Independently of max.poll.interval.ms, the consumer sends heartbeats to the coordinator in a background thread. If the coordinator misses session.timeout.ms of heartbeats, the consumer is declared dead and its partitions are reassigned. heartbeat.interval.ms should be ~1/3 of session.timeout.ms. The defaults are session.timeout.ms=45000 (10000 before Kafka 3.0) and heartbeat.interval.ms=3000; the 45s / 15s pair in the sample config above is a tuned example, not the default.

fetch.min.bytes

The broker waits until this much data is available (or fetch.max.wait.ms expires) before responding to a fetch. Raising this trades a few ms of extra latency for much higher throughput on high-volume topics — the broker responds with big batches instead of thousands of small ones.

CooperativeStickyAssignor

As of recent Kafka versions, this is the recommended assignor for most consumer groups — Kafka's rebalance protocol continues to evolve, so check your broker/client version's documentation if you're pinned to an older release. Only partitions that need to move actually pause during a rebalance under the cooperative protocol — consumers that keep their partitions never stop reading. Compared to the older eager protocol (which stops the entire group on every rebalance), this is a significant stability improvement, especially for groups with many members.

Kafka → database: a concrete failure sequence

Walk the exact sequence: (1) consume event at offset 42; (2) apply it to Postgres; (3) process crashes before the Kafka offset commit; (4) on restart, event 42 is consumed again. If the Postgres write isn't idempotent, you now have a duplicate business effect. A workable pattern: inside the same Postgres transaction that applies the business mutation, also write the event's stable identity (or the partition/offset you just processed) to a tracking table, and commit both together. On redelivery, check that table first and skip if already applied. This needs care around partition concurrency (two partitions can be processed concurrently by different threads — don't let the tracking table become a bottleneck or a source of lock contention), ordering (a later offset recorded before an earlier one, if processing is out of order, can mask a gap), and what happens if recovery itself fails partway. A Kafka Connect sink connector's own guarantees vary by connector and must be checked individually — don't assume any specific connector gives you end-to-end exactly-once into an arbitrary external database.

After durability, the next outages usually come from correlated failures, skewed keys, oversized records, and one tenant overwhelming everyone else.

Rack / failure-domain awareness

RF=3 only helps if the replicas are in different failure domains. Set broker.rack to a real rack or AZ identifier and verify your critical topics actually spread replicas across them; otherwise a single rack outage can take out multiple replicas of the same partition and invalidate the resilience you thought RF=3 gave you. Client-side rack-aware reads are a separate optimization — don't confuse them with replica placement.

Hot partitions

A hot partition happens when traffic isn't actually uniform across keys: a celebrity user, a single large tenant, a low-cardinality partition key (e.g. partitioning by country when 80% of traffic is one country), or a custom partitioner with a bug. The partition-level parallelism Kafka gives you only helps if the partitioner actually spreads load — one overloaded partition caps the consumer that owns it regardless of how many other consumers in the group are idle.

Evidence: per-partition produce/fetch byte rates (not just topic-level aggregates), broker-level traffic imbalance, consumer lag concentrated on specific partition numbers, and leader skew (one broker hosting disproportionately more leader partitions for the hot topic). Mitigation depends on whether ordering matters for that key: if strict per-key ordering isn't required, add a bounded random/hash salt to spread a hot key across multiple partitions; if ordering must be preserved, you may need to isolate the hot entity onto its own topic/partition with dedicated capacity, or restructure the key.

Large messages

Kafka enforces message-size limits on the producer (max.request.size), broker/topic (message.max.bytes), and consumer (max.partition.fetch.bytes) — all three need to agree, or you'll get a confusing failure on one side only. Large messages also cost more than their raw size suggests: they reduce batching effectiveness, increase per-request memory pressure on brokers, and hurt replication/network throughput disproportionately. If you're regularly producing multi-MB payloads (images, video chunks, large documents), Kafka is usually the wrong place to store the blob — store it in object storage (S3-equivalent) and publish a small event carrying a reference/key to it instead.

Quotas and overload protection

Quotas (producer/consumer byte-rate limits, request-rate limits) exist so one misbehaving or unexpectedly busy client can't starve every other tenant sharing the cluster. Without them, a single runaway producer or a consumer stuck in a tight retry loop can consume enough broker CPU/network/IO to degrade latency for everyone else. The specifics of quota configuration are managed-service- and version-dependent — the engineering point to take away is that shared Kafka infrastructure needs admission control the same way any shared multi-tenant system does; don't assume good behavior from every client.

Not dashboards for looking pretty in review meetings — metrics that page you before customers do.

Kafka exposes hundreds of JMX metrics. Most of them are noise unless you're diagnosing a specific problem. This is the short list of metrics you should alert on directly, and what each one actually means.

UnderReplicatedPartitions > 0

The ISR has shrunk on at least one partition — some replica is behind. Severity depends on context: a brief blip during a rolling restart or a short GC pause that self-heals in seconds is expected and not an outage; a sustained shrink, especially on a topic with min.insync.replicas close to its replication factor, is a real risk — you're closer to a write-rejecting or data-loss scenario if the leader fails next. Treat a sustained or growing count as page-worthy, and investigate which broker is behind and why (disk, network, GC, fetcher starvation) before restarting anything.

OfflinePartitionsCount > 0

There is no leader for one or more partitions. Producers to those partitions are failing. Consumers are stuck. Page, drop everything, find and restart the responsible broker, or force a leader election with kafka-leader-election.sh.

Consumer lag (records-lag-max)

The gap between the log-end-offset and the consumer's committed offset. If lag rises steadily while producer rate is flat, the consumer group is losing ground. This is the single most useful application-level Kafka metric because it tells you the business effect of everything else on the dashboard.

Broker idle percent collapsing

Watch both request-handler and network-processor idle percent. When either stays low, the broker has stopped keeping up. Request-handler pressure usually means disk or replication work is backing up; network pressure usually means client traffic, replication traffic, or TLS cost is saturating the NIC path. In either case, the answer is usually to reduce load or add broker capacity — not just to give the node more CPU.

Controller or quorum failovers recurring

Repeated controller elections mean metadata instability. In KRaft mode, suspect quorum membership or network problems; in ZooKeeper-based clusters, suspect the ZK ensemble. Each failover briefly stalls metadata operations across the cluster, so recurring churn is an early warning, not background noise.

Broker JVM GC pauses becoming visible

Long broker GC pauses can shrink ISR membership and trigger downstream timeouts. Keep heap sizing boring: fixed heap, G1GC, and enough RAM left for the OS page cache, which Kafka depends on more than on a giant JVM heap. Benchmark your own pause times instead of copying a magic heap size or pause target.

Four failure patterns that recur in Kafka deployments, with their symptoms, causes and fixes. They are representative patterns, not reports of specific incidents.

The rebalance storm

Symptom: consumer group churns constantly, lag grows despite consumers being "up". Rebalance events every few seconds. Everyone's paging you.

Cause: max.poll.interval.ms is too low relative to the slowest handler. A slow record occasionally exceeds it → coordinator ejects the consumer → rebalance → the new owner starts on a cold cache → also exceeds the timeout → also ejected. The group is now in a death spiral.

Fix: raise max.poll.interval.ms to 2–3× your P99 handler time. Shrink max.poll.records so one poll doesn't return a giant batch. Switch to CooperativeStickyAssignor so genuine rebalances don't stop the whole group. Profile the slow handler.

"We lost a day of data"

Cause: unclean.leader.election.enable=true was set on a topic override. A follower that was not in the ISR was elected leader after a cascading failure. When the true leader came back, its log was ahead of the new leader — and Kafka truncated it to match. Every record the old leader held beyond the new leader's log end was silently truncated. Producers had received acks for those records (for example with acks=1, or with acks=all once the ISR had shrunk to the leader alone). Everyone assumed they were durable. They weren't.

Fix: unclean.leader.election.enable=false cluster-wide, audit every topic override, add an admission policy that rejects topic-level override attempts to enable it. This should be a hard rule, no exceptions.

Broker disk fills up

Cause: a rogue topic with retention.ms=-1 filled a broker's disk. Kafka refuses to accept new records and the broker goes offline. Cluster is running RF=3 so nothing is lost — but throughput crashes because every producer is now retrying to the two remaining replicas, which are also under pressure.

Fix: enforce a maximum retention.ms at admission time (topic-creation policy). Alert on disk utilization with enough lead time before the disk actually fills — the exact thresholds depend on your ingest rate and how fast you can react. Never allow retention.ms=-1 outside of intentional log-compacted topics. Keep meaningful disk headroom above what capacity planning says you need so unexpected growth doesn't turn into an outage.

"Kafka is slow" is almost never Kafka. Try these in order — the first two win most of the time.

Producer side

Enable compression. compression.type=zstd is one of the highest-leverage, lowest-risk changes for most workloads. Raise `linger.ms` from its near-zero default to let batches actually form. Raise `batch.size` if records are small — small batches with small records are pure per-request overhead. Exact values depend on your latency budget and record size; benchmark rather than copying a fixed number. Use one producer per JVM per SLA class, not per topic — producers share I/O threads inside the JVM. Never call `flush()` in the hot path — it defeats every batching optimization.

Broker side

Put log dirs on local NVMe/SSD, one dir per physical disk, not a single RAID volume. Kafka's parallelism is at the log-dir level. Never enable swap. Set vm.swappiness=1 and confirm on every broker. Leave plenty of RAM for page cache — the log is served from there via zero-copy, and RAM you give to the JVM is RAM you're taking away from Kafka's real cache. Bump `num.replica.fetchers` on brokers hosting many leaders. If broker idle percent stays low under normal traffic, scale broker capacity instead of assuming more CPU alone will fix it.

Consumer side

Increase `fetch.min.bytes` if you can tolerate a few ms of extra latency — it dramatically raises throughput on high-volume topics. Set `max.partition.fetch.bytes` big enough for your largest record — otherwise consumers stall. Parallelize processing inside the consumer thread using async I/O batches, or run one consumer thread per partition if the handler is truly CPU-bound. Never do synchronous DB calls per record without batching — it turns Kafka throughput into DB latency.

Healthy clusters still fail during changes. These are the two operations worth understanding before you touch production.

Rolling upgrades

Upgrade brokers one at a time (or in very small batches), and do not move on until the broker has rejoined ISR cleanly and the controller/quorum is stable again. Check the release notes for your exact Kafka version for compatibility and metadata-version steps — especially if you're crossing major versions or changing KRaft/ZooKeeper-era assumptions. Watch UnderReplicatedPartitions, controller/quorum stability, and consumer lag throughout, and have a tested rollback path before you start.

Partition reassignment

Reassignment moves full replicas, so treat it like a controlled data migration, not a metadata tweak. Use it to rebalance after adding brokers, shift leadership off a hot node, or change replication factor — but throttle it so the copy traffic does not saturate the same disks and NICs serving live traffic. Avoid large reassignments during peak windows, and watch both progress and user-facing latency while it runs.