Engineering Insights

Kafka's foundational choice — store everything as an immutable log — is what enables every other feature. Everything else is a consequence.

In a traditional queue, a message is a transient item: delivered, acknowledged, deleted. That means the broker must track per-message state, coordinate delivery, manage retries, and eventually clean up. It also means once a message is gone, it's gone — and reconstructing the past is impossible.

Kafka chose the opposite. Records are appended and kept for as long as a topic's retention or compaction policy says — not deleted the moment a consumer reads them. Consumers track their own read position instead of the broker tracking per-consumer delivery state. The broker becomes dramatically simpler: a sequential file writer, a socket sender, and a metadata bookkeeper. State that used to live in complex broker data structures now lives mostly as an integer — the offset — tracked per consumer group.

That single flip is where every downstream capability comes from:

Why retention is independent of consumption

This is the second-order insight that falls out of "the log is the truth," and it's worth separating from the first one. In a traditional queue, a message's lifecycle is tied to delivery: consume, ack, the message is gone. In Kafka, a consumer advancing its offset and a record leaving the log are two unrelated events — the first is driven by what a consumer group has read, the second is driven entirely by the topic's retention or compaction policy. That decoupling is what makes replay, multiple independent consumer groups reading the same topic at different speeds, and rebuilding a downstream projection from scratch all possible without coordinating with anyone else reading the topic. The cost is symmetric: if a consumer falls behind for long enough, retention can delete records it hasn't read yet — "the log remembers everything" is only true within the window your retention policy actually keeps.

“The log is the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time.”

— Jay Kreps — I ❤️ Logs

Some messaging and log-shipping systems push messages to consumers; Kafka follows the more traditional split where producers push to the broker and consumers pull. The rationale is worth understanding, because it drives half of what makes Kafka different.

The problems with push

In a push model, the broker sends messages to consumers as soon as they arrive. This sounds simple until you have to manage what happens when the consumer is slower than the producer.

The broker has to buffer. That buffer has a limit. When the limit is hit, the broker has to either drop messages (silent data loss) or reject writes (back-pressure the producer, which the producer usually doesn't handle gracefully). Either way, the broker has to know each consumer's capacity — which it can only guess at — and it has to adapt in real time as consumers slow down or speed up.

Now multiply that by 100 independent consumers per topic, each with different processing speeds, and you have a distributed flow-control problem that never quite works. Push-based brokers spend a large fraction of their code and their operational complexity managing this.

How pull solves it

With pull, the consumer decides how fast to read. It asks for a batch. It processes the batch. It asks for the next batch. The broker still stores the records in the log and still tracks per-group committed offsets — that bookkeeping doesn't disappear. What goes away is the broker having to *actively manage delivery pace per consumer*: it doesn't need to guess each consumer's capacity or decide in real time whether to drop or back-pressure. A slow consumer simply falls behind — its lag grows — rather than forcing the broker into a buffering decision.

The trade-off is latency on empty topics: if there's nothing to fetch, a pull-based consumer either polls tightly (wasted CPU) or waits (added latency). Kafka's answer is long-polling: the consumer sends a fetch, the broker holds it open until fetch.min.bytes of data has accumulated or fetch.max.wait.ms elapses. This gives you push-like latency on an otherwise pull-based model, without the broker having to actively manage per-consumer delivery pace.

Many replicated systems let followers serve reads. Kafka consumers don't by default, even when followers are sitting idle with the same data (follower fetching, KIP-392, is an opt-in exception covered below). The reason matters.

In MySQL, Cassandra, DynamoDB read-replicas, and most "eventually consistent" datastores, replicas other than the writer (in Cassandra, any replica) serve reads. It spreads load. It's obvious. Why does Kafka refuse to do it?

The high-watermark problem

In a replicated log, at any moment a follower may have records past the high-watermark — records the leader has appended but which are not yet safely replicated to the full ISR. These records are speculatively on the follower's disk, but they could still be truncated if the leader fails and a different follower takes over with a shorter log.

If a consumer read these speculative records from a follower, it could see records that later disappear. That would break Kafka's core durability contract: once a consumer reads a record, that record is committed and will never vanish. Kafka refuses to expose this hazard, so by default consumers read from the leader — which is the only broker that knows the current high-watermark authoritatively.

Client routing becomes trivial

Because there's exactly one place to read from per partition, client routing is a lookup — no stale-replica decisions, no read-your-writes gymnastics, no consistency tuning. "Find the leader from metadata, connect there." Every client library does this identically.

Throughput isn't the bottleneck anyway

The throughput a single broker can serve via zero-copy sendfile depends on NIC speed, TLS usage, and hardware — benchmark your own cluster rather than assuming a fixed number. The general point holds regardless of the exact figure: followers are rarely the throughput ceiling for reads, brokers are. Distributing reads across followers would help with a bottleneck that doesn't exist, while trading away Kafka's consistency guarantee.

No single trick explains Kafka's throughput — it's the combination of a handful of unglamorous mechanical decisions, each boring on its own, that compound into something fast.

Trick 1 — Sequential I/O only

Producers append to the tail of one file per partition. Consumers read forward from a chosen offset. That's it — log segments are written sequentially.

Sequential I/O is meaningfully faster than random I/O on both SSDs and spinning disks — the exact multiple varies by device and workload, so treat any specific ratio as illustrative rather than a benchmark you can quote. Almost every database that struggles under write load struggles partly because random writes fragment files and destroy cache locality. Kafka's log segments sidestep this by being append-only. It's not a clever technique — it's a refusal to do the thing that hurts.

Trick 2 — Page cache is the cache

Kafka doesn't maintain a JVM-side cache. It writes log segments through the filesystem and lets the kernel page cache hold hot data. This has two profound consequences.

First, the JVM heap can stay modest relative to total machine RAM — the exact size depends on partition count and broker load, but it doesn't need to scale with data volume the way it would for an in-process cache. GC pauses stay shorter as a result, which helps broker stability. Giving Kafka a much larger heap "for performance" usually backfires: it shrinks page cache (Kafka's real read-path cache) without meaningfully helping GC.

Second, the more RAM you leave for the OS rather than the JVM heap, the more of the log can be served from page cache instead of disk. A consumer that's only slightly behind the tail can often read directly from RAM with no disk I/O at all — which is why, for workloads where consumers stay close to real-time, the RAM available to the OS matters as much as raw disk speed for read throughput.

Trick 3 — Zero-copy sendfile

When a consumer fetches, the broker asks the kernel to send bytes directly from the page cache to the network socket using sendfile(2). No copy into user space. No CPU-mediated data movement.

Many message queues historically did this instead: read bytes from file → copy to kernel buffer → copy to user-space JVM buffer → copy to another kernel buffer for the socket → send — several CPU-mediated copies per record. Kafka's plain-text fetch path avoids those extra copies via sendfile. Two caveats worth knowing: zero-copy only applies to the plaintext network path — enabling TLS on the broker-client connection forces data through user space for encryption, losing the zero-copy benefit on that hop (inter-broker replication and client connections are affected differently depending on configuration). And the exact mechanism and its applicability can shift across Kafka client/broker versions and OS platforms, so verify against the version you're actually running rather than assuming it always applies.

Kafka performance is not magic

Every mechanism above explains why Kafka can be fast — none of it guarantees your deployment is fast. Kafka gets slow for ordinary, diagnosable reasons: too few partitions capping parallelism, a hot partition absorbing traffic a single consumer can't keep up with, acks=all with a geographically spread-out ISR forcing every write to wait on a slow replica, poor batching (tiny linger.ms, small records, no compression), compression CPU becoming the bottleneck on underpowered brokers, disk or network saturation, replication traffic competing with client traffic, oversized records hurting batching and memory, a slow downstream consumer or database turning Kafka lag into business lag, and cross-zone/cross-region traffic adding latency and egress cost. When something feels slow, look for which of these is actually true before assuming Kafka itself is the ceiling.

"Kafka or RabbitMQ?" is one of the most asked questions in engineering interviews and design reviews. It's also the wrong question — they solve different problems.

The fundamental data model

RabbitMQ's core abstraction is queues and exchanges with delivery/ack semantics. A message enters through an exchange, gets routed to one or more queues, waits for a consumer, gets delivered, gets acked, and is typically removed once fully acknowledged. RabbitMQ does support durable queues and persistent messages — "transient" isn't an inherent property of the broker, it's a per-queue/per-message choice. State (queue depth, unacked messages, per-consumer channels) lives in the broker either way; the broker is the source of truth about what has and hasn't been delivered.

Kafka is a log of durable events. Records are written to partitions, replicated, retained by policy, and read by consumers who track their own progress. State lives on the log (records) and in the consumers (offsets). The broker doesn't know or care who's read what.

What each is good at

RabbitMQ excels at task queues, complex routing, and per-message features. Exchange types (direct, topic, fanout, headers) give you flexible routing without touching producers or consumers. Per-message TTL, priority queues, dead-letter exchanges, and delayed delivery (via the delayed-message-exchange plugin or TTL plus dead-lettering) are built around the broker's queue model. RabbitMQ Streams (a log-style queue type with retention and replay) narrows the replay gap for RabbitMQ users — see RabbitMQ Engineering Insights. If your problem looks like "distribute work to a pool of workers with retry semantics", RabbitMQ is often the cleaner tool.

Kafka excels at event streaming, high fanout, and replay. Retention independent of consumption enables event sourcing, backfills, and independent downstream systems reading the same topic at different speeds. Partition-based parallelism lets throughput scale with partition count and consumer parallelism, up to the limits of your cluster's hardware — not an unbounded or automatically linear curve. Not needing per-message delivery bookkeeping on the broker is a meaningful part of why Kafka can sustain high throughput, but the actual ceiling for either system depends on hardware, configuration and workload.

The honest answer

Plenty of organizations end up running both, each for what it's good at — RabbitMQ for task queues (billing runs, email sends, image processing), Kafka for the event backbone (user actions, business events, CDC from databases). That's a workload-fit decision, not a maturity signal: a company running only one of the two isn't automatically doing it wrong.

For when the RabbitMQ comparison isn't the one you actually need.

AWS Kinesis

Kinesis is AWS's managed answer to the same broad problem: a sharded, retained event stream. You trade Kafka's cluster operations for AWS-managed infrastructure, tighter IAM/service integration, and a more opinionated scaling and cost model. That usually fits best when you're already AWS-centric and want less operational surface area, but it gives you less ecosystem breadth and less portability.

Apache Pulsar

Pulsar's big design departure is a compute–storage split: brokers stay stateless while BookKeeper handles storage. That makes elasticity and native multi-tenancy stronger than Kafka's default model, especially for SaaS platforms where tenant isolation is a first-class requirement.

The trade-off is ecosystem depth and operational complexity. Kafka still tends to win on connectors, tooling, and overall maturity, so Pulsar is strongest when elastic storage or tenant isolation matters enough to justify that trade.

When to actually consider each

Kafka: ecosystem and established operating patterns matter most. Kinesis: you're AWS-first and want managed infrastructure. Pulsar: multi-tenancy or compute-storage separation is the deciding requirement.

The architectural reasoning above explains Kafka's strengths. This chapter closes the loop on its hard boundaries: where parallelism stops, what "exactly once" really means, and when Kafka is the wrong tool.

Why partitions are the scaling boundary

A partition is Kafka's unit of ordering, replica placement, and consumer-group assignment, so it is also the unit of parallelism: one partition can be actively processed by only one consumer in a group at a time. That makes partition count a hard ceiling on group throughput, but raising it has real cost — more metadata, more files, more rebalance work, and key reshuffling if you expand later.

Kafka exactly-once semantics, one layer at a time

"Exactly once" is one of the most misused phrases in Kafka discussions because it means something different at each layer. Most confusion comes from collapsing application duplicates, Kafka duplicates, Kafka transactions, and external side effects into one promise.

When not to use Kafka

Kafka is not a default for every asynchronous problem. It is usually the wrong choice for simple low-volume task queues, request/response workflows, workloads that need rich per-message routing or delayed delivery, anything pretending Kafka is a queryable database, and systems where Kafka's operational cost outweighs the replay/fanout/throughput benefits. Large blobs belong in object storage with references in Kafka, not in Kafka itself.

The mistakes you'll only hear about after the incident. Now you'll hear about them before.

Treating Kafka like a database

Kafka is optimized for sequential append + sequential scan. It has no random-lookup-by-key primitive. Log compaction gives a rough approximation — the latest value per key — but it's not a real index. If you find yourself wanting SELECT * WHERE id = ?, you want a database, not Kafka. Use Kafka to feed the database. Don't try to make Kafka be the database.

One giant topic for everything

"Just publish everything to events" seems clean until three problems compound: you can't apply different retention policies to different event types, you can't apply different ACLs, and you can't partition by different keys. A topic is a boundary of change — retention, schema, and access policy all evolve at the topic level. Split when any of those need to evolve independently.

Choosing partition count for hardware

Partitions are the unit of ordering, replication, and rebalance cost — not just throughput. "32 partitions because we have 32 cores" is meaningless. Pick partition count based on your business key cardinality and your target throughput per partition. Rebalancing 200 partitions is cheap; reshuffling data six months later because you didn't provision enough is expensive.

Skipping schema governance

"We'll just send JSON" works right up until an upstream team adds a required field and four downstream consumers explode simultaneously. The fix isn't a specific serialization format — it's owning compatibility as a first-class concern: a schema registry with Avro or Protobuf enforces this automatically and is a common choice, but disciplined JSON (explicit versioning, additive-only changes, contract tests between producer and consumers) can work too if the team actually enforces it. What matters is that schema evolution rules are enforced somewhere, not hoped for.

Shared consumer groups across teams

Two teams sharing a consumer group means their processing latencies are coupled, their rebalances affect each other, and neither can deploy without paging the other. Kafka gives you cheap consumer groups — use one per logical application, and never assume another team is safe to share yours.