Kafka Fundamentals

Kafka is a distributed, replicated, append-only log — designed to move billions of records per day between systems with strong ordering and durability guarantees.

That single sentence hides three ideas doing all the work. First, Kafka is distributed: the data lives on many machines at once, so a single machine failing doesn't take the system down. Second, it is replicated: every piece of data is copied to more than one machine, so a machine failing doesn't lose data. Third, it is an append-only log: writers add to the end and never modify what came before. That third property is the least glamorous but by far the most important — every other property Kafka is famous for is a consequence of it.

It helps to compare Kafka to what it isn't. A traditional message queue like RabbitMQ or ActiveMQ treats a message as a transient item: the broker delivers it, the consumer acknowledges, and the broker deletes it. Kafka treats a message as retained data on a log: it stays on disk for as long as the topic's retention or compaction policy keeps it, and any consumer can replay it from any offset still available. The message is not delivered — it is made available, and consumers pull what they need at their own pace. Consumption and retention are decoupled: a consumer reading a record does not remove it, and retention eventually does — Kafka is not a permanent, forever-growing log by default.

This single design choice — the log is the truth — is what makes Kafka usable simultaneously as a message queue, a stream-processing substrate, an event-sourcing store, and a change-data-capture backbone. It's not that Kafka happens to be good at four things; it's that all four are the same thing when your underlying primitive is a durable, ordered log.

The defining characteristics

Before we look at the pieces, it's worth being explicit about the properties Kafka guarantees. Everything in the rest of this chapter book is written to preserve them.

Kafka wasn't designed in an ivory tower. It was built at LinkedIn in 2010 because every existing option had already failed them at scale.

The pipeline explosion problem

Modern systems often start with direct point-to-point integrations: service A sends to B, B forwards to C, C aggregates for D. That works briefly. Then every new producer or consumer adds another brittle connection to build, monitor and evolve.

With N producers and M consumers, you can end up with N × M integrations. LinkedIn hit that wall early: too many services, too many downstream systems, and too much coordination every time schemas or consumers changed.

Kafka's founding idea was simple: one bus, many independent subscribers. Every producer publishes to the bus once. Every consumer subscribes to what it needs. The N × M integration problem becomes N + M.

The speed-mismatch problem

Downstream systems also consume at different speeds. A search indexer may be near real time while a warehouse load trails by hours. Direct pipelines couple them. Kafka's answer is retention: producers write once, and each consumer reads the same log at its own pace.

The freshness problem

Kafka also collapsed the old trade-off between durable batch pipelines and fast but forgetful real-time queues. The same log can feed an overnight warehouse load and a low-latency streaming application. Batch and real time become different ways of reading the same data.

“The log is the simplest possible storage abstraction. It is an append-only, totally-ordered sequence of records ordered by time. It looks like this: [record][record][record]. That's it.”

— Jay Kreps, one of Kafka's original authors

Kafka internals come down to four mechanisms that the rest of this guide unpacks. Each partition is an append-only log that brokers write to sequentially. Each partition has one leader that serves reads and writes, while followers replicate it and form the in-sync replica set (ISR). Consumers pull records and track their own position with offsets, so reading never removes data. A controller (a KRaft quorum in current Kafka) tracks which brokers are alive and which broker leads each partition.

The Kafka architecture section below walks through the brokers, controller, and clients; the partition log and replication chapters go deeper into the log and the ISR.

Kafka's architecture is deliberately boring. That is its strength.

A Kafka cluster is a group of processes called brokers that collectively store the log data for every topic. Producers and consumers are client libraries running inside your applications — they never share Kafka's process boundaries. There are no gateways, no proxies, no middlemen. A client library opens a TCP connection to a broker, speaks Kafka's binary protocol, and moves on. The Kafka architecture diagram below shows the three moving parts: producers, brokers, and consumers.

Brokers

Each broker is one JVM process running on one host. It stores copies of some partitions, accepts producer writes, serves consumer fetches, and replicates with its peers. Kafka is fast partly because a broker does only those few things; it is not trying to be a general-purpose database.

The controller

Cluster metadata — which brokers are alive, which partitions live where, which broker currently leads each partition — is managed by a controller. Current Kafka manages this metadata through KRaft, a Raft-based controller quorum running across a small set of brokers rather than a single elected broker; don't picture "one broker is the controller" as the general architecture. When a broker dies, the controller quorum notices, picks a new leader from the surviving replicas, and propagates the new metadata to everyone. Older Kafka releases used an external ZooKeeper ensemble for this role; ZooKeeper support was removed in Kafka 4.0, so that mode is historical in current Apache Kafka and is called out here only so you recognize it in older deployments or documentation.

Producers and consumers

Producers are library code that lives in your application. When you call producer.send(record), the library serializes the record, chooses a partition, batches records per partition, and sends them to that partition's leader broker. Batching and compression are a big part of Kafka's throughput story because they amortize network overhead across many records.

Consumers are also library code. They coordinate via Kafka's group protocol to divide partitions among themselves, poll their assigned partitions for new records, and commit progress back to Kafka. Everything about how fast, how much, and in what order a consumer reads is under the consumer's control — Kafka never pushes.

The optional ecosystem

Around the core cluster sits an ecosystem you'll often meet in production. Kafka works without these pieces, but they solve common operational problems.

The core vocabulary below covers most Kafka design conversations. Learn it precisely once and you'll never be lost in a design review.

If you remember only one diagram, make it this one. Every other property of Kafka is a consequence of what happens in a partition log.

A partition is not a queue, and not literally one file either. It is an ordered append-oriented log made of segment files plus indexes. Kafka writes to the active segment, closes older ones behind it, and lets retention or compaction clean whole segments over time.

Anatomy of a record

Each record has a small amount of metadata plus the payload your application cares about:

What happens to ordering when you add partitions

A keyed producer relies on a stable key → partition mapping to preserve per-key order. That mapping depends on the current partition count. If a topic grows from 6 partitions to 12, a common hash-based partitioner may start routing customer-42 to a different partition. Old events stay in the old partition; new ones go to the new partition. Nothing is lost, but reconstructing a single ordered history for that key now requires reading and merging across both partitions.

Why the log format matters

The log format is a major reason Kafka is fast. Appends happen at the tail, reads usually scan forward from an offset, and records are not modified in place. That removes the read-vs-update contention you get in mutable storage, even though Kafka still has to coordinate replication, group membership and transactional visibility.

The log format explains why Kafka is fast. Replication explains why it doesn't lose your data.

Every partition has a replica set — a fixed list of brokers that hold a copy. One of those replicas is the leader; the rest are followers. The leader accepts producer writes; the followers pull records from the leader and append them to their local logs, in the same order, at the same offsets.

What in-sync actually means

A follower is in-sync when Kafka's replication protocol considers it caught up with the leader — concretely, when it has not failed to fetch and apply the leader's latest records for longer than replica.lag.time.max.ms. If a follower falls behind that window — because its host is slow, its disk is saturated, its network is congested, or its JVM is in a long GC pause — Kafka removes it from the ISR until it catches up again. The leader is always counted as part of its own ISR.

The ISR is not a fixed list. It shrinks and grows dynamically as followers get slower or faster. This adaptive membership is what makes Kafka's durability guarantees so cheap — a temporarily slow broker doesn't block the whole cluster, it just doesn't count for durability decisions until it's healthy again.

The acks / min.insync.replicas contract

The producer's acks setting decides when a write is considered successful:

The broker side of the contract is min.insync.replicas. This tells the broker: "if the ISR is smaller than this number, reject `acks=all` writes". Setting it to 2 with a replication factor of 3 gives you a well-known trade-off: you tolerate 1 broker failure without either data loss (because 2 replicas still have the record) or availability loss (because the write still succeeds). If a second broker fails, writes start failing loudly — which is the correct behaviour for a system that would otherwise be one failure away from data loss.

Log end offset, high watermark and last stable offset

Three offsets matter once replication is involved, and conflating them is a common source of confusion. The log end offset (LEO) is the next offset a replica will assign — effectively "how far this replica's log has grown", tracked independently per replica. The high watermark (HW) is the leader's replicated/committed visibility boundary: the point up to which the leader knows the current ISR has durably replicated, and therefore the point up to which consumers are allowed to read. A record can sit past the HW on the leader's disk — written, but not yet safely replicated — without being visible to consumers yet.

Thinking of the high watermark as roughly "the minimum replicated progress across the ISR" is a useful mental model, but Kafka has to track it more carefully when ISR membership changes or leaders change. For transactional producers, there is also the last stable offset (LSO), which holds back records from uncommitted or aborted transactions for read_committed consumers.

Producer idempotence and Kafka transactions

Idempotent producer (enable.idempotence=true) solves one specific problem: if a producer retries a send because it didn't receive an acknowledgement, the broker recognizes the retry (via a producer id and a per-partition sequence number) and does not append a second copy. That is exactly-once for that specific retry scenario at the log-append level — it does not know whether your application already submitted the same logical command twice through a different path, and it does not make a database write or an HTTP call exactly-once.

Kafka transactions go further: they can atomically group multiple partition writes together with consumer offsets for read-process-write workflows that stay inside Kafka. A read_committed consumer will not see data from a transaction that later aborts. What transactions do not do is make external systems — a database update, payment call or email send — part of the same atomic unit automatically.

A single Kafka topic can be consumed by one script or by a thousand-node fleet. The mechanism that makes both possible is the consumer group.

A consumer group is a set of consumers that agree to share the work of reading a topic. Inside a group, Kafka guarantees that each partition is owned by exactly one member. This is the key invariant — the one that preserves ordering under parallel consumption. Within a stable assignment, two consumers in the same group are never assigned overlapping partitions, so they do not process the same record concurrently (after a rebalance, or a crash before an offset commit, a record can be redelivered to a different member).

How assignment works

When a consumer joins a group, it announces itself to the group's coordinator — a specific broker chosen by hashing the group id. The coordinator collects the current members, runs the configured partition assignor, and hands each member its list of assigned partitions. Membership or subscription changes — a consumer joining, leaving, crashing, missing its session timeout, or a topic's partition count changing — cause the coordinator to reconcile ownership, commonly called a rebalance. Exactly how disruptive that is depends on the assignor and protocol in use: eager-style reassignment can briefly pause the whole group, while incremental/cooperative assignment limits the pause to only the partitions actually changing hands. Verify the current default and behavior against your Kafka client version rather than assuming one assignor is permanently "the modern one".

The upper bound on parallelism inside a group is the partition count. If a topic has 6 partitions and a group has 10 consumers, only 6 are assigned a partition — the other 4 sit idle with nothing assigned. They are not instant hot standbys: if one of the 6 crashes, Kafka first has to detect the failure (missed heartbeats or a session timeout), then trigger a rebalance to reassign that partition to one of the idle members — a real, if usually short, interruption, not an instantaneous failover. Adding more consumers than partitions doesn't add throughput; it adds idle capacity that only helps once a rebalance has completed.

Consumer offsets: what a commit actually promises

A committed consumer offset conventionally represents the next offset this consumer group will read for that partition — not a generic "the last record we successfully finished processing". That distinction drives both ways a consumer can misbehave around a crash.

Process, then crash before commit: the consumer fetches offset 100, processes it (maybe writing to a database or calling an API), and crashes before the corresponding commit becomes durable. A replacement consumer resumes from the last committed offset, which is still behind 100 — so record 100 is fetched and processed again. This is the at-least-once shape, and it is why idempotent processing matters more than "did Kafka lose anything" when debugging duplicates.

Commit, then crash before the side effect finishes: if offset progress is committed before — or concurrently with, and faster than — the actual processing, a crash between the commit and the side effect means the record is skipped from the application's perspective on restart, even though Kafka itself did nothing wrong. This is the opposite risk, and it's why when you commit relative to when you finish processing is a correctness decision, not a performance one (see Production Engineering → Consumer Processing Patterns).

Groups don't share, groups compose

Two different groups subscribed to the same topic don't split the work — each group sees the full stream independently. This is what turns a Kafka topic into a broadcast primitive. One producer publishes orders; a warehouse group consumes it to trigger fulfillment, a fraud group consumes it to score transactions, an analytics group consumes it for the data warehouse. None of them coordinate. None of them slow each other down. This is the entire foundation of event-driven architecture.

You've met every piece. Let's watch one message travel through all of them.

This section is the mental model to keep in your head whenever you're debugging Kafka behaviour. It's the answer to "what actually happens when I call producer.send()?" — end to end, from send() to commit().

The producer side

The application calls producer.send(record). The record is serialized to bytes using the configured key/value serializers. The producer's partitioner looks at the record's key (or, if no key, uses sticky/round-robin routing) and picks a partition — say partition 3. The record is added to a batch for that (topic, partition). If the batch is now full (batch.size reached) or the linger timer has fired (linger.ms elapsed since the batch started filling), the batch is compressed and sent to the leader broker for partition 3.

The broker side

The leader appends the batch to its partition-3 log at the next set of offsets. It also updates its log-end-offset for the partition. Concurrently, all followers for partition 3 have long-running fetch requests open — they now receive the new records and append them to their local logs, updating their own log-end-offsets. Once the ISR has replicated the batch (their LEOs are ≥ the batch's last offset), the leader advances the high-watermark — the offset up to which records are considered committed and visible to consumers.

With acks=all, only now does the leader acknowledge the producer. The producer receives the ack, marks the batch as complete, and moves on to the next batch. If the ack is lost in transit, the producer retries — and if idempotence is enabled, the broker deduplicates the retry using the producer's PID and sequence number.

The consumer side

Meanwhile, consumers in various groups have long-running fetch requests open for the partitions they're assigned. When the high-watermark advances, the broker responds to their pending fetches with the new records, delivered via zero-copy — bytes flow directly from the broker's page cache to the consumer's socket without any user-space copy. Each consumer processes the records in its application code and then commits progress (conventionally, the next offset to read) back to Kafka. On restart, the consumer resumes from that committed offset. Whether that means "exactly the records you'd expect" depends entirely on the order of processing and commit — process-then-commit can replay a record after a crash, commit-then-process can skip one; see Consumer Offsets above for both directions.