Distributed Logging System Design Interview Guide

Because the backend's slowness and outages would become the application's own. If every request makes a network call to log, a heavy search on the log server raises checkout latency in lockstep, and when the server is down requests either fail or hang until a timeout. That inverts the purpose of logging, which is to help you diagnose problems, not cause them. The fix is to make logging a local, non-blocking operation: the application appends to a bounded in-process queue, and a background thread writes to stdout, a file, or a local socket that an agent ships onward. The cost is a small loss window (lines still in the queue when the process dies) and one more component to run on every host. For a small internal tool with no latency target, a synchronous call to a hosted logging endpoint with a short timeout is a reasonable first version; add the agent when the backend's behavior starts to show up in request latency.

• What's the actual fix? — An in-process queue drained by a background thread to a file, stdout or a local socket, plus an agent that ships independently. The request returns as soon as the app finishes its own work.

It has to drop or spill, and it must not block. Blocking the application on a full buffer brings back the exact coupling you just removed, so the real question is what to give up. Error and audit records go to a bounded local disk spool; once the spool is also full, the lowest-value records, usually debug, are dropped by an explicit policy, and every drop is counted in a metric so the loss is visible rather than silent. Size the spool from arithmetic, not hope: pick an outage window to absorb, say 30 minutes, multiply by the node's average write rate, and add headroom. An unbounded spool is only a delayed outage, because the host eventually runs out of disk and takes the application with it. Records that must never be dropped, such as an audit trail, should not depend on best-effort log shipping at all; write them to a transactional store or outbox owned by the application, and let the logging pipeline carry a copy.

• Why not just block the app until there's room? — Blocking reintroduces exactly the synchronous coupling application isolation exists to remove — it's never an acceptable option on the request path.

Identify the file by what it is rather than where it lives, and remember how far you have safely got. Concretely: a file identity (device and inode, plus a fingerprint of the first bytes, because inode numbers get reused), the byte offset up to which downstream has acknowledged the data, and never an offset that points into the middle of a line. With rename-based rotation, logrotate moves checkout.log to checkout.log.1 and creates a new file; the agent keeps its open handle, reads the old file to the end, then starts the new checkout.log at zero. With copytruncate the file keeps its identity but shrinks, so a size smaller than the saved offset is the cue to restart at zero, accepting that bytes written between the last read and the truncation are gone, which is the main reason to prefer rename-based rotation. A crash re-reads from the last acknowledged offset, so duplicates are possible and are removed downstream with a record ID derived from file and offset. If you control the application, pushing structured events over a local socket avoids rotation altogether, at the cost of an SDK and a plan for when the socket is unavailable.

It depends on which claim you mean, and that is the point of the question. For this pipeline, durable means the record is committed on the replicated bus: the partition leader and every in-sync replica have it, and at least min.insync.replicas of them (two of three, in the reference setup) did before the acknowledgment went back. Before that point the record lives in application memory, an agent's spool on one disk, or a collector's memory, and each of those loses it under a different failure. So every acknowledgment in the pipeline should say which stage it covers. An ack from the collector's memory queue (option A) is cheap but can silently lose a record if the collector dies before publishing; an ack after the broker commit (option B) closes that gap. Even option B is not 'forever': it survives one broker failing, and survives a zone failing only if replicas are spread across zones. A team that truly needs minimum ack latency and accepts the loss window can choose A, as long as the choice is written down as a trade-off and not hidden behind the word durable.

• Why is option B worth the extra latency? — Because logging acks happen off the customer request path — there's no user-facing latency being protected at this stage, so the latency cost is nearly free to absorb.

At the first component you control that sees the data, before anything is written durably. In this design that is the collector, ahead of the bus publish. The order matters because encrypting a store protects a secret from someone who steals the disk, but it does not unsay that the secret was written, replicated to three brokers, and possibly read by an indexer. Redaction removes it before it multiplies. Pattern rules are fallible, so decide in advance what happens when a rule cannot judge a field: for sensitive classes, hold back the field or the whole record and raise an alert instead of forwarding it. If even the collector must never see raw values, for example card data under a strict compliance scope, redaction has to move into the application's logging library, which is a stronger guarantee but depends on every service adopting it correctly, so it is usually paired with collector rules as a second layer.

• What should happen if the redaction rule isn't sure a field is sensitive? — Fail closed — hold back the field or the whole record rather than guess it's safe and forward it. A delayed log line is a far smaller cost than a leaked secret.

You checkpoint once per acknowledged batch, not once per line, and the batch boundary comes from the handoff rather than from a timer. Persisting a checkpoint safely means writing a temp file, fsyncing it and renaming it, which takes milliseconds; doing that per line would cap a file's throughput at a few hundred lines a second. Instead the agent hands a batch of lines to the next durable stage (its fsynced spool, or the collector), waits for the acknowledgment, and only then moves the checkpoint to the end of that batch. If it crashes in between, the checkpoint is still behind, so the batch is read and sent again: duplicates, never gaps. Batch size, for example 1,000 lines or 200 milliseconds, sets the most that can be replayed after a crash. What you must not do is advance the checkpoint when lines were merely appended to an in-memory buffer; a crash then leaves the checkpoint ahead of data that was never persisted, and those lines vanish silently. If even a few hundred duplicated lines are unacceptable you can shrink batches and pay in throughput, but deduplicating by record ID is usually the cheaper answer.

Only when bursts and failures make direct indexing the bottleneck, so walk through the numbers. Normal load is 50,000 events a second (about 40 MB/s); a 12x promotion peak is 600,000 a second (about 480 MB/s). Without a bus, the indexing tier has to be provisioned for the peak, or the collector blocks on slow indexing and pushes the pressure back into the agents. With a bus, the collector acknowledges once the record is appended durably, and indexers consume at whatever rate they have: lag grows during the peak and shrinks afterward. The bus also lets you replay into a rebuilt index and attach other consumers, such as archiving, to the same stream. What it costs is a stateful cluster with its own problems: partition skew, disk pressure, consumer lag. A small system with steady traffic and an autoscaling managed search service may be better off having collectors write batches straight to the index and accepting simple backpressure.

• What's the actual cost of adding the bus? — A new, stateful, operationally complex component with its own failure modes — partition hot-spotting, disk pressure, consumer lag — not a free win.

Start from what the key has to protect, then use the cheapest key that does it. Ordering is only guaranteed within a partition, and within a consumer group one consumer reads a partition at a time, so a key that is too coarse limits both. Service name alone puts all of checkout-api on one partition: in the burst scenario it sits at 95% load while payment-service and inventory-service run at 15 to 20%, and adding brokers changes nothing. A composite of service plus a hashed shard, or plus instance ID, spreads one service across many partitions. The price is losing strict ordering across that service's instances, which logs rarely need, and instance ID in the key keeps order per instance. A tenant that dominates traffic can be given dedicated partitions and a quota so it cannot crowd out others. Choose the partition count with headroom, because adding partitions later changes which partition a key maps to.

It depends on the order of the two steps, and the useful answer shows both. If the indexer writes the document and then commits the offset, a crash between the two means it restarts at the old offset and indexes the record again. If the write is an upsert keyed on a stable record ID, the second write overwrites the first with identical content and nobody sees a duplicate. If it commits first and then writes, a crash in between loses the record for good: the restart begins after it and nothing will ever replay it. So index first, commit second, and make the write idempotent; that is at-least-once delivery with an effectively-once result. Commit-first is only defensible when duplicates are truly unacceptable and the index cannot dedupe, and then you need an independent reconciliation that compares produced and indexed counts per time window to catch the silent gaps.

• How do you even detect a commit-then-index skip if it's silent? — Only via an independent count-reconciliation process comparing records produced to records indexed over a time window — never via the indexer's own metrics, which look perfectly healthy.

It goes to a dead-letter queue and the partition keeps moving. The alternatives are worse: if the indexer crashes on the record, it restarts, reads the same record and crashes again, and everything behind it is blocked. So the indexer catches the parse or schema failure, writes the raw record plus the reason to a quarantine topic, raises an alert when the quarantine grows, and commits past it. A growing quarantine is an incident signal by itself, because some producer is emitting garbage, and keeping the raw bytes means you can fix the parser and replay them. Skipping is not safe when strict per-key ordering matters, as in a state-change stream; there you would pause only that key's stream, which needs more machinery. For logs, quarantine and continue is the right default.

RF=3, acks=all, min.insync.replicas=2, unclean leader election disabled, and replicas spread across three failure zones. That means a write is acknowledged only when at least two replicas, including the leader, have it, so any single broker failure loses nothing and does not stop writes. If a second broker goes, the partition can no longer meet the minimum, producers get NotEnoughReplicas, and you lose write availability rather than quietly lowering durability. Unclean election stays off because promoting an out-of-sync replica would discard acknowledged records. The tempting claim that RF=3 survives losing two brokers holds only if the surviving replica was in sync when the data was acknowledged, and min.insync.replicas=2 does not promise that. The price of all this is write latency set by the slowest in-sync follower, which is acceptable because logging acks are off the request path. A debug-only stream nobody expects to be durable can use RF=2 or acks=1 to save cost.

Mostly it will not help, and it can make things worse, for two separate reasons. First, a consumer group cannot use more consumers than partitions: with 12 partitions and 8 consumers, five more makes 13, and the thirteenth sits idle, so check the partition count before scaling. Second, every membership change triggers a rebalance, and with the classic eager protocol all consumers stop consuming while partitions are reassigned; five consumers joining one after another can cause several back-to-back rebalances, so lag rises just when you wanted it to fall. Mitigations are the cooperative incremental rebalance protocol, static membership so a restarting pod keeps its partitions, scaling in measured steps with a cooldown, and scaling on the trend of lag rather than a momentary spike. If a short pause is harmless, one larger rebalance reaches full capacity sooner than staggering the joins.

The decision is about how people search, not about which product is better. An inverted-index engine such as Elasticsearch or OpenSearch indexes the tokens in your fields, so free-text search and ad hoc aggregation are fast; you pay on every ingested event, and the index can be as large as, or larger than, the compressed raw logs. A label-and-chunk engine such as Loki indexes only low-cardinality labels and stores compressed chunks that are scanned at query time; ingest is far cheaper, but broad searches over unscoped data read a lot of chunks. This design uses the first for the seven-day hot tier, where incident search needs flexibility, and the cheaper label-and-chunk model for older data, where queries are already scoped by service and time. A team that never does free-text search across everything can run label-and-chunk all the way down, and a team whose investigations are unpredictable and whose budget is generous can keep the inverted index longer. A wrong choice shows up as a cost line nobody can justify or an on-call engineer waiting minutes for a search.

• What breaks if you pick wrong? — Picking an inverted index for a huge-volume, rarely-searched tier wastes enormous cost on indexing nobody queries; picking label-chunk for a tier where investigators need broad free-text search makes incident response slower and more frustrating.

A label defines a stream, and the index keeps an entry per unique combination of labels. Service, region and level give you a few dozen streams. Add request_id and every request becomes its own stream holding a handful of lines: millions of tiny streams, each with its own index entry and an under-filled chunk, so index size, memory use and chunk counts explode and queries slow down. The fix is not tuning; the identifier has to live in the log line or in structured metadata and be filtered at query time, after labels have narrowed the search. The same trap exists in an inverted-index engine as a dynamic mapping that creates a new field for every unpredictable key, such as user IDs used as JSON keys. The rule of thumb is that labels and field names should be things you can enumerate; values you cannot enumerate belong in values.

By making the query planner say no before any work starts. Each query gets a maximum time range, a scan budget in bytes, a cap on concurrent shard requests and a timeout, all derived from the caller's tenant policy, and a query that would exceed them is rejected or narrowed at plan time instead of being killed after it has already saturated the workers. Enforcing limits afterwards is too late, because the cost has been paid. Beyond the per-query limits, give interactive searches and heavy analytical queries separate pools or queues, and give each tenant a fair share. The strongest isolation is running query and ingest on separate resources, so a runaway query degrades search but can never touch the write path that agents depend on.

Put the same trace_id in every log line at every hop, and query by it. The first service creates the ID, or accepts the one from an incoming header such as W3C traceparent, passes it to downstream calls, and the logging library adds it to each record automatically. Searching for that ID then pulls the checkout-api, payment-service and inventory-service lines into one timeline. What you get is grouping, not causality: the lines are ordered by timestamps from different hosts, so tight interleaving is approximate. A tracing system records spans with parent and child links and durations, which is what you need to see where a three-second timeout actually came from. So logs with trace IDs are a cheap and very useful first step that complements tracing; when you need span timing and critical-path analysis, add a tracing backend instead of stretching the logs.

Sort by timestamp and then by a field that is unique and stable, so the pair forms a total order. Without a tie-break, two shards returning the same millisecond can interleave differently on each request, and page two repeats or skips rows from page one. A record ID, or a shard ID plus a per-shard sequence number, works well. The cursor is the last (timestamp, tie-break) pair, so the next page asks for everything after it, rather than skipping N rows, which also keeps deep pages cheap. Finer timestamp resolution reduces collisions but never removes them, and the hosts' clocks are not synchronized to that precision anyway, so do not rely on it.

Because the value of a log line decays quickly while its storage cost does not. Most investigations look at the last few hours or days, so keeping 90 days on fast, fully indexed storage pays hot-tier prices for data that is almost never read. Tiering matches cost to access: seven days hot and fully indexed, then 30 days warm on cheaper or label-scoped storage, then out to 90 days in object storage that is cheap to keep and slow to query. Those windows are policy, not physics, so set them from query statistics and compliance needs. The costs are slower, sometimes asynchronous, queries on older data and a migration process that can stall and silently leave data on expensive storage, so alert on the age of the oldest data in each tier. If retention is short or volume is small, a single tier is simpler and is the right answer.

Never let the request tell you who the tenant is. Authenticate the caller, derive the tenant scope from the credential (token claims or the mTLS identity), and inject that scope as a mandatory filter on every query on the server side. A request that names a different tenant is rejected with a 403 before any data is read. The same rule applies on the way in: the collector stamps tenant_id from the authenticated connection and overwrites anything in the payload. Then add depth, because any single layer can have a bug: a tenant-scoped namespace on the bus, tenant-scoped indexes or enforced filters in the store, and an audit record of the resolved scope for every query. For stronger isolation than a filter, very sensitive tenants can get their own indexes or clusters, which costs more and is a product decision, not a substitute for the checks. I do not see a legitimate design in which the client chooses its own tenant.

• What if the mistake is in the database query scope, not the API layer? — That's exactly why every layer — collector attribution, bus namespace, index scoping, and query authorization — needs its own independent tenant check; a single missed layer can leak data even if every other layer was correctly scoped.

Take it stage by stage. Applications keep appending to their local queues, and agents read and hand off as fast as they can; if collectors are slower, the agents' spools absorb the excess instead of the request path. Collectors are stateless, so you add replicas; they authenticate, redact, and publish, acknowledging only after the broker commit. The bus takes the full 600,000 events a second (about 480 MB/s), because appends are sequential writes, provided the partition key spreads load so that no single partition is hot. The indexers are the slow part: sized for the 40 MB/s average, they fall behind, so consumer lag and the age of the oldest unprocessed message climb. Search gets stale but nothing is lost, as long as bus retention outlasts the lag. After the peak, indexers need headroom above ingress to drain; with 80 MB/s of indexing against 40 MB/s of ingress, the net drain is 40 MB/s. So what you size for the burst is bus retention and disk plus indexing headroom, not 12x everywhere. If bursts are rare and an hour of search delay is acceptable, you can provision less.

Not automatically, and for residency-bound tenants never. The design runs an independent pipeline per region and keeps each region's data in that region. If eu-west fails, the agents there spool locally until it returns. When the outage outlasts the spool, some records are dropped by policy and that is reported, which is a reduction in availability we accept instead of moving EU data into us-east without consent. A federated query layer can read across regions only where policy allows, as an authenticated and audited query, never as background replication of raw logs. A tenant without residency constraints can opt into a secondary region for ingestion, but that should be an explicit per-tenant choice with its costs and consistency caveats spelled out, not a global default.

Send the pipeline's health signals through a path that does not depend on the pipeline. Agents, collectors, brokers, indexers and the query tier export a small set of metrics: records dropped, spool fill, age of the oldest unprocessed message, consumer lag, indexing delay, and query latency by class. These go to a separate metrics system with its own alerting, ideally with a synthetic canary log line sent end to end every minute and checked on arrival, so that silence is itself an alert. If the platform's own logs were the only way to learn it is broken, an outage would be invisible exactly when you need to see it. Also rate-limit the platform's own logging, since a retry loop can produce a flood of error lines that overloads the pipeline. A small single-team setup can get by with a simple external check; once several teams depend on the platform, independence is a requirement.

The race exists because the hold is applied by a process that can be slower than the deletion job, so close it structurally. Make the deleter consult the hold registry at the moment of deletion, not rely on an earlier scan, so a hold placed before the delete runs always wins. For categories that may plausibly be held, delay physical deletion by a grace period after the TTL. And keep a longer-lived archive copy for those categories, so a hold that arrives after the hot-tier copy expired can still be honoured against the archive. If no copy survives anywhere, tell counsel plainly what was deleted and when, under which routine policy; the deletion log is what shows it was routine. A non-regulated debug stream can simply let the TTL win.

• What if no surviving copy exists anywhere? — That's an unrecoverable gap, and it has to be reported to legal counsel honestly as exactly that — not hidden or glossed over.

Compare two numbers: how long the bus keeps data, and how long the backlog takes to clear. Applications and agents kept publishing during the outage, so the records exist on the bus; the indexers are just behind. At 40 MB/s average ingress, 30 minutes is 40 x 1,800 seconds = 72,000 MB, or 72 GB. If the indexers come back with 80 MB/s of capacity, they clear 80 minus 40 = 40 MB/s net, so 72 GB takes 1,800 seconds, another 30 minutes. As long as retention is longer than the outage plus the drain time (an hour here, with margin) nothing is lost, and searches in that period are stale by up to the age of the backlog, which the UI should say. If you cannot afford that retention, you either need indexers that scale faster, shortening the drain, or you accept that old records may age out of the bus before being indexed, which should be a named, alerted condition. Also check broker disk, because 72 GB of backlog sits on top of normal retention.

Nothing is lost if the agents' spools are large enough, because the collector never acknowledges anything the bus has not committed. While the bus is down, collectors fail the publish, agents get no acknowledgment, and they keep the unacknowledged batches in their spools and retry with backoff. The collectors stay stateless; the data waits at the edge. The arithmetic is per agent: spool capacity against that node's ingress over 20 minutes. If a spool fills, the overload policy takes over, with debug dropped first, error and audit records protected as long as possible, and every drop counted and reported. When the bus returns, thousands of agents resend at once, so use jittered backoff and collector rate limits, or the recovery itself overloads the bus. If you need lossless behavior at any duration, the application has to block or reject work, a trade almost no logging platform makes.

No. There is order within a partition, and that is all the bus gives you. Across partitions, services and pods, records can be indexed in any order, and arrival order says little about when things happened. Timestamps help but are not exact, because each comes from a different host's clock: two events a few milliseconds apart on different hosts can appear reversed, and a skewed clock can reorder them by seconds. A trace_id tells you which lines belong to one request but does not order them. For causal order inside a request, use parent and child relationships between spans from tracing, or a sequence number carried in the request; logical clocks such as Lamport or vector clocks are the formal tool but require every service to propagate them. For routine application logs the pragmatic position is to sort by event time within a host, correlate by trace_id, and expect an uncertainty window about as large as your clock synchronization error.

Collect from the node, not from the pod. The application writes to stdout and stderr, the container runtime stores that output as files on the node, and a DaemonSet agent tails those files and attaches pod metadata such as namespace and labels. Because the agent is not part of the pod, a pod that crashes or is evicted still leaves its last lines in the node's log files for the agent to read. Two limits matter. The kubelet rotates container logs and removes them after the pod is deleted, so an agent that is behind or restarting can lose the tail; it has to keep up and keep its checkpoints. On the application side, flush on SIGTERM and set a termination grace period long enough that the final lines are written before the container exits. A sidecar per pod is the alternative when a workload needs an isolated pipeline, custom parsing or its own file volume, at the cost of memory and CPU in every pod.

Do not measure it with the producers' timestamps. Subtracting wall-clock times across hosts gives the true delay plus the difference between the two clocks' errors. A producer whose clock is 4 seconds fast stamps events 4 seconds late, so they look 4 seconds fresher than they are, possibly even negative lag; a clock 4 seconds slow makes them look 4 seconds more delayed. Measure each stage where the clock can be trusted. Between the collector, the broker and the indexer, use timestamps from platform-managed servers with monitored time sync, for example t_indexed minus t_durable. For time spent inside one process, such as a batch waiting in the agent's spool, use a monotonic clock, which does not jump when NTP adjusts the wall clock and needs no synchronization, though it is only valid within one process and one boot. On the bus, consumer lag in records and the age of the oldest unprocessed message are direct freshness measures. Keep event_time for users, but do not treat it as a latency measurement. If all hosts are tightly synchronized, with offsets bounded to milliseconds, a coarse now minus event_time is acceptable for dashboards.

Because accepted and searchable are different moments. The bus commit is the durable moment; the document becomes visible only after an indexer has read the record, written it, and the index has refreshed, which normally takes seconds and takes much longer when indexing lags. A query run in that gap finds nothing, and from a single query you cannot tell late from lost. So before concluding that data is lost, check the indexing lag, the age of the oldest unprocessed bus message, and the agents' drop counters; if the lag is longer than the time since the record was produced, wait and re-run the query. If lag is normal and the record is still absent, investigate real loss: dropped counters, the dead-letter queue, the tenant scope, or the time range of the query. A system can make writes wait for index visibility, which makes search-after-write exact at a large cost in ingest latency and throughput; logging platforms generally do not pay it.

• How would you actually debug this live, during an incident? — Look at current indexing lag and the age of the oldest unprocessed bus message first. If lag is within normal bounds, wait a moment and re-query; if it is elevated, that is an indexing backlog, which is a separate and actionable problem.

Make it structurally impossible to sample audit logs, rather than a matter of discipline. Classify records where they are created, with the level or log class in the schema, and apply sampling only to the debug class, in a stage that reads that field and passes everything else through untouched. Audit, security and error records never enter the sampler. Back that with a test that fails if an audit record is ever dropped, and a metric that reports sampled-out counts per class so a mistake is visible. Where sampling is applied, make it deterministic by trace_id, so you keep all the lines of the sampled requests instead of random fragments. If a team has no compliance data at all, uniform sampling is simpler; the moment any class carries legal weight, this separation is required.

Treat it as a different product, not a slower version of live search. Archive queries read from object storage, which is cheap to keep and expensive and slow to scan, so they need a mandatory narrow time range and label filters, an estimate of bytes scanned shown to the user before the query runs, and an asynchronous job model: submit, receive a job ID, get results or a notification when ready. Prune with a partitioned layout (date, tenant, service) and columnar formats with min and max statistics, so a scan reads only the relevant files. Apply per-tenant quotas so one request cannot drain the budget. If users routinely need year-old data interactively, the tiering was wrong: keep that data in a warmer tier, or pre-aggregate what they actually ask for, such as daily error counts per service, into a small fast store.