Distributed Logging System Design: From a Local File to a Multi-Tenant Platform

Northstar Commerce (fictional, for this module) runs a dozen services across thousands of short-lived instances and multiple regions. A customer reports a failed checkout. Nobody yet knows which service, which instance, or which region produced the error — or whether evidence of it even still exists.

This is not primarily a storage problem, even though it looks like one. Writing a line of text somewhere is trivial. What's actually hard is making that line of text reliably survive the lifecycle of the ephemeral compute that produced it, correlating it with the other lines produced by the same logical request across service boundaries, making it searchable within a useful time budget, and doing all of this without that machinery ever being allowed to slow down or fail the request it's merely observing.

A logging system earns every piece of its complexity by solving one of these demonstrated problems — survivability, correlation, searchability, isolation from the request path, bounded retention, tenant privacy, and regional reliability — never by assumption. This module adds exactly one new mechanism per chapter, each introduced immediately after the failure that makes it necessary.

Scope this explicitly before any diagram — the scope decisions are half the design.

Functional

Collect structured and unstructured log records from every application instance, across every service and region. Make records durable before acknowledging receipt. Make them searchable by free text, structured field, service, time range, and correlation ID. Enforce tenant-scoped access on every search. Retain records on a documented, tiered schedule, with an explicit override for legal hold. Administration UIs, alerting rule engines, and full APM/tracing backends are explicitly out of scope — this module focuses on the collect → transport → index → query pipeline itself.

Non-functional

Start with the simplest thing that could possibly work: the order service writes a timestamped line to a local file for every event. An engineer greps it when something breaks.

This is not a strawman. For a single service on a single host, a local file and grep is a completely legitimate solution — fast, debuggable, and free of any distributed-systems complexity. A logging platform earns its complexity later, by solving demonstrated problems at scale, not by assumption.

The cracks appear the moment a request crosses more than one service. checkout-api calls payment-service calls inventory-service, each on a different host, each with its own local file and no identifier connecting the three fragments of one logical request. And ephemeral compute makes the fragility worse: when Kubernetes restarts a crashed pod, the old container's filesystem — and whatever log evidence it held — disappears with it.

Replace local files with one central server that every application POSTs log lines to, synchronously, and that also serves search. At low volume, this is a legitimate, shippable V1.

As usage grows, two problems appear at once. First, every application request now pays a synchronous network call to the central server just to log something — a cost paid on the hot path of the actual customer request, not a background concern. Second, that same box is also answering search queries, and a broad, unindexed search competing for the same CPU and I/O as live ingest slows both down together.

When an engineer runs a heavy search during a normal traffic period, the central server's write path degrades along with its read path — and because every application's own request blocks on that write, checkout-api's own customer-facing latency rises in lockstep with the logging backend's load. A component whose entire purpose is to help diagnose problems has become capable of causing one, directly on the customer request path.

Give the application an in-process async buffer, and move the actual network call to a separate node agent process shipping on its own schedule.

The agent's in-memory buffer is finite. When the backend is briefly unreachable, that buffer fills — and the design has to make an explicit, honest choice about what happens next.

Three real options exist once the memory buffer is full: drop the newest events, block the application until space frees up, or spool overflow to local disk. Blocking the application is never acceptable — it reintroduces exactly the synchronous coupling the previous chapter removed. The real choice is between dropping and spooling, and the right answer depends on the log's class: best-effort debug logs can tolerate drops, while error and audit-class events generally cannot.

Insert a collector gateway between agents and the rest of the platform — the natural place for authentication, payload validation, redaction, and batching.

A sales promotion pushes ingest from an illustrative 50,000 events/s average to 600,000 events/s peak — twelve times normal. Nothing downstream sized for the average day survives this burst unmodified.

If every event still had to be synchronously indexed before the collector could acknowledge it, indexing throughput would become the ceiling for the entire ingest path during a burst. The fix is to insert a durable, partitioned log bus — conceptually Kafka-like — between the collector and indexing, and to redefine what 'accepted' means: success now means durably written to the bus, not indexed and not searchable.

Partitioning by raw service name alone sends one chatty service's entire volume to a single partition — overloading it while others sit idle, regardless of how many brokers exist in the cluster.

A consumer group's parallelism for a single partition is capped at one consumer reading it at a time — adding more brokers doesn't raise that ceiling. The fix is a better partition key, chosen for what it's meant to protect: ordering scope, a specific noisy tenant, or load distribution. A genuinely dominant tenant may need a dedicated partition set and its own quota rather than sharing the default pool.

Independent indexers consume the bus as a separate consumer group, turning durable records into searchable documents on their own schedule. 'Durably accepted' and 'searchable' are now visibly different moments — typically single-digit seconds apart under healthy operation, and measurably longer during an indexing outage (Failure & Scale Scenarios covers this in depth).

Name every stage a record passes through explicitly — this is the single sentence that should anchor every durability claim this design makes.

Each arrow in this chain is a place a record can stop existing, or become stale, independently of every other stage. 'The application logged it' only means generated. 'The agent sent it' only means flushed. 'The collector accepted it' is weaker than 'the broker durably committed it,' which is weaker still than 'the indexer has processed it,' which is weaker than 'it's visible in search results right now.'

How a record becomes searchable is a genuine architectural fork, not a vendor preference — two meaningfully different philosophies exist, and neither is universally better.

A full-text inverted index tokenizes chosen fields and the message body, enabling fast, flexible free-text search and rich aggregations — at the cost of real write amplification and storage overhead paid continuously, on every event. A label-and-chunk design indexes only a small set of low-cardinality labels (service, region, level) and stores raw content in compressed, text-scanned chunks — dramatically cheaper at ingest, with a real trade-off in free-text flexibility and a sharp failure mode if a high-cardinality value (a request_id, a user_id) is mistakenly used as a label.

A query is not a direct call to storage. It passes through a frontend that authenticates, scopes, prunes, fans out under budget, and merges deterministically.

Every query authenticates the caller and resolves the authorized tenant scope from that verified identity — never from a client-supplied tenant field, which is a hint at best, never an authorization boundary. The frontend prunes the search to the requested time range before fanning out to the shards or time-sliced chunks that could possibly match, running that fan-out in parallel under an enforced scan-byte and concurrency budget so one expensive query can't starve every other operator.

Results from multiple shards merge under a deterministic sort — timestamp plus a stable tie-break — because two shards can genuinely report events with the same timestamp, and pagination has to stay stable across that merge. With a consistently propagated trace_id, a correlated incident view across services becomes possible for the first time in this story.

Not every log needs to live in the fastest, most expensive tier for its entire retention period.

An illustrative tiered policy ages recent, high-value logs through a fast hot tier, a cheaper warm tier, and finally a cold object-storage archive for long-term retention. These exact windows are policy choices, not laws of physics — a different compliance posture would choose different numbers. What matters is that the choice is explicit, that moving data between tiers is itself a monitored background process, and that querying the archive tier honestly costs more latency than the hot tier, not the same speed behind a spinner.

Every layer a log record passes through needs its own tenant-scoping check — a single missed layer can leak one tenant's data to another even if every other layer was correctly scoped.

At multi-region scale, two more constraints appear that don't exist in a single-region deployment: data residency and the question of who notices when logging itself breaks.

Every component below exists because of a specific, demonstrated failure earlier in this story. A small, single-service deployment doesn't need every piece at once.