System Design · 9 chapters · 23 decisions · ~38 min read
Distributed Logging Design Trade-offs & Decisions
Twenty-three concrete decisions, each with a real failure or cost that motivates it, the realistic alternatives, and why one was chosen for this design — including the two decisions worth defending in the most depth: whether a durable bus like Kafka is needed at all, and inverted-index versus label-chunk storage.
Decisions 1-2
Getting Logs Off the Host
Before any storage decision, logs have to leave the producing process safely — these two decisions shape everything downstream.
Decision 1 — Synchronous write vs. async local buffer + agent
| Option | Pros | Cons |
|---|---|---|
| Synchronous write to a remote backend | Simple mental model; no separate agent process to operate | Backend latency or outage directly adds latency or failure to the request path |
| Async in-process buffer + separate agent | Application never blocks on logging; agent failures are isolated | Added component to deploy and monitor per host; genuine data-loss window if the agent itself crashes before flushing |
Decision 2 — Push from the app vs. pull/scrape by the agent
| Option | Pros | Cons |
|---|---|---|
| App pushes to agent (local socket/pipe) | Lower latency to agent; works for structured, non-file-based emission | Requires an SDK or sidecar protocol integrated into every app |
| Agent tails local files | No app-side integration needed; works for any process that writes a file | Rotation handling complexity; small latency floor from poll interval |
Decisions 3-4
Buffering and Durability Posture
Decision 3 — Drop vs. block vs. spool on backpressure
Swipe to pan
This is a direct application of the general backpressure pattern to the agent-to-collector hop specifically — the same high-water-mark reasoning, applied to log buffers instead of a generic consumer queue.
| Option | Pros | Cons |
|---|---|---|
| Drop newest events | Zero added latency or resource use; simplest to implement | Silent, policy-driven data loss — must be explicitly measured and disclosed |
| Block the application | No data loss at the agent layer | Violates application isolation — never acceptable on the request path |
| Spool to bounded local disk | Survives brief outages and agent restarts without loss | Disk I/O cost; still drops once the spool itself is full |
Decision 4 — Ack-at-memory-queue (option A) vs. ack-at-durable-broker-commit (option B)
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| A — ack at collector memory queue | Lower ack latency; simpler collector implementation | A collector crash between ack and forward loses the record — the ack lied about durability |
| B — ack at replicated broker commit | The ack genuinely means 'survives a single broker failure' (with RF=3, acks=all and min.insync.replicas=2) | Higher ack latency; requires the bus to be healthy to ack at all |
Decision 5
Is a Durable Bus (Kafka) Even Needed?
This is one of the two decisions worth defending in real depth — the answer depends entirely on burst tolerance and acceptable backlog behavior, not on bus technology being inherently 'more correct.'
| Option | Pros | Cons |
|---|---|---|
| No bus — collector writes straight to the index | Simpler pipeline, fewer moving parts, lower steady-state latency to searchable | Indexing throughput becomes the ingest ceiling; a burst or an indexer outage backs pressure all the way to the collector and agents |
| Durable partitioned bus between collector and indexer | Ingest and indexing fully decoupled; absorbs bursts up to the bus's retention window | A new, stateful, operationally complex component with its own failure modes (partition hot-spotting, disk pressure, consumer lag) |
The worked case for this module: an illustrative 50,000 events/s average, 40 MB/s, jumping to a 12× promotion-day peak of 600,000 events/s, 480 MB/s. Without a bus, that peak has to be absorbed by indexing throughput directly — and indexing (tokenization, inverted-index writes, segment merges) is typically far more expensive per event than a sequential, replicated append to a log. A bus converts 'indexing must instantly scale 12×' into 'the bus must have enough retention and disk to hold a 12× burst for the duration indexing needs to catch up' — a fundamentally easier problem.
Decisions 6-7
Partitioning and Consumer Scaling
Decision 6 — Partition key: service name vs. composite key
Swipe to pan
This is the same underlying problem consistent hashing solves for a different system — spreading keys evenly across a fixed or growing set of nodes/partitions without concentrating load behind a single poorly-chosen key.
| Option | Pros | Cons |
|---|---|---|
| Service-name-only key | Simple, predictable partition assignment per service | One hot service saturates its single partition regardless of broker count |
| Composite key (service + instance/shard) | Spreads a hot service's load across many partitions | Slightly more complex consumer-side reassembly if per-service ordering is needed |
Decision 7 — Index-then-commit vs. commit-then-index checkpoint ordering
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Index then commit offset | A crash produces a duplicate, fixable with an idempotent upsert key | Requires every indexed document to have a stable, deterministic ID |
| Commit offset then index | Slightly simpler indexer code, no ID-stability requirement | A crash produces a silent, unrecoverable skip with no automatic replay |
Decision 8
Inverted Index (ES/OpenSearch) vs. Label + Chunk (Loki)
This is the second decision worth defending in real depth — the two philosophies optimize for genuinely different access patterns, and the honest answer names the trade-off rather than declaring a universal winner.
Elasticsearch (and its OpenSearch fork) is the canonical inverted-index engine this comparison is built around — understanding its indexing and sharding model directly explains why it's write-expensive relative to a label-chunk alternative like Loki.
Swipe to pan
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Inverted index (ES/OpenSearch style) | Fast, flexible free-text search and rich aggregations across any indexed field | High write amplification; index frequently several times the raw compressed log size |
| Label + chunk (Loki style) | Dramatically cheaper at ingest — only a handful of labels are ever indexed; scales well for very high log volume | Free-text search means scanning more chunks; catastrophic cost blowup if a high-cardinality value is used as a label |
An inverted-index engine pays its cost continuously, at write time, on every single event, win or lose on whether that event is ever searched. A label-chunk engine defers nearly all of that cost to query time, paying only when a search actually needs to scan a chunk — cheap if most queries are well-scoped by label and time, expensive if most queries are broad, label-less free text.
Decisions 9-10
Retention and Tiering
Decision 9 — Fixed-duration TTL vs. tiered hot/warm/cold retention
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Single TTL, one tier | Simplest to operate and reason about | Pays hot-tier cost for data almost nobody queries after the first few days |
| Tiered hot/warm/cold | Meaningfully lower average cost per retained byte | Background migration jobs are a new failure surface; archive queries are slower, and that has to be communicated honestly |
Decision 10 — TTL-only deletion vs. an explicit legal-hold override
| Option | Pros | Cons |
|---|---|---|
| TTL-only, no override | Simple, predictable, fully automated deletion | Cannot honor a legal or compliance preservation obligation on a specific record |
| TTL plus an explicit, auditable legal-hold flag | Can preserve specific records past their normal schedule, with an audit trail of who/why/when | Added governance surface; a forgotten or mismanaged hold silently inflates storage cost |
Decisions 11-13
Query Planning and Tenancy
Decision 11 — Client-supplied tenant scope vs. credential-derived scope
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| Trust a client-supplied tenant_id | Simpler request shape; no server-side scope-resolution step | Trivially spoofable — any caller can claim any tenant's data |
| Resolve scope from the authenticated credential | Cannot be spoofed by a malicious or buggy client | Requires a credential-to-scope mapping service on every query's critical path |
Decision 12 — Shared query and ingest resource pools vs. separated pools
| Option | Pros | Cons |
|---|---|---|
| Shared compute/resource pool | Simpler capacity planning, fewer pools to size independently | An unbounded or abusive query can starve ingest, turning a search problem into a data-loss problem |
| Separated query and ingest pools | A query storm degrades search latency only, never the write path | More infrastructure to provision and right-size independently |
Decision 13 — Best-effort freshness vs. a guaranteed indexing SLA
| Option | Pros | Cons |
|---|---|---|
| Best-effort freshness, documented typical lag | Simpler to operate; no false promise during a genuine backlog | Operators must be comfortable with 'usually seconds, sometimes longer' rather than a hard number |
| Guaranteed indexing SLA | A crisper operational promise for dependent teams | Enforcing it requires significant reserved indexing headroom, raising baseline cost |
Decisions 14-16
Multi-Region and Observability
Decision 14 — Global replicated pipeline vs. independent regional pipelines
Swipe to pan
| Option | Pros | Cons |
|---|---|---|
| One global pipeline spanning regions | A single place to query everything, no federation layer needed | Cannot honor per-tenant data-residency requirements; a regional outage risks global impact |
| Independent regional pipelines + federated query | Residency-compliant by construction; regional outages stay regional | A cross-region query is slower and more complex than a single-cluster query |
Decision 15 — Cross-region failover vs. bounded local buffering during a regional outage
| Option | Pros | Cons |
|---|---|---|
| Fail over to another region automatically | Higher apparent availability during a regional outage | Moves a residency-bound tenant's data across a compliance boundary without consent |
| Bounded local buffering, reduced availability | Never silently violates residency | A long enough outage still means real, disclosed data loss or delay for that tenant |
Decision 16 — Self-observability through the same pipeline vs. an independent path
| Option | Pros | Cons |
|---|---|---|
| Monitor logging health via the logging pipeline itself | No separate monitoring infrastructure to build | A total logging outage makes its own failure invisible exactly when it matters most |
| Independent alerting path for pipeline health | A logging outage remains detectable even if logs themselves stop flowing | A second, smaller monitoring system to build, operate, and trust |
Decisions 17-23
Seven Additional Required Decisions
These close specific gaps against the module's own decision-bank specification — distinct questions from the sixteen above.
Decision 17 — Agent topology: sidecar vs. node DaemonSet vs. a gateway-only tier
This decision only exists because of how Kubernetes schedules pods onto shared nodes — a DaemonSet is a Kubernetes-native way to guarantee exactly one agent instance per node regardless of how many pods land there.
| Option | Pros | Cons |
|---|---|---|
| Sidecar per pod | Strong per-pod isolation; a noisy or crashing agent affects only its own pod | Resource overhead multiplied by pod count — one agent process per pod, even at idle |
| Node DaemonSet (chosen) | One agent per node regardless of pod density — far better resource efficiency at scale, and survives individual pod churn | Less per-pod isolation — a misbehaving agent affects every pod scheduled on that node, and the agent must handle multiplexing many pods' log streams correctly |
| Gateway-only (no per-node agent) | Simplest topology — applications ship directly to a central gateway | Loses all the local-disk-spool durability and backpressure benefits an on-node agent provides; reintroduces a direct network dependency on the application's critical path |
Decision 18 — Delivery guarantee: best-effort vs. at-least-once vs. dedup-approximated effectively-once
| Option | Pros | Cons |
|---|---|---|
| Best-effort (no retry) | Simplest, lowest overhead | Any transient failure anywhere in the pipeline is a silent, permanent loss — unacceptable for audit/compliance-class logs |
| At-least-once (chosen default) | No record is silently dropped on a retryable failure — the failure mode shifts to 'possible duplicate,' which is recoverable | Requires downstream consumers (indexer, alerting) to tolerate or dedupe duplicates |
| Dedup-approximated effectively-once | Downstream behavior looks like exactly-once for most practical purposes (search, counting) | Requires a stable per-record identity and a dedup window — true exactly-once delivery across independent systems isn't actually achievable, only approximated |
Decision 19 — Log representation: raw text vs. structured JSON vs. OTLP log records
| Option | Pros | Cons |
|---|---|---|
| Raw unstructured text | Zero integration work for legacy applications already printing plain lines | Every field of interest has to be extracted at query time via regex/grok parsing — slow, fragile, and duplicated per query |
| Structured JSON (chosen default) | Fields are parseable at ingest time once, queryable directly without per-query extraction; schema can be validated at the boundary | Requires application-level logging library changes to emit structured output instead of printf-style text |
| OTLP log records | Interoperates directly with the broader OpenTelemetry ecosystem (traces, metrics) using one shared data model and correlation approach | Heavier specification to implement fully; newer and less universally supported across legacy logging libraries than plain JSON |
Decision 20 — Batching and compression: latency vs. bytes, CPU, and large-event amplification
| Option | Pros | Cons |
|---|---|---|
| Small batches, no/light compression | Lower per-record latency to durable storage; simpler failure/retry semantics per batch | More per-batch overhead (network round trips, broker write amplification) at high volume |
| Large batches, aggressive compression (chosen default) | Significantly reduced bytes-on-wire and storage cost at scale; fewer broker writes per unit of log volume | Higher per-record latency (waiting for the batch to fill or a flush timer); a single corrupt or oversized record can hold up an entire batch; CPU cost of compression itself |
Decision 21 — Ingest-time parsing/enrichment vs. query-time extraction
| Option | Pros | Cons |
|---|---|---|
| Query-time extraction | No ingest-time cost; flexible — can extract any field a future query needs, even ones not anticipated at write time | Every query pays the parsing cost, every time, against potentially the full matched record set — this is what makes unbounded free-text regex queries dangerous (Failure & Scale Scenarios, F17) |
| Ingest-time parsing/enrichment (chosen for known fields) | Parsing cost paid once per record, not once per query; enables fast structured filtering on common fields | Commits to a schema at ingest time — a field nobody anticipated needing still requires query-time extraction as a fallback, or a reindex |
Decision 22 — Sampling: none for audit/error-class logs vs. selective debug-log sampling
| Option | Pros | Cons |
|---|---|---|
| No sampling, ever | No statistical blind spots — every log line is available for any future investigation | At high debug-log volume, storage and indexing cost can dominate the entire pipeline's budget for records that are rarely, if ever, queried |
| Selective sampling for debug-level logs only (chosen) | Materially reduces cost for the highest-volume, lowest-value log class | Introduces a real statistical blind spot — a rare bug reproducing only in sampled-out traffic may never appear in the retained sample |
Decision 23 — Search semantics: full query flexibility vs. bounded time/service selectors
| Option | Pros | Cons |
|---|---|---|
| Full, unrestricted query flexibility | Maximum expressiveness for an investigator who doesn't yet know what they're looking for | An unbounded wildcard/regex query with no time range is exactly the failure mode in F17 — it can single-handedly starve the query tier for every other tenant |
| Bounded selectors required (chosen) | A mandatory time range and a cost/complexity budget per query bounds worst-case resource consumption, protecting multi-tenant fairness | Forces an investigator to narrow scope upfront, which can feel like friction during an active incident when the right narrowing isn't obvious yet |