Distributed Logging Failures: Backlogs, Duplicates, Leaks & Recovery

A drop counter metric on the agent spikes from zero; self-observability telemetry alerts on any sustained non-zero drop rate. Mitigation is architectural, not reactive: a bounded disk spool (Chapter 06 of Complete Design) absorbs an outage of this length without resorting to drops at all.

Once the collector recovers, the agent resumes normal shipping immediately — no replay mechanism exists for already-dropped events, because they were never durably recorded anywhere. The correctness result is an honestly-measured, bounded loss window, not a silent one: the drop counter is the permanent record of how much was lost and when.

The spool's own fill-percentage metric crosses a warning threshold well before it's full, giving operators lead time; mitigation at that point is operational (escalate the outage) rather than architectural — no spool can be made infinite.

On recovery, the agent drains the spool in FIFO order before resuming live traffic, so backlog records are shipped in their original relative order. The correctness result: everything that fit in the spool survives with its original event_time intact; everything generated after the spool filled is permanently absent, not delayed.

A spike in duplicate record IDs (if idempotent keys exist downstream) or a detectable gap in a monotonic per-instance sequence number surfaces this. Mitigation: track file identity (inode plus a fingerprint) and an acknowledged byte offset, never checkpointing inside a partial line, as described in Architecture Deep Dive Chapter 03 — this correctly distinguishes a renamed file (same inode, new name) from a copytruncate rotation (same name, truncated content, new effective identity).

With file-identity tracking, the agent finishes the old file before switching to the new one, so a rename-based rotation produces no gap. After a crash the agent re-reads from its last acknowledged offset, so a few lines can be delivered twice, and downstream deduplication absorbs them. The correctness result is no silent gap and no unbounded duplication: at-least-once delivery, in order within the file. Copytruncate remains lossy for lines written between the last read and the truncation, which is a reason to prefer rename-based rotation.

Certificate expiry should be monitored proactively (days of lead time), making detection at the failure instant itself a sign the proactive check already failed. Immediate mitigation: agents fall back to local disk spooling exactly as in an ordinary collector outage — this failure mode is, from the agent's perspective, indistinguishable from F01/F02 and handled by the same mechanism.

Once the certificate is rotated and the collector tier restarts or reloads it, agents reconnect and drain their spools. Correctness result depends entirely on whether the outage fit inside spool capacity — same bound as F02.

Consumer lag on the affected partition climbs sharply while every other partition stays healthy — a distinctive single-partition-stall signature. Mitigation: route records that fail schema validation to a dead-letter queue instead of raising an unhandled error, so the indexer can skip past the poison record and keep consuming.

Once a DLQ path exists, recovery is automatic — the malformed record is quarantined and alerted on, while the rest of the partition continues processing without interruption. Correctness result: a quarantined record is not lost, just deferred for manual inspection and potential reprocessing after a fix; it is categorically different from a record dropped due to a capacity policy.

Collector queue depth and agent spool fill rates spike together, fleet-wide, correlated with the traffic event. There is no fast mitigation available once this starts — the architectural fix (a durable bus decoupling ingest from indexing) has to exist before the burst, not be added during it.

Recovery happens only as fast as indexing throughput allows the backlog across every buffering layer to drain, and any layer whose buffer filled before recovery began has already dropped data by this point. Correctness result: a bus-less design under-provisioned for its real burst ratio has no bounded, calculable loss — loss is whatever each layer's buffer policy happened to produce.

This gap is invisible from the agent's perspective by construction — the ack lied about the durability stage reached. Detection, if it happens at all, comes later: an investigation expecting to find a specific record in the durable store finds nothing, despite an agent-side log showing a successful ack. Mitigation requires switching to ack option B (acknowledge only after the broker commit) before the fact; there is no after-the-fact fix for records already lost this way.

No recovery is possible for the specific records lost in the crash window — they were never durable. The correctness result under option B instead: an ack is only issued after a replicated broker commit, so a collector crash after ack can never lose the record, by construction, because durability was already established before the ack was sent.

Detection typically comes from a secret-scanning sweep over indexed content, or from the credential itself being observed in unexpected use. Mitigation at that point is incident response: rotate the exposed credential immediately, and purge or redact the indexed copies after the fact — which does not undo the exposure window that already occurred.

There is no way to make an already-exposed secret retroactively unexposed; recovery is limited to credential rotation and purging the stored copy. The correctness result under a fail-closed redaction policy instead: a record the redaction engine cannot confirm is safe is held back or quarantined rather than forwarded, trading a false positive (a safe record briefly delayed) for eliminating this false negative entirely.

Per-partition consumer lag metrics show one partition's lag diverging sharply from its peers. Mitigation without a repartition: temporarily throttle or sample checkout-api's lowest-priority log levels to relieve the hot partition, accepting reduced fidelity for that service until a real fix ships.

Full recovery requires repartitioning with a composite key (service + instance or shard), which requires a migration — existing data in the old partition layout doesn't retroactively redistribute. Correctness result: no data is lost, only delayed, as long as the single hot partition's retention window outlasts the delay — the same backlog math as F16 applies to a single partition instead of the whole bus.

A spike in rebalance events correlated with the scale-up event is visible in consumer-group metrics. Mitigation: stagger new consumer joins rather than adding them all simultaneously, and use a cooperative (incremental) rebalancing strategy where supported, which avoids a full stop-the-world reassignment on every membership change.

The group settles once joins stop and a stable rebalance completes; consumption resumes from each partition's last committed offset, so no records are lost — only momentarily delayed. Correctness result: a temporary, bounded freshness dip, never a correctness violation, as long as offsets are managed correctly through the rebalance.

Detection: monitor time-sync offset on every host, because by the time an investigator notices an ordering inconsistency in search results the skew has probably existed for a while. A second signal is built into the data: compare event_time with observed_time per source. A source whose gap is consistently negative (the event is stamped after the collector saw it) has a fast clock; a source whose gap is large and constant has either a slow clock or a long hold time in its agent, and the agent's monotonic hold-time metric separates the two. Mitigation in the query layer: always keep observed_time, which the platform stamps on a monitored clock, as an alternative sort key so an investigator can cross-check.

No retroactive correction is applied to already-recorded event_time values, because rewriting them would fabricate data. If the offset of a source is known from monitoring, a query tool may display a corrected time as a clearly labeled derived value beside the original. Correctness result: this design never claims a single trustworthy global ordering across hosts. It offers two honestly labeled orderings, event_time (the producer's claim) and observed_time (the platform's receipt time, which also includes buffering delay), and lets the investigator choose which fits the question (invariant I9). For true causal order across services, use trace and span relationships rather than wall-clock timestamps.

If the index write is a true upsert keyed by a stable record ID, this is invisible to a searcher — the second write overwrites the first with identical content. Detection, where it matters, is via indexer-side duplicate-write counters, useful for understanding crash frequency even when the user-facing effect is zero.

No explicit recovery action is needed — replay self-heals via the idempotent upsert. Correctness result: exactly-once visible search behavior is achieved despite at-least-once delivery, specifically because the index write is idempotent, not because the pipeline achieves true exactly-once delivery (invariant I3).

This is detectable only through an independent count-reconciliation process — comparing total records produced to total records indexed over a time window — never through the indexer's own consumption metrics, which show a perfectly healthy, monotonically advancing offset with no visible error. Mitigation requires exactly this kind of independent audit, run continuously, because the failure produces no local symptom.

Recovery is only possible if the raw record is still available from an upstream retention window (the broker's own retention, if long enough, or an independent raw-archive copy) and a reconciliation process specifically re-reads and re-indexes the gap. Correctness result: this ordering should not be the default (Trade-offs & Decisions, Decision 7) precisely because its failure mode is silent and requires external detection rather than self-healing.

A stream-count metric (or equivalent cardinality gauge) spikes abruptly, correlated with the label-schema change. Immediate mitigation: revert the label configuration for new writes; already-created high-cardinality streams typically need to age out via normal retention rather than being surgically removed.

No data is lost — this is a cost and performance failure, not a durability one. Recovery is simply reverting the schema and waiting for the bad streams to retire via TTL. Correctness result: the fix is architectural (move request_id to a structured field or metadata, never a label) rather than a configuration tuning knob.

A single query's resource consumption (bytes scanned, worker-seconds) spikes far above typical query cost, visible in query-layer telemetry. Mitigation: an enforced per-query scan-byte budget and concurrency cap, applied at plan time before any shard is dispatched (Architecture Deep Dive Chapter 07), rejects or truncates the offending query before it can monopolize capacity.

Once the budget-enforcing query planner is in place, recovery is automatic — the storm query is rejected or bounded, and other queries proceed normally without ever being affected. Correctness result: separating query and ingest resource pools (Trade-offs & Decisions, Decision 12) additionally guarantees this failure, even unmitigated, could never degrade the write path.

The oldest-unprocessed-message-age metric, watched through the self-observability path (invariant I10), crosses an alert threshold well before the outage resolves. At an illustrative 40 MB/s average ingress, 30 minutes (1,800 seconds) of accumulation is 40 × 1,800 = 72,000 MB ≈ 72 GB of raw, unindexed backlog by the time indexing resumes.

Recovery requires indexing throughput to exceed ingress for a sustained period (see F17's drain math). Correctness result: this is a freshness incident, not a data-loss incident, as long as the bus's retention window exceeds the outage duration plus the subsequent drain time — the backlog math only stays survivable if that condition holds.

The oldest-unprocessed-message age plateaus instead of declining — a clearly distinguishable signal from a successful drain, where it should be dropping. Mitigation: scale indexing capacity meaningfully above ingress — this design's reference target is roughly 80 MB/s raw-equivalent against the ongoing 40 MB/s ingress.

At a net drain rate of 80 − 40 = 40 MB/s, the 72 GB backlog clears in 72,000 MB ÷ 40 MB/s = 1,800 seconds, or 30 minutes. Correctness result: the general rule — consumer throughput must strictly exceed sustained arrival rate, not merely match it — is the only condition under which any backlog actually drains rather than merely stabilizing.

Broker-disk-headroom alerts fire ahead of actual exhaustion. Mitigation has two parts: upstream agents fall back further onto their own local spools (buying additional time for sources that can tolerate it), and the platform applies an explicit, documented load-shedding policy by log class — throttling or dropping best-effort debug traffic first, while protecting reserved retention headroom for error and audit-class events.

Recovery proceeds as indexing drains the backlog (per F17's math), relieving broker disk pressure from the read side while load-shedding relieves it from the write side. Correctness result: a class-aware, explicit shedding policy preserves the most important data under genuine capacity exhaustion; the alternative — a silent, undifferentiated overwrite of the oldest records regardless of class — is never acceptable, because it turns a capacity problem into an unaudited, undisclosed data-loss event.

This isn't a system failure in the technical sense — it's an expectation mismatch. Mitigation is interface-level: the query UI should clearly label which tier a query will hit and give an honest latency estimate before the engineer commits to waiting, rather than presenting a spinner indistinguishable from a hot-tier query.

No recovery action is needed beyond the query eventually completing. Correctness result: the honest trade-off (Trade-offs & Decisions, Decision 9) stands — cheap long-term retention and fast interactive search are not simultaneously achievable, and communicating that trade-off clearly at query time is itself part of the design.

Collector-to-broker publish error rates spike in the affected region specifically, while the broker's own health checks (reachable from elsewhere) show it as healthy — a signature that distinguishes a partition from a genuine broker failure. Mitigation is the same spool-and-wait behavior as other collector-side outages; there is no safe cross-region reroute for this traffic without violating residency (F23 covers that constraint directly).

Once the partition heals, collectors resume publishing and agents drain their spools. Correctness result: bounded by spool capacity exactly as in F01/F02 — a network partition is, from the data-loss-math perspective, just another outage duration to measure against spool and bus retention.

The planner's scope-resolution step detects the mismatch between the claimed and credential-derived tenant immediately, before any shard is queried. Mitigation is structural, not reactive: this check exists on every single query, unconditionally (Architecture Deep Dive Chapter 08) — there's no separate detection step because prevention is built into the request path itself.

The request is rejected with a 403 before touching any tenant's data; no recovery action is needed because no data was exposed. Correctness result: invariant I4 holds by construction — a caller-supplied tenant_id is a hint at best, never an authorization boundary, and this scenario is exactly what that invariant exists to prevent.

A per-tier data-age distribution metric (what fraction of hot-tier data is older than its intended window) catches this directly; a pure cost-dashboard anomaly is a slower, indirect signal of the same problem. Mitigation: restart the migration job and verify it processes the accumulated backlog of overdue data.

No data is lost — this is purely a cost and tiering-policy violation, not a durability or search-correctness one. Recovery completes once the migration job catches up on the backlog of overdue records.

The regional outage itself is detected through standard regional health monitoring. Mitigation is the agent-level spool fallback already covered in F01/F02/F04/F20 — the only difference here is that there is no cross-region escape valve available for this specific tenant class, by design.

If the outage outlasts the local spool's capacity, the tenant experiences a genuine, disclosed reduction in log availability — some events are delayed or lost — rather than a silent residency violation. Correctness result: the honest trade-off from Trade-offs & Decisions, Decision 15, accepted explicitly rather than discovered during the incident.

This is detected the moment the hold request is processed against a record ID that no longer resolves. Mitigation depends entirely on whether a longer-retention or immutable archive copy still exists elsewhere (e.g. a compliance-grade cold archive with its own, longer default retention) — if one does, the hold can still be honored there even though the primary tier already deleted its copy.

If no surviving copy exists anywhere, this is an unrecoverable gap, and it must be reported as such to legal counsel honestly rather than hidden. Correctness result: this scenario is exactly why a legal hold should be checked proactively against records approaching their retention boundary, not only applied reactively after a request arrives — invariant I8 is about accountability for this exact race, not a guarantee that it can never occur.

This is invisible until investigated — there's no error, just an absence. Detection comes from comparing expected log volume (from a healthy baseline) against actual volume in the window before a crash, or noticing a suspiciously clean gap right before a crash-loop restart. Mitigation is structural, not reactive: minimize the in-memory batching window for a process already showing memory pressure, or write log lines synchronously to a local buffer the OOM killer can't take with it (a sidecar's buffer survives the app process dying, which is the whole reason the agent model exists).

There is no recovery for the specific lines lost in an unflushed in-process buffer — they never left the process's memory. The correctness result: any log line is only as durable as the earliest point it leaves the originating process; this is exactly why the sidecar/agent-based pipeline (Complete Design, Chapter 03) exists instead of trusting the application process to flush its own logs reliably under memory pressure.

Detection: a pattern of missing 'final lines' specifically correlated with pod terminations (not crashes generally) — comparing container exit timestamps against the collector's last-read-offset timestamp for that container's log file surfaces the gap. Mitigation: extend the termination grace period specifically to give the collector a bounded extra window to catch up, and have the collector treat a container-terminating event as a signal to immediately flush-read that container's remaining log file content, rather than waiting for its normal poll interval.

No recovery once the container's log file is garbage-collected post-termination — the source no longer exists. Correctness result: 'the container exited successfully' and 'every line it wrote was collected' are two different claims, and this failure is exactly the gap between them — a collection pipeline must treat the window between container exit and log-file garbage collection as a real, bounded deadline, not an afterthought.

Detection: a duplicate-content signature (same source, same offset range, same content hash) arriving twice in the durable store, distinguishable from a legitimate repeated log message by the identical byte-for-byte offset range. Mitigation is architectural, not reactive: downstream deduplication keyed on a stable per-record identity (source + offset range, or a content hash) at the point of durable commit, so a resend is absorbed rather than double-counted.

No recovery needed if downstream idempotency-based dedup exists — recovery *is* the dedup. Correctness result without it: this ordering (hand off, receive the acknowledgment, then checkpoint) can only produce at-least-once delivery, never exactly-once, by construction. Checkpointing earlier, for example after appending to an in-memory buffer, flips the failure mode to at-most-once: a crash leaves the checkpoint ahead of data that was never persisted, an unrecoverable skip. That is why the chosen order is read, hand off, wait for the acknowledgment, then checkpoint, paired with downstream dedup: a duplicate is recoverable, a skip is not.

Detection: a leader-election event (visible in broker cluster metadata/metrics) correlated with a drop in acked-write throughput and, under acks=1, a gap between producer-reported-acked records and what the new leader actually has. Mitigation: acks=all with a minimum in-sync-replica count greater than one is what prevents this loss window from existing at all — the producer simply never gets an ack for a record that didn't survive the leader's death, and retries instead.

Under acks=1, records acked but not yet replicated before the leader died are unrecoverable — they were never anywhere but the dead leader's local log. Under acks=all with min.insync.replicas ≥ 2 and unclean leader election disabled, every acknowledged record was on at least two replicas, and a new leader is chosen only from in-sync replicas, so a single leader failure produces at worst a producer-visible retry, never a silent loss of something the producer believed had succeeded. If every in-sync replica is gone, the partition stays offline rather than electing a stale replica; enabling unclean election would trade that unavailability for permanent loss of acknowledged records.

Detection: index cluster field-mapping count climbing in lockstep with a specific deployment's rollout, or a spike in type-conflict rejection errors in the indexer's own logs immediately following that deployment. Mitigation: a strict, versioned schema with an explicit allowlist of indexed fields and types (Architecture Deep Dive's schema chapter) rejects or quarantines a record with an unexpected shape at ingest time, rather than letting an uncontrolled schema-on-write index absorb it and silently degrade.

Recovery requires identifying and reindexing (or dropping and rebuilding) the polluted index if mapping explosion has already degraded cluster performance — an expensive, often hours-long operation on a large cluster. Correctness result with strict schema enforcement instead: a malformed-shape record is quarantined at ingest, visible in a dead-letter path for investigation, and never reaches the index at all — the index's mapping stays bounded regardless of what any single deployment ships.

Detection: measure the pipeline stages with timestamps from the platform's own monitored clocks, for example t_indexed minus t_durable and t_durable minus observed_time, and measure waiting inside the agent with a monotonic clock. A dashboard of those lags lets an on-call engineer check whether current end-to-end lag is within normal bounds before concluding anything is lost. Do not use now minus event_time as the lag metric: it includes the producer's clock error. Without this instrumentation the only tool available is waiting a bit and querying again, which is a poor incident-response posture.

No recovery is needed if the record is simply late: it becomes visible once the indexer has processed it, and the lag dashboard would have shown that this was expected. If lag is elevated beyond the normal range, that is a distinct, actionable signal (an indexing backlog, per F16/F17) rather than a mystery. Correctness result: 'missing from search' is not a well-formed incident claim on its own. It has to be qualified, for example 'missing while indexing lag is 3 seconds, normal is under 10,' to be actionable.

Detection: a periodic PII-scanning sweep over indexed content (pattern-matching plus, ideally, a classifier) run as an ongoing compliance control, not a one-time check. Mitigation has three separate, necessary parts because the data now exists in three separate places: revoke or redact the live searchable copy in the hot index, locate and redact or delete the colder-tier copy, and locate and redact or delete the archived copy — a single 'delete from the database' action does not reach all three.

Recovery cannot undo any exposure that already happened (a query that already returned the record, a dashboard that already displayed it) — recovery is limited to stopping *further* exposure from this point forward. Correctness result: a system promising 'PII removal' without an explicit answer for the archive tier is making an incomplete promise — a lifecycle policy that forgets a record was copied to cold archive before redaction was applied is exactly how this goes wrong.