Search & Autocomplete — Failure & Scale Scenarios

Health checks against search nodes (on some regular interval) detect the failure and stop routing to the failed node.

Remaining/replica nodes absorb redistributed load — expect some latency increase, not necessarily a full outage, and the exact behavior depends on replica count and consistency configuration for that specific system.

The failed node, once restarted, needs to sync its shard data from a healthy replica before rejoining the serving pool — for large shards this sync can take a non-trivial amount of time, during which the cluster runs with reduced redundancy.

Collapsing many concurrent identical requests into a single upstream query is the primary defense — this helps regardless of exact hit-rate math, since it prevents the same hot query from being computed redundantly many times over.

Randomizing TTL slightly per cache entry avoids many entries expiring in the same instant, which is a common self-inflicted cause of stampede-like spikes even without an actual cache outage.

Serving a slightly-expired result while refreshing in the background avoids a hard cliff at TTL expiry.

The cluster needs enough spare capacity to absorb a realistic cache-loss spike, and circuit breaking/load shedding to protect itself if the spike still exceeds that headroom — rather than assuming the cache will always be there to absorb load.

The search index is a read-optimized derivative, not the source of truth — pipeline failure causes staleness, not necessarily search downtime, as long as the index itself is still reachable.

Catching up depends on: the source/broker having actually retained the backlog of events (not expired past a retention window); the pipeline having sufficient catch-up throughput to work through the backlog faster than new events arrive; correct ordering/versioning so catch-up doesn't apply stale events over newer ones; and no single poison event blocking the whole backlog from progressing.

Track both lag (how far behind, in time or event count) and the age of the oldest unprocessed event — the second metric catches a stuck-but-still-advancing pipeline that simple lag-count monitoring can miss.

Already-cached hot queries are absorbed by the cache layer; genuinely novel queries (a brand-new trending term, a new product just launched) bypass the cache entirely on first occurrence and hit the search cluster directly.

Each user session doing a single full-text search may have generated many autocomplete requests (one per keystroke, moderated by debounce) — autocomplete QPS during a spike can be a meaningfully larger multiple of "logical searches" than full search QPS is.

Adding search nodes during a spike isn't instant relief — new nodes typically need shard data synced/loaded before they can usefully serve queries, so auto-scaling has a lag between "capacity added" and "capacity actually helping."

Protects cluster stability by shedding or throttling excess load rather than letting every node degrade uniformly under overload.

The coordinator needs an explicit timeout for how long it will wait on any single shard before deciding to proceed without it — without this, one slow shard can drag the entire query's latency arbitrarily high.

If the specific node is slow/unreachable but a healthy replica for that shard exists, routing to the replica instead can avoid the problem entirely.

As covered in Trade-offs → Fail Query vs Return Partial Results, whichever choice is made, a partial response must be clearly marked as such.

This scenario is precisely why scatter-gather tail latency scales with the number of shards queried — more shards means more chances that at least one is the slow outlier for any given query.

Requires active relevance/quality monitoring (query success-rate proxies, click-through-rate shifts, automated relevance regression checks) — a bad deploy of this kind doesn't set off traditional uptime/error-rate alerts.

This is the direct payoff of the versioned-index-plus-alias pattern (see Architecture Deep Dive → Zero-Downtime Reindexing): validate before cutover, and if a bad version somehow reaches production, switch the alias back to the last known-good versioned index — rolling back a live in-place index is far riskier or may be impossible.

If the bad index also wrote incorrect data (not just a bad mapping), a rebuild from the authoritative source database is the reliable fix, not attempting to patch the corrupted index in place.

Every update carries a monotonic version/sequence number from the source; the indexer compares the incoming version against the currently-indexed version and rejects/no-ops any update whose version is not newer (see Architecture Deep Dive → Indexing Pipeline Correctness).

A particularly sharp edge case: a document is deleted (tombstoned), and then a delayed update for a pre-delete version of that document arrives. Without version/sequence awareness, this can incorrectly resurrect a document that should stay deleted — the indexer needs to treat a tombstone as a terminal state that a stale update cannot override.

Compare the index against the authoritative source (document counts, checksums, or targeted sampling) to detect and scope the actual divergence — see Source/Index Divergence below for the general-purpose version of this.

If the gap is significant or can't be safely bounded, a full reindex from the source snapshot is the reliable recovery path, using the zero-downtime migration pattern rather than reindexing the live index destructively.

After a snapshot rebuild, apply any changes that occurred during the rebuild, then validate document counts/spot-check quality before cutting traffic over to the rebuilt index.

This is exactly why the recommended pattern builds into a new versioned index and only atomically switches the alias after validation (see Architecture Deep Dive → Zero-Downtime Reindexing) — a failed build simply means the new index isn't ready yet, not that live search is broken.

Restart or resume the new index build (from the last good checkpoint, if the pipeline supports one, or from scratch) without touching the still-serving old index. Investigate and fix the root cause (schema bug, capacity issue, throughput mismatch) before attempting cutover again.

If re-ranking can't be reached, fall back to serving results ordered by the first-stage (BM25-style) score alone, rather than failing the whole query.

Any business-rule or personalization layer that depends on the same failed service is skipped along with it, in favor of continuing to serve some reasonable result.

Request coalescing (again) plus, for genuinely extreme cases, replicating the hot cache entry across multiple cache nodes rather than relying on a single node/key to absorb all the load.

For a hot posting list or hot shard specifically, ensuring adequate replica capacity for that shard specifically, not just average per-shard replica count.

Protects against a single hot term (or a client hammering it) from degrading service for unrelated queries.

For a hot prefix, a precomputed/cached suggestion list is far cheaper to serve repeatedly than recomputing ranking live for every request.

Bound the maximum number of clauses, term expansions, or wildcard breadth a single query is allowed to generate.

Every query needs an enforced deadline so one expensive query can't monopolize a shard indefinitely.

As covered in Typo Tolerance and Fuzzy Search, bounding these directly prevents the worst-case explosion in expanded term count.

Reject or degrade query classes that repeatedly trip resource thresholds, protecting overall cluster health over serving every possible query shape.

Cap how many distinct facet values are aggregated/returned, rather than allowing an unbounded high-cardinality field to make a single facet request arbitrarily expensive.

Per-shard size and query-load monitoring, not just cluster-wide aggregates — a hot shard can be invisible in average/aggregate metrics while being a real bottleneck for the queries that happen to hit it.

Redistributing data (splitting an oversized shard, adjusting the routing scheme for future writes) to even out the skew — this is genuinely more involved than simple document copying, since it can require replanning the routing key or reprocessing affected documents.

Adding extra replica capacity targeted at the specific hot shard, rather than uniformly increasing replica count cluster-wide, can be a faster mitigation than a full rebalance.

If autocomplete uses a dedicated structure or a precomputed popularity-ranked suggestion list, it's reasonable (and common) for it to have its own update cadence distinct from the main index's — this should be an explicit, monitored SLO rather than an accidental side effect nobody set out to build.

Understand whether the autocomplete structure updates incrementally (new terms/documents trickle in continuously) or via periodic full rebuild (a batch job regenerates suggestions on a schedule) — this determines both the expected staleness window and how to reason about a stuck update process.

Periodically compare aggregate document counts, per-partition checksums, or targeted random sampling between source and index to detect divergence before it's reported by confused users.

A scheduled reconciliation job that identifies and corrects specific divergent documents, rather than relying solely on the live event pipeline to eventually self-correct.

For divergence too large or too systemic to patch incrementally, a full reindex from the authoritative source remains the reliable reset — the same pattern used for corruption recovery and event-loss recovery above.

Treating index-source divergence as a continuously monitored metric (not just something investigated reactively after a user complaint) catches drift early, before it compounds.