Elasticsearch in Production

For log/event data, creating one index per day/week (e.g. logs-2026.08.08) makes it trivial to drop old data cheaply — deleting a whole index is far faster than deleting matching documents. In modern Elasticsearch, prefer a data stream (for append-only time-series data) managed by an index lifecycle management (ILM) policy: the policy rolls over the backing index by age or size and handles retention automatically, instead of cron jobs that create and delete dated indices by hand.

An alias points at one or more underlying indices, letting you reindex into a new index and atomically switch traffic over without any application changes.

Define mappings explicitly rather than relying on dynamic mapping in production — it prevents type-guessing surprises and keeps mapping growth (the 'mapping explosion' problem) under control.

Map a field as both text and keyword (e.g. name and name.keyword) when you need both full-text search and exact matching/sorting/aggregation on the same underlying value.

Elastic's shard-sizing guidance is to aim for roughly 10GB-50GB per shard (and under about 200 million documents) — a guideline, not a hard limit. Much smaller wastes per-shard overhead; much larger hurts recovery time and search latency. With ILM, the rollover action's max_primary_shard_size can enforce the upper bound for you.

Too many small shards multiply cluster-state overhead and file handles per node — a common cause of slow cluster operations and degraded performance even with light data volume.

At least 1 replica per shard is standard so a single node failure never causes data loss or a search gap — replicas are promoted automatically.

Because search requests can be served by any copy of a shard (primary or replica), adding replicas beyond the minimum for HA can directly increase search throughput for read-heavy workloads.

Defaults to 1 second. Lowering it makes data searchable sooner but creates more, smaller Lucene segments that must later be merged — raising it (or disabling refresh) during bulk loads significantly improves indexing throughput.

Larger batches amortize overhead further but increase memory pressure per request and delay when documents in that batch become searchable. Typical production batch sizes range from a few hundred to a few thousand documents, tuned empirically.

Multiple concurrent bulk requests (rather than one giant serial stream) generally achieve higher aggregate throughput, up to the point where the cluster's write queues start rejecting requests.

Clauses in filter context (no scoring needed) are cacheable and typically faster than the same clause in query context — use filter for anything that doesn't need relevance scoring, like date ranges or status flags.

from/size pagination gets progressively more expensive the deeper you page, because every shard must compute and sort from + size results. Use search_after for deep pagination instead.

Production clusters typically separate dedicated master nodes (cluster coordination, no data), data nodes (store shards, serve search/index), and sometimes coordinating-only nodes (fan-out/merge, no data) — isolating responsibilities so a heavy query load can't destabilize cluster coordination.

For time-series/log data, newer indices live on fast (hot) nodes for active writes, older indices move to cheaper (warm/cold) nodes for read-mostly access — balancing cost against performance as data ages. Elasticsearch implements this with data tiers (hot, warm, cold, frozen): nodes are assigned tier roles, and an ILM policy moves indices between tiers as they age.

Configuring rack/zone awareness prevents all replicas of a shard from landing in the same failure domain, so a single rack or availability zone outage can't take out both a primary and its replica.

Green = all primaries and replicas allocated. Yellow = all primaries allocated but some replicas aren't (often just a node restart in progress). Red = at least one primary shard is unallocated — data for that shard is unavailable.

JVM heap usage and GC pause time, indexing/search latency percentiles (not just averages), thread pool rejections, and pending tasks are the highest-signal indicators of cluster stress.

Dynamic mapping on unpredictable JSON creates thousands of fields, bloating cluster state and slowing every operation.

Shards that grow far past the recommended range slow down recovery, relocation, and even individual query latency.

Elasticsearch trips a circuit breaker and rejects requests when an operation (often a huge aggregation) would exceed configured memory limits — protecting the node from an out-of-memory crash at the cost of that request failing.

_cat/shards, _cat/indices, _cat/nodes give fast, human-readable snapshots of cluster and shard state — usually the fastest way to spot an unbalanced or unhealthy cluster.

"profile": true on a search request breaks down exactly which query clauses and which shards consumed the most time — invaluable for diagnosing a single slow query rather than guessing.