Interactive guide · 8 chapters · ~20 min read
Elasticsearch Fundamentals
How Elasticsearch turns documents into searchable terms, finds and ranks matches, and spreads data across a cluster of nodes with no single point of failure.
Mangesh MandlikLast updated:
01 · Foundations
What is Elasticsearch?
Elasticsearch is a distributed search and analytics engine built on top of Apache Lucene — designed to find relevant documents out of millions in milliseconds, not just retrieve rows that match an exact condition.
A traditional database answers "which rows have status = 'active'". Elasticsearch answers a fundamentally different question: "which documents are most relevant to this query" — ranking free-text matches by how well they fit, tolerating typos, synonyms and word order, and doing it across a cluster of machines instead of one.
Swipe to pan
Elasticsearch Architecture: Inverted Index, Shards, and Replicas
The architecture pairs a Lucene-based inverted index with a distributed layer: each index is split into shards spread across nodes, and replicas keep extra copies, so search scales by adding nodes. The diagram above shows the inverted-index half of that picture — a document analyzed into terms.
The defining characteristics
Document-oriented
Data is stored as JSON documents inside indices — no fixed relational schema is required, though mappings define how each field is analyzed and stored.
Distributed by default
Every index is split into shards spread across nodes — search scales horizontally by adding nodes, not by making one machine bigger.
Full-text search first
Search is the primary use case: relevance scoring, fuzzy matching, and language-aware analysis are core features, not add-ons.
Near real-time
New documents become searchable within about a second (the refresh interval), not instantly — a deliberate trade-off for throughput.
Interactive
Visual Learning
Watch a document travel from index request to inverted index to ranked search result — tracing sharding, relevance scoring and near-real-time search through one accumulating cluster.
02 · History
Why Elasticsearch Was Created
Elasticsearch was built to make Apache Lucene — an extremely powerful but low-level Java search library — usable as a distributed, horizontally scalable service with a simple REST API.
Lucene provides the core search primitives (inverted indexes, scoring, tokenization) but is a single-machine, embedded library with no clustering, no REST API, and no JSON document model. Elasticsearch (created by Shay Banon in 2010) wraps Lucene with distribution, replication, a REST/JSON API, and cluster management — turning a search library into a search system.
03 · Data model
Documents, Indices & Mapping
A document is a JSON object; an index is a collection of documents with a shared mapping — Elasticsearch's rough equivalent of a table.
Document
A single JSON record, identified by an _id, stored in exactly one index and (via routing) exactly one primary shard.
Index
A logical namespace of documents that share the same mapping — physically split into one or more shards.
Mapping
The schema for an index: which fields exist, their data types (text, keyword, date, long, ...), and how each is analyzed. Mapping is set at index-creation time and is far more expensive to change than a SQL ALTER TABLE.
04 · Search fundamentals
The Inverted Index
The inverted index is the single data structure that makes full-text search fast — it maps every term to the list of documents containing it, instead of mapping documents to their content.
Given the query "find documents containing the word 'shoes'", a naive approach scans every document — O(n). An inverted index instead looks up "shoes" directly in a sorted structure and gets back the exact list of matching document ids — no scanning required.
05 · Search fundamentals
Analyzers & Tokenization
Before text becomes a term in the inverted index, it passes through an analyzer — a pipeline of character filters, a tokenizer, and token filters.
Character filters
Run first, on the raw string — e.g. stripping HTML tags or punctuation before tokenization happens.
Tokenizer
Splits the (filtered) text into individual tokens — usually on whitespace and punctuation for the standard tokenizer.
Token filters
Transform tokens after splitting — lowercasing, stemming ("running" → "run"), removing stop words ("the", "a"), etc.
jsonIllustrative: see what an analyzer does with POST _analyze (response abbreviated to the token field)
POST _analyze
{ "analyzer": "standard", "text": "The Quick Brown Foxes!" }
// tokens: the, quick, brown, foxes
POST _analyze
{ "analyzer": "english", "text": "The Quick Brown Foxes!" }
// tokens: quick, brown, fox (stop word removed, "foxes" stemmed)This is also why a term query for Foxes finds nothing on a text field analyzed with standard: the index holds foxes, and term does not lowercase its input.
06 · Search fundamentals
text vs keyword — Full-Text vs Exact Matching
Elasticsearch's two most important field types answer two different questions: 'does this field roughly match?' versus 'does this field exactly equal this value?'
Swipe to pan
text
Analyzed into terms at index time. Queried with match, which analyzes the query the same way — good for free-text search, forgiving of case, word order, and stemming.
keyword
Stored exactly as-is, unanalyzed. Queried with term (or used for sorting/aggregations) — must match the full string exactly, including case.
07 · Ranking
Relevance & Scoring (BM25)
Search results are ranked, not just filtered — Elasticsearch calculates a relevance score for every match using the BM25 algorithm by default.
Term Frequency (TF)
How often a search term appears in a given document — appearing more often (up to a saturation point) suggests more relevance.
Inverse Document Frequency (IDF)
How rare a term is across the whole index — common words like "the" contribute less to the score than rare, distinctive words.
Field-length normalization
Penalizes matches in very long fields relative to short ones, so a short, focused match isn't unfairly outranked by a long document that happens to mention the term once.