Chunking Strategies

The animation shows the intuition. This page owns the engineering trade-offs behind chunk boundaries, overlap, metadata inheritance, and re-indexing cost.

Chunking is the decision about what a single searchable record contains. There is no universally best chunking strategy for RAG; the right one follows the structure of the corpus:

• Fixed-size windows with overlap: unstructured or flat text where no reliable markup exists.
• Structure-aware: documents with headings, tables, lists or code, such as policies and manuals.
• Semantic chunking: long prose that shifts topic without markup, when its extra ingestion cost is justified.
• Parent-child: small chunks for precise matching, with the larger parent section returned as context.

A good chunk is small enough that its embedding and keywords describe one thing, large enough to carry the conditions and exceptions that make its statements true, and tagged precisely enough to trace back to its source. The practical method is to respect the document's own structure first, split further only when a section exceeds a token budget, add overlap sparingly, and then compare candidate strategies on real questions. Ranking the resulting chunks is the job of the retrieval pipeline.

Here is a paragraph from the refund policy, exactly as it appears in POL-REF-004:

Duplicate charges. A customer is eligible for an automatic refund when two
settled captures share the same order reference. The refund is issued within
5 business days to the original payment method. Exception: if either capture
is under an open dispute, the automatic refund is paused until the dispute
is closed, and support must not promise a refund date.

A pipeline that splits at a fixed character count happens to cut after "original payment method." The paragraph now lives in two records. Chunk 1 holds the eligibility rule and the 5-day promise. Chunk 2 begins "Exception: if either capture is under an open dispute..." A support agent asks, "Will the customer get an automatic refund for a double charge?" The search ranks chunk 1 first, because it contains nearly the words of the question. The system passes only the top result to the model, and the model answers, correctly with respect to what it saw, "Yes, within 5 business days." The exception is never retrieved, because chunk 2 on its own talks about disputes and matches nothing in the question.

The figure below follows that cut through retrieval to the model.

No component misbehaved. The embedding model, the vector index, and the language model all did their jobs on the pieces they were given. The error was introduced before any of them ran, by a boundary that separated a rule from its qualifier. It is invisible in aggregate metrics and can only be found by looking at the chunks. This is why chunking deserves design rather than a default.

A document is divided into sections, sections into passages, and passages become chunks, the records you index. A chunk has two jobs that pull in opposite directions. It must be a good match target: its text should be about one thing so that a question about that thing finds it. And it must be a good answer source: once found, it should contain enough to support an answer, including the qualifications. Tiny chunks match precisely but lack context; large chunks keep context but blur topics, so that a vector summarizing three unrelated paragraphs matches none of them well.

Size must be measured in the units that matter. Embedding models and language models consume tokens, not characters or words. The same 400-word passage can be 500 or 700 tokens depending on the tokenizer, and code, numbers, and non-English text tokenize differently from English prose. If your embedding model accepts a maximum input length, chunk sizes should be defined with that model's tokenizer, or you will silently truncate text and lose the tail of long chunks. The Context Window lesson explains why token counts, not characters, are the budget.

The simplest strategy slides a window of N tokens over the text, stepping forward by N minus the overlap. Take a 1,000-token document with a 400-token window and a 100-token overlap. The step (stride) is 300. Offsets are token positions and end-exclusive, so [0, 400) contains tokens 0 through 399.

Counting what this costs: three chunks hold 400 + 400 + 400 = 1,200 tokens for a 1,000-token document, so embedding and storage grow by 20%. Including the redundant fourth window would add another 100 tokens (1,300 total) for no benefit, so the loop should stop once a window reaches the end. The benefit is that a sentence straddling token 400 appears whole in chunk 1. The costs are duplicated storage, duplicated embedding work, and a higher chance that one passage is retrieved twice via neighboring chunks.

The weakness is that boundaries ignore meaning. A window boundary can fall in the middle of a sentence, a table row, or the example in the paragraph above. Overlap softens this only if the important context happens to fit inside it. Fixed windows are a reasonable baseline for homogeneous prose and a fine starting point for measurement, but they are rarely the end of the story.

A step up is to split on grammatical boundaries: sentences or paragraphs. This guarantees that no chunk begins or ends mid-sentence. A paragraph-based chunker packs whole paragraphs until the next one would exceed the budget. If a single paragraph is longer than the budget, it falls back to sentences, and as a last resort to a token window.

This is segmentation, not understanding. It keeps sentences intact but does not know that sentence 4 qualifies sentence 2. A paragraph is usually a coherent unit because writers group related thoughts, which is why this works well on clean prose. It fails on text where paragraphs are short fragments (chat transcripts, bulleted lists), where a single paragraph is a page long, or where the exception is placed in the next paragraph by an author who considers it a separate point.

Many documents already declare their structure: headings, subheadings, list items, tables, code blocks, HTML sections, PDF layout regions. Structure-aware chunking treats those as the primary boundaries and splits within a section only when it exceeds the token budget. Each chunk keeps its heading path, such as Refunds > Duplicate charges, which is both useful metadata and a strong retrieval signal when prepended to the chunk text before embedding.

Two examples show why structure matters. A table of deadlines has a header row (Region, Method, Days) and data rows. If a window cuts after the third row, the second chunk contains "EU, Card, 5" with no column names; its embedding means nothing, and a reader (or model) cannot tell what the 5 refers to. The fix is to treat the table as an atomic unit, or to repeat the header in every chunk. Code is similar: a function body chunked away from its signature and docstring loses the name and purpose that a query would match. Chunk code by function or class, and carry the file path and symbol name as metadata.

Semantic chunking decides boundaries by topic change rather than by markup. A common formulation embeds each sentence, compares neighboring sentences, and starts a new chunk where similarity falls below a threshold. Here is a toy with six sentences and made-up neighbor similarities:

s1 Duplicate charges are refunded automatically.
s2 The refund goes to the original payment method.
s3 Refunds usually settle within 5 business days.
s4 Our chat support is open 8am to 8pm CET.
s5 Phone support is available on weekdays.
s6 Escalations go to the support lead.

neighbor similarity:  s1-s2 0.82   s2-s3 0.79   s3-s4 0.31   s4-s5 0.77   s5-s6 0.80
threshold 0.50  ->  split between s3 and s4
chunks: [s1 s2 s3]  [s4 s5 s6]

It looks appealing but comes with real caveats. It requires embedding every sentence, which costs compute at ingestion. The threshold is data-dependent and brittle: a value that works on one corpus may over-split or under-split another, and many implementations use relative thresholds (for example, a percentile of the similarity drops in the document) to cope. Adjacent similarity is noisy for short sentences. And it can behave inconsistently: a small edit changes embeddings and therefore boundaries, which in turn changes chunk IDs and forces more re-indexing. Note also that semantic chunking only uses an embedding model to choose boundaries; it is not the same thing as generating the embeddings you store, which is covered in Embeddings Explained. Treat it as an option to evaluate against simpler baselines, not a default upgrade.

Drag the threshold below to see how the toy boundaries move; the five similarities are the ones above.

The tension between precise matching and sufficient context has a clean structural answer: index small pieces but return larger ones. In parent–child (or small-to-big) retrieval, each section is a parent, and its sentences or small passages are children. Only children are embedded and searched, so matches are precise. When a child is retrieved, the system returns its parent, which includes the surrounding conditions.

{ "chunk_id": "POL-REF-004#v4#c0007.2", "parent_id": "POL-REF-004#v4#c0007",
  "source_id": "POL-REF-004", "version": 4, "tenant_id": "acme-eu",
  "acl": ["support-agent"], "text": "The refund is issued within 5 business days ..." }

The parent must carry exactly the same version and authorization boundaries as its children. If a parent section merges content with different ACLs, returning it can disclose a child the user could not have searched for. If the parent spans two versions, you may mix a retired sentence with a current one. Design parents so that every parent lies within one document version and one ACL scope, and re-check authorization on the parent at expansion time.

Overlap guards against a boundary falling in the wrong place, and the numbers above show its price: 20% more stored tokens in the short toy document, and for a long document a 100-token overlap on a 400-token window approaches 400 / 300, about 33% more (a 200-token overlap on a 600-token window would be 600 / 400, 50% more). Beyond storage and embedding cost, overlap changes retrieval behavior. Neighboring chunks share text, so both match the same query and fill the top-k with near duplicates, which then also waste prompt tokens and can over-weight the repeated passage.

Change the document length, window and overlap to recompute the cost.

You can keep the benefits and drop the duplicates after retrieval. If chunk 1 [300, 700) and chunk 2 [600, 1000) of the same document and version are both retrieved, merge them into the span [300, 1000) and present one passage. Because chunks carry offsets and document IDs, this is a mechanical operation: group by source and version, sort by start offset, and merge ranges that overlap or touch. Overlap is a hedge against bad boundaries; with structure-aware chunking that respects sections, you often need much less of it.

A chunk without provenance cannot be updated, deleted, cited, or authorized. The record below is the minimum that supports those operations:

{
  "chunk_id": "POL-REF-004#v4#c0007",   // deterministic: doc + version + ordinal
  "doc_id": "POL-REF-004",
  "version": 4,
  "heading_path": ["Refunds", "Duplicate charges"],
  "start_offset": 5120,                  // characters in the cleaned source text, end-exclusive
  "end_offset": 5894,
  "effective_date": "2026-10-01",
  "tenant_id": "acme-eu",
  "acl": ["support-agent", "support-lead"],
  "text": "Duplicate charges. A customer is eligible ..."
}

The stable mapping to a source span is what lets a citation highlight the exact passage and lets an update job find every record belonging to a document. If chunk IDs are derived only from position, an edit near the top of a document shifts every later ordinal and makes every chunk look new; deriving IDs from a hash of normalized text plus the heading path, or re-chunking only changed sections, limits that churn.

Many "chunking problems" are parsing problems. A two-column PDF extracted in naive reading order interleaves lines from both columns, and no chunk size can rescue sentences that are scrambled. Footnotes get inserted in the middle of paragraphs. A scanned document yields nothing without OCR, and OCR errors ("5" read as "S") pass straight into the index. HTML pages carry navigation, cookie banners, and related-links lists; indexed as-is, they place the same boilerplate in thousands of chunks and pollute matches. Tables often flatten into a sequence of numbers that has lost its columns. Repositories need syntax-aware splitting. FAQs are naturally question-and-answer pairs and should be chunked that way, with the question included. Transcripts need speaker labels and timestamps preserved so a hit can link back to a moment. Multilingual text needs a tokenizer and embedding model that actually cover the languages used.

Before tuning sizes, read a sample of the chunks your pipeline produces for each document type. It is the cheapest and most informative evaluation you can do.

To compare strategies you need questions whose answers stress chunking. Build a small labeled set that includes: questions answered by one paragraph; questions that need two sections combined; questions whose answer depends on an exception placed near the rule; and questions about values in tables. For each, record which passages (by source ID and offsets) are required. Then index the same corpus under at least three configurations, for example:

1. 1

   Config A: fixed 200-token windows, no overlap

   The baseline. Small and precise, likely to separate rules from exceptions.
2. 2

   Config B: fixed 400-token windows, 80-token overlap

   Larger units with a hedge against bad boundaries.
3. 3

   Config C: heading-aware sections up to 500 tokens, children of 150 tokens returned as parents

   Structure-aware with parent expansion.

Measure for each: whether the required passages are in the retrieved set (recall), whether the final answer is supported by what was supplied, how many duplicate or overlapping passages appear, the number of tokens passed to the model, index size, and ingestion and query latency. Do not expect one configuration to win on every measure. Larger chunks often raise coverage and token use together; the decision is a trade-off between answer quality and cost on your questions. Resist copying numbers from blog posts. Your corpus's structure, tokenizer, and question types decide the outcome.

This pseudocode implements heading-aware chunking with a token budget, overlap, metadata, stable IDs, and a way to handle updates and deletes. Functions such as parse_sections and count_tokens are assumed; they are not a specific library's API.

MAX_TOKENS, OVERLAP = 400, 60

def chunk_document(doc):                           # doc: id, version, tenant, acl, text
    chunks = []
    for sec in parse_sections(doc.text):           # each has heading_path, start, text
        units = split_into_sentences(sec.text)     # never split inside a sentence
        window, tokens = [], 0
        for u in units:
            n = count_tokens(u.text)               # use the embedding model's tokenizer
            if window and tokens + n > MAX_TOKENS:
                chunks.append(make_chunk(doc, sec, window))
                window = carry_overlap(window, OVERLAP)   # last sentences up to OVERLAP tokens
                tokens = sum(count_tokens(w.text) for w in window)
            window.append(u); tokens += n
        if window:
            chunks.append(make_chunk(doc, sec, window))
    return chunks

def make_chunk(doc, sec, units):
    start, end = units[0].start, units[-1].end
    text = " > ".join(sec.heading_path) + "\n" + doc.text[start:end]
    key = hash(f"{doc.id}|{doc.version}|{sec.heading_path}|{start}")
    return {"chunk_id": f"{doc.id}#v{doc.version}#{key[:10]}",
            "doc_id": doc.id, "version": doc.version, "tenant_id": doc.tenant,
            "acl": doc.acl, "heading_path": sec.heading_path,
            "start_offset": start, "end_offset": end, "text": text}

def sync(doc):                                      # update and delete handling
    new = {c["chunk_id"]: c for c in chunk_document(doc)}
    old = set(index.ids_for(doc.id))
    index.upsert(new.values())                     # idempotent
    index.delete(old - set(new))                   # stale pieces of earlier versions
def on_document_deleted(doc_id):
    index.delete(index.ids_for(doc_id))            # tombstone, then purge

Return to the duplicate-charge paragraph. Under the new pipeline, the section "Refunds > Duplicate charges" is about 85 tokens, far below the 400-token budget, so it becomes a single chunk that begins with its heading path. The sentence beginning "Exception:" cannot be separated from the rule because the splitter only breaks at the budget, and the budget was not reached.

Query: "Will the customer get an automatic refund for a double charge?"

top-1: POL-REF-004#v4#a91c03e7f2   score 0.83
       "Refunds > Duplicate charges
        Duplicate charges. A customer is eligible for an automatic refund when two
        settled captures share the same order reference. The refund is issued within
        5 business days ... Exception: if either capture is under an open dispute,
        the automatic refund is paused until the dispute is closed, and support
        must not promise a refund date."

With the exception in the same retrieved record, the model can answer that an automatic refund applies unless either capture is under an open dispute, and that support should not promise a date in that case. The fix did not use a larger model or a cleverer search: it changed which text travels together. The exception could still be lost in a different failure (for example, if the section were over budget and split mid-way), which is why the evaluation set must include exception questions and why parent expansion is a useful safety net.

The chunking determines what embeddings represent and what the retrieval pipeline can ever return, so verify it before tuning anything downstream.

Once documents are cut into chunks, each chunk must be turned into something a computer can compare by meaning.