Chunking Strategies for RAG

Why chunking decides retrieval quality

An embedding model turns a whole chunk into one vector, so the chunk is the unit the retriever reasons about. Oversized chunks blend unrelated topics into one vector and the retriever cannot tell which part matched. Undersized chunks lose the surrounding sentences that make a fact interpretable. Chunking also controls cost: more chunks mean more embedding calls and a larger index, while fewer, larger chunks burn more prompt tokens per retrieval.

There is no universal chunk size. The right setting depends on the document structure, the embedding model, and the question types. What transfers across projects is a method: pick a strategy, build a labeled set of questions, and measure recall at k while changing one variable at a time.

Fixed-size splitting

The simplest strategy cuts text into blocks of N characters with an overlap of M characters. It needs no parsing and behaves predictably, but it splits mid-sentence and mid-paragraph, which hurts both embedding quality and readability of retrieved context.

```python

from langchain_text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(

chunk_size=1000,

chunk_overlap=100,

separator="\n\n",

)

chunks = splitter.split_text(document_text)

```

Fixed-size splitting is fine for uniform prose where paragraph boundaries do not matter much, and it is a reasonable baseline to beat before reaching for fancier methods.

Recursive character splitting

Recursive splitting tries a list of separators in order and only falls back to a finer separator when a piece is still too large. The default order is paragraphs, then lines, then spaces, then characters. This keeps natural language boundaries intact whenever possible and is the default choice for general RAG work.

```python

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(

chunk_size=1000,

chunk_overlap=200,

length_function=len,

separators=["\n\n", "\n", " ", ""],

)

chunks = splitter.split_text(document_text)

```

A common production starting point is chunk_size around 1000 characters with 200 characters of overlap, then tuning on measured recall. Overlap of roughly 10 to 20 percent of the chunk size protects facts that sit on a boundary from being split across two chunks.

Measuring in tokens, not characters

Character counts are a proxy. Embedding models and language models count tokens, and token density varies with language and content: code and dense technical prose pack more tokens per character than plain English. Size chunks with a tokenizer to keep prompt math honest.

```python

import tiktoken

from langchain_text_splitters import RecursiveCharacterTextSplitter

encoding = tiktoken.get_encoding("cl100k_base")

splitter = RecursiveCharacterTextSplitter(

chunk_size=500, # tokens, not characters

chunk_overlap=50,

length_function=lambda text: len(encoding.encode(text)),

)

chunks = splitter.split_text(document_text)

```

The cl100k_base encoding matches the tokenizer family used by many current models. Token-based sizing keeps every chunk under the limits the downstream model actually enforces.

Structure-aware splitting

When documents have real structure, split on it. Markdown documents split on headers so each chunk carries its section context, and the header path becomes metadata that improves both filtering and prompt citations.

```python

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers_to_split_on = [

("#", "title"),

("##", "section"),

("###", "subsection"),

]

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

chunks = splitter.split_text(markdown_text)

# each chunk's metadata includes the header path, e.g. {"section": "Billing"}

```

For source code, the recursive splitter supports language-aware separators that respect function and class boundaries:

```python

splitter = RecursiveCharacterTextSplitter.from_language(

language="python",

chunk_size=1500,

chunk_overlap=150,

)

```

HTML documents get the same treatment with header-based splitters that walk h1 through h3 tags. The principle holds across formats: never let a chunk boundary land where the document's own structure says the thought continues.

Parent document retrieval

Small chunks retrieve precisely but read poorly. Parent document retrieval resolves the tension: index small child chunks for search, and at query time return the enclosing parent chunk (a section or whole page) to the model. Retrieval stays precise, generation gets context.

Implementation is straightforward. Store each child chunk with a parent_id in its metadata, keep parents in a document store keyed by id, and after the vector search fetch the unique parents of the top children before building the prompt. The cost is a second lookup per query and a larger prompt, which is usually worth it for dense reference material.

Enriching chunks with metadata

A chunk stripped of its origin is hard for the model to cite and hard for you to filter. Attach source file, section path, page number, and date to every chunk at index time. Two practices pay off immediately: prepend a short context header to ambiguous chunks (for example, the document title and section) so the embedding captures what the raw text leaves implicit, and add the same metadata as filterable fields in the vector store so queries can scope to a date range or a single source.

Tuning loop

Change one variable, measure, keep what moves recall. The order that usually pays: fix obviously broken boundaries first (mid-sentence splits, tables torn apart), then set chunk size against a labeled question set, then tune overlap, then add structure-aware splitting for the document types that dominate the corpus, and only then consider parent-document retrieval or hypothetical-question indexing. Chunking is the cheapest part of the pipeline to experiment with and one of the highest-leverage.