Chunking is the decision most RAG tutorials treat as a default setting and most production systems end up revisiting. How you cut documents determines what the retriever can possibly find. A perfect embedding model cannot return an answer that was sliced in half across two chunks, and it cannot make a 4,000-word chunk about five topics match a query about one.

What a Chunk Has to Do

A good chunk is a unit that can answer a question on its own, or nearly so. It needs enough context that a reader dropped into it understands the subject, and it needs to be focused enough that its embedding represents one idea rather than an average of many. Those two goals pull in opposite directions, which is why there is no universal size.

Chunk size also interacts with the rest of the pipeline: your embedding model has an input limit, your generator has a context window you are paying to fill, and retrieving several oversized chunks crowds out room for other evidence. Check your embedding model’s current documentation for its input limits instead of assuming.

Fixed-Size Chunking

The simplest method: split every N tokens or characters, often with some overlap between neighbors. It is predictable, fast, and easy to reason about. The downside is that it ignores meaning, so it will cut sentences, tables, and code blocks in awkward places.

It is a reasonable baseline, and a common trap is never moving beyond it. If your content is uniform prose, it may be perfectly adequate. If it has headings, lists, or tables, you can usually do better.

Structure-Aware Chunking

Split along the document’s own boundaries: headings, paragraphs, list items, sections of a contract, functions in source code. This tends to keep related text together and gives chunks natural titles you can store as metadata. Markdown, HTML, and many PDFs expose enough structure to do this reliably, though PDF extraction quality varies a lot and deserves manual inspection.

When a section is too long, fall back to splitting it by paragraph or sentence, and when sections are tiny, merge neighbors up to a target size.

Semantic Chunking

This approach places boundaries where the topic shifts, often by comparing embeddings of adjacent sentences and cutting where similarity drops. It can produce coherent chunks from unstructured text, at the cost of extra compute at ingestion and more tuning. It is worth trying when documents are long, flowing prose without headings, such as transcripts. For well-structured material, simple structural splitting often matches it.

Overlap and Context Carry-Over

Overlap repeats a slice of text at chunk edges so that a sentence near a boundary appears whole in at least one chunk. Modest overlap helps; heavy overlap inflates your index and returns near-duplicate results. A more useful technique is prepending context: attach the document title and section heading to each chunk before embedding, so a chunk reading “the limit is 30 days” still knows it belongs to a returns policy.

Another pattern is retrieving small chunks for precision and then expanding to the surrounding section when passing text to the generator.

Comparing the Approaches

Strategy	Strength	Weakness	Good fit
Fixed-size with overlap	Simple, predictable	Cuts mid-thought	Uniform prose, first baseline
Structure-aware	Keeps sections intact, useful metadata	Needs clean structure	Docs, wikis, policies, code
Semantic	Follows topic shifts	Extra compute, tuning	Transcripts, long unstructured text
Small chunk, expand on retrieval	Precise match, rich context	More moving parts	Dense reference material

Special Content Types

Tables are the usual casualty. Splitting a table row away from its header leaves numbers with no meaning, so keep headers with every table chunk or convert rows into short sentences. Code works best split at function or class level. Chat logs and emails often work as one chunk per message or thread segment. Treat each content type as its own ingestion path rather than forcing one splitter on everything.

A Hypothetical Example

Imagine a support knowledge base where each article has a title, a short intro, and several how-to sections. The team starts with fixed chunks and notices that answers to “how do I reset X” keep returning the intro paragraph instead of the steps, because the intro mentions the same terms densely. They switch to splitting on headings, prepend the article title to each chunk, and the steps section starts ranking first. No change to the embedding model or database was needed.

How to Decide With Evidence

Do not pick a size by feel. Build a small evaluation set: twenty to fifty real questions with known source passages, written from the perspective of actual users. Run each chunking variant through the same retriever and check how often the right passage appears in the top results. Change one variable at a time: size, overlap, or strategy. Re-embedding the corpus is the expensive part, so test on a representative subset first.

Look at failures by reading the retrieved chunks yourself. Patterns show up quickly: answers split across chunks, chunks dominated by boilerplate, or headings missing.

Frequently Asked Questions

What chunk size should I start with? Start with a few hundred tokens as a baseline, then test smaller and larger. The right size depends on your content and your embedding model, so measure rather than copy a number.

Does more overlap always help? No. Beyond a modest amount it mostly adds duplicates and cost. Prepending titles and headings usually helps more than extra overlap.

Should I strip headers, footers, and navigation text? Yes. Repeated boilerplate pollutes embeddings and ranks highly for unrelated queries. Clean it before chunking.

Do I need to re-chunk when I change embedding models? You must re-embed, and it is a good moment to revisit chunking, since the new model may handle different lengths better.

Can I use different strategies for different document types? Yes, and you probably should. Store the strategy used as metadata so you can compare results by source type.

Next Step

Write down fifteen real user questions and the exact passages that answer them, then check whether your current chunks keep each answer in one piece. The questions where it fails tell you which strategy to try next.