What Is Chunking and How Does It Work?

A language model cannot read an entire knowledge base at once, and a retrieval system cannot usefully search one giant blob of text. Both need the material broken into smaller, meaningful pieces. Chunking is that step, and though it sounds trivial, it quietly determines how well a retrieval system finds the right information.

Split a document the wrong way and a single idea is cut in half, or unrelated topics are jammed together, and retrieval returns pieces that do not quite answer the question. Split it well and each chunk is a coherent unit that can be found and used. That is why chunking is one of the most important and underrated parts of a RAG pipeline.

This guide explains what chunking is, how it works, the main strategies, and how to do it well.

What is chunking?

Chunking is the process of dividing a long document into smaller segments, called chunks, that can each be embedded, stored, and retrieved independently. Each chunk is meant to be a self-contained piece of text, large enough to carry meaning but small enough to be precise and to fit alongside other chunks in a model's context.

Think of breaking a long book into index cards. One enormous card is useless for finding anything, and cards cut mid-sentence are confusing. The goal is cards that each capture one coherent idea, so you can quickly pull the few that answer a question. Chunking is making those cards from your documents.

The key point is that chunks are the unit of retrieval. The model never sees the whole document; it sees the chunks that were retrieved, so how you form them shapes what the model can work with.

How does chunking work?

Chunking is the first stage of preparing documents for retrieval.

• Start with the document. A long piece of text, such as a manual, article, or transcript, is the input.
• Split into chunks. The text is divided by size, structure, or meaning into smaller segments, often with some overlap.
• Embed each chunk. An embedding model turns every chunk into a vector that captures its meaning.
• Store the chunks. The vectors and their text are saved in a vector database for search.
• Retrieve at query time. When a question arrives, the most relevant chunks are fetched and passed to the model.

Overlap between chunks, repeating a little text at the boundaries, helps avoid cutting an idea cleanly in two. The right chunk size and overlap depend on the content and are usually found by testing.

Fixed-size vs. semantic vs. recursive chunking

There are a few main strategies. Fixed-size chunking splits the text every N tokens with some overlap; it is simple and fast but can cut through the middle of an idea. Semantic chunking splits on meaning, keeping related sentences together and breaking at topic shifts, which produces more coherent chunks at more cost. Recursive or structural chunking splits along the document's own structure, by sections and paragraphs first, then by size as needed, which respects the natural boundaries of the text.

In practice, structural and semantic approaches usually beat naive fixed-size splitting, but they are more work. The right choice depends on how structured your documents are and how much retrieval quality matters.

Why does chunking matter?

• Retrieval quality. Coherent chunks are easier to match to a query, so the right information is found.
• Precision. Smaller chunks return focused results instead of burying the answer in unrelated text.
• Context fit. Right-sized chunks let several relevant pieces fit together in the model's window.
• Grounding. Better retrieval means the model answers from the right source, reducing hallucination.
• Cost. Retrieving tight, relevant chunks instead of whole documents saves tokens and money.

What are the pitfalls?

• Chunks too large. Big chunks dilute relevance and waste context on unrelated text.
• Chunks too small. Tiny chunks lose the surrounding context needed to make sense.
• Bad boundaries. Splitting mid-idea produces chunks that are hard to match and use.
• One size fits all. Different document types often need different chunking strategies.

When should you care about chunking?

You should care about chunking whenever you build a retrieval system or RAG pipeline over your own documents. Because retrieval quality rests on it, tuning chunk size, overlap, and strategy is one of the highest-leverage things you can do to improve answers.

If you are not doing retrieval, chunking does not apply. But for any system that searches a knowledge base to ground a model, how you chunk is a decision worth getting right rather than defaulting.

Best practices for chunking

• Split along natural boundaries, like paragraphs and sections, before falling back to size.
• Add a small overlap between chunks so ideas are not cut cleanly in two.
• Tune chunk size to your content and test retrieval quality, rather than guessing once.
• Keep related metadata, like the source and section, with each chunk for filtering and citation.
• Match the strategy to the document type instead of using one approach everywhere.
• Measure retrieval quality end to end, since chunking is only good if it improves answers.

Conclusion

Chunking splits long documents into smaller, coherent pieces so they can be embedded, retrieved, and fed to a model. Because chunks are the unit of retrieval, how you form them, their size, overlap, and boundaries, directly shapes how well a RAG system finds and uses the right information. It is a foundational and often underrated step in building retrieval systems.

TrueFoundry helps teams build and serve retrieval-augmented systems on their own infrastructure, with embedding, retrieval, and generation managed in one place behind a gateway. Whatever model sits at the end of the pipeline, good chunking upstream is much of what makes its answers accurate and grounded.