What Is Chunking and How Does It Work?

It involves splitting a document into smaller chunks so each can be embedded, searched, and fed to the model. The model does not read the entire document, only the retrieved chunks, so how you split matters.

It depends on the content, but chunks are typically several hundred tokens long — big enough to contain a meaningful idea but small enough not to become imprecise. The best size is determined by experimenting with retrieval results on your own documents.

Overlap repeats a small bit of text at the boundary of each chunk so an idea is not divided into two pieces. It allows retrieval of complete ideas rather than fragmented ones.

There is no single best approach. Chunking based on structure or semantics — dividing by the document's structure or its meaning — generally performs better than naive fixed-size approaches, but it requires additional resources. The right choice depends on your documents and how much quality matters.

Because chunks are precisely what a search looks for and retrieves. The better organized and sized the chunks are, the easier it is to match them to a query and provide focused context, while poor chunks retrieve irrelevant fragments.