Chunking Strategies for RAG

Document chunking is one of the most important design decisions in a retrieval-augmented generation (RAG) system. Before documents can be embedded and retrieved, large sources such as web pages, PDFs, documentation, manuals, and knowledge-base articles are usually divided into smaller pieces called chunks.

The quality of these chunks directly affects retrieval. If a chunk is too small, it may lose the context needed to understand its meaning. If it is too large, it may contain several unrelated topics and become difficult to retrieve precisely. A good chunking strategy creates units that are focused enough for accurate retrieval while still containing enough information to be useful to the language model.

There is no single chunking strategy that works best for every RAG application. The right approach depends on the structure of your documents, the type of questions users ask, the embedding model, the retrieval method, the context window, and the amount of context required to answer typical queries.

What Is Chunking in RAG?

Chunking is the process of splitting a larger document into smaller pieces before indexing it for retrieval. Each chunk can be stored with its text, embedding, metadata, and relationship to the original document.

For example, a 30-page technical manual might contain dozens of sections covering installation, configuration, authentication, troubleshooting, and API usage. Instead of creating one embedding for the entire manual, a RAG system can divide it into focused chunks and create an embedding for each one.

When a user asks a question, the retrieval system can then identify the chunks most relevant to that question instead of retrieving the entire document.

Why Chunking Matters

Embeddings represent the meaning of their input. If one chunk contains several unrelated topics, its embedding represents a mixture of those topics. This can make the chunk less precise during vector retrieval.

The opposite problem occurs when chunks are too small. A short sentence may contain a reference to an option, parameter, or concept without explaining what it refers to. The resulting embedding may lack enough information to match the user's query.

The Goal of RAG Chunking

The goal is not simply to create chunks of a particular number of characters or tokens. The goal is to create retrieval units that represent coherent pieces of information.

A useful chunk should ideally answer one part of a question, explain one concept, describe one procedure, or contain a logically connected group of information. It should also contain enough surrounding context that it remains understandable when retrieved independently.

Chunk Size

Chunk size determines how much text is placed into each retrieval unit. It can be measured in characters, words, sentences, or tokens, although tokens are often the most useful unit when working with language models.

There is no universal ideal chunk size. A short FAQ answer may work well as one chunk, while a long technical section may need to be divided into several smaller chunks.

The appropriate size should be determined by the structure of the content and the questions the system needs to answer. Testing different sizes against real retrieval queries is usually more reliable than following a fixed recommendation.

Chunk Size vs Retrieval Precision

Smaller chunks generally provide more precise retrieval because each vector represents a narrower piece of information. However, they can lose relationships between statements that belong together.

Larger chunks preserve more context but can contain several concepts. A query may match one small part of a large chunk, causing the entire chunk to be considered relevant even though most of its content is unrelated.

The trade-off can be summarized as precision versus context. Good chunking attempts to find a balance between these two requirements.

Chunk Overlap

Chunk overlap means repeating a portion of one chunk at the beginning or end of the next chunk. Overlap is used to reduce the risk that an important piece of information is separated from the context that explains it.

For example, if a paragraph explaining an API is split across two chunks, a small overlap can ensure that both chunks contain some of the surrounding explanation.

Overlap can improve retrieval continuity, but excessive overlap creates more indexed content and can cause multiple retrieved chunks to contain nearly identical information.

Fixed-Size Chunking

Fixed-size chunking divides a document into pieces of approximately the same size. The boundary can be based on characters, words, or tokens.

Document
    ↓
    ├── Chunk 1
    ├── Chunk 2
    ├── Chunk 3
    └── Chunk 4

Fixed-size chunking is simple, predictable, and inexpensive. It can be a good baseline for experimentation and for documents with little structural information.

Its main weakness is that boundaries do not necessarily correspond to meaningful concepts. A paragraph, sentence, table, or code example can be split in the middle.

Fixed-Size Chunking with Overlap

A common variation combines fixed-size chunks with overlap. Instead of starting each chunk exactly where the previous one ended, the next chunk begins slightly earlier.

Chunk 1
   ↓
Chunk 2
   ↓
Chunk 3

Each chunk overlaps with the previous one

This approach reduces the probability that a relevant statement will be separated from the context around it. The amount of overlap should remain controlled because large overlaps increase storage and retrieval redundancy.

Sentence-Based Chunking

Sentence-based chunking splits documents at sentence boundaries instead of arbitrary character positions. Multiple sentences can then be grouped together until the desired chunk size is reached.

This usually produces more coherent chunks than fixed character splitting because complete sentences remain intact.

Sentence-based chunking can work well for articles, documentation, FAQs, and ordinary prose. However, sentence boundaries alone do not always correspond to complete topics, so additional structural rules may still be useful.

Paragraph-Based Chunking

Paragraph-based chunking treats paragraphs as natural retrieval units. Related sentences that were intentionally grouped by the author remain together.

This approach is particularly useful when source documents have well-written paragraphs and clear formatting. However, paragraphs can vary significantly in length, and a very long paragraph may still need to be divided.

Recursive Chunking

Recursive chunking uses a hierarchy of separators to split content while attempting to preserve larger structural units whenever possible.

For example, a system might first try to split by headings, then paragraphs, then sentences, and finally smaller units if a section is still too large.

Document
  -> Sections
     -> Paragraphs
        -> Sentences
           -> Smaller units if necessary

The advantage of recursive splitting is that it adapts to the document structure while still respecting a maximum chunk size.

Structure-Aware Chunking

Structure-aware chunking uses the actual organization of a document to determine chunk boundaries. Instead of treating a document as a plain string, the ingestion process recognizes headings, sections, lists, tables, code blocks, and other structural elements.

This is particularly valuable for technical documentation. A section titled "Authentication" should generally remain associated with its explanatory content, while a code example should not be arbitrarily separated from the explanation that describes it.

Structure-aware chunking is often a strong choice when the source format provides reliable structural information.

Semantic Chunking

Semantic chunking attempts to divide text according to changes in meaning rather than fixed sizes or simple formatting boundaries.

A semantic chunker can analyze neighboring sentences or passages and detect when the topic changes significantly. Sentences discussing the same concept can remain together even when their lengths vary.

Semantic chunking can produce highly coherent retrieval units, but it usually requires additional computation during document ingestion and can be more difficult to tune and reproduce than simpler approaches.

Topic-Based Chunking

Topic-based chunking groups content according to a specific subject or concept. It is closely related to semantic chunking but can use explicit document structure, classifiers, or topic detection to determine boundaries.

For example, a product manual might contain separate chunks for installation, configuration, authentication, troubleshooting, and API usage. Each chunk can be indexed with metadata describing its topic.

Hierarchical Chunking

Hierarchical chunking represents documents at multiple levels. A large document can contain sections, subsections, paragraphs, and smaller chunks, with relationships preserved between those levels.

This allows retrieval to happen at different granularities. A system can first identify a relevant section and then retrieve a more specific child chunk from that section.

Document
  ├── Section A
  │   ├── Chunk A1
  │   └── Chunk A2
  └── Section B
      ├── Chunk B1
      └── Chunk B2

Hierarchical retrieval can provide precise matching while allowing the application to recover broader context when needed.

Parent-Child Chunking

Parent-child chunking is a practical form of hierarchical retrieval. Small child chunks are used for precise search, while each child is associated with a larger parent section or document.

When a child chunk matches the query, the application can return the child itself, its parent, or a combination of both depending on the amount of context required.

This strategy can be particularly effective for documentation and long-form content where small passages need broader context to be useful.

Contextual Chunking

Contextual chunking adds information from the surrounding document structure to each chunk. For example, the chunk text can be associated with the document title and heading hierarchy.

A chunk containing "This setting controls the timeout" is ambiguous by itself. If it is associated with a heading such as "HTTP Client Configuration", the retrieval representation becomes more informative.

Context can be stored separately as metadata or incorporated into the text representation used for embeddings, depending on the retrieval architecture.

Chunking for Technical Documentation

Technical documentation requires special care because it often combines prose, code, tables, configuration examples, API references, and version-specific information.

A good strategy should preserve relationships between an API description and its parameters, a configuration option and its explanation, or a code example and the instructions surrounding it.

• Preserve document titles and heading paths.
• Keep related paragraphs together.
• Avoid splitting code examples unnecessarily.
• Preserve programming language metadata.
• Keep table headers associated with table values.
• Store product and version information.
• Associate examples with the concepts they demonstrate.

Chunking Code

Source code and code examples often require different chunking rules from ordinary prose. Splitting code in the middle of a function or class can produce fragments that are difficult to understand and retrieve.

For source repositories, syntax-aware chunking can use functions, classes, methods, modules, or other language constructs as boundaries. For documentation, code blocks can often be kept intact and associated with the explanatory text around them.

The appropriate level depends on the retrieval task. A developer asking about one function may benefit from a function-level chunk, while a question about how several functions interact may require a larger parent context.

Chunking Tables

Tables should not automatically be treated as ordinary paragraphs. Their meaning often depends on column headers and relationships between rows and values.

If a table is split incorrectly, a retrieved chunk may contain values without the headers that explain them. Preserve the table structure or transform it into a representation where each value remains associated with the relevant field names.

Chunking PDFs

PDF documents can be difficult to chunk because the extracted text may not preserve the original logical structure. Headers, footers, columns, tables, page numbers, and reading order can become mixed together during extraction.

Before chunking a PDF, perform document parsing and cleanup. Remove repeated headers and footers where appropriate, reconstruct logical sections, preserve page and section metadata, and handle tables separately when they contain important information.

Chunking Web Pages

Web pages often contain large amounts of content that are not part of the main article, such as navigation menus, cookie notices, advertisements, related links, and repeated site elements.

Extract the primary content before chunking whenever possible. Preserve the page title, headings, canonical URL, publication information, and other metadata that can help identify the source.

Chunking FAQs

Frequently asked questions often have a natural chunk boundary: each question and its answer can form one retrieval unit.

This structure is particularly effective because a user's query often resembles the question directly. If an answer is very long, it can still be divided into smaller sections while retaining the original question as contextual metadata.

Chunking Structured Data

Structured data such as JSON, XML, CSV, or database records may not benefit from ordinary prose chunking. Splitting a structured object arbitrarily can remove relationships between fields.

For structured sources, consider using record-level chunks, field-aware representations, or specialized retrieval methods. The correct strategy depends on the questions users need to ask about the data.

Chunking and Embeddings

Every chunk that participates in semantic retrieval is typically converted into an embedding. The embedding represents the information contained in that chunk, so chunk boundaries directly influence what the vector represents.

If a chunk contains one coherent concept, its embedding can represent that concept more clearly. If it contains several unrelated topics, the vector may become less specific.

This is why chunking should be considered part of embedding design rather than an independent preprocessing detail.

Chunking and Keyword Search

Chunking also affects lexical retrieval. Smaller chunks can make exact term matches more focused, while larger chunks provide more surrounding context.

In hybrid search systems, chunking should therefore work reasonably well for both keyword and semantic retrieval. A structure-aware strategy can often provide a good balance.

Chunking and Hybrid Search

Hybrid search combines lexical and semantic retrieval, so chunks need to work for both types of matching. Exact identifiers may be found through keyword search, while semantic embeddings handle conceptual queries.

Preserving titles, headings, identifiers, and other exact terms inside or alongside chunks can improve both retrieval signals.

Chunking and Reranking

Rerankers can improve the ordering of retrieved chunks, but they cannot fix poorly constructed chunks. If a chunk combines several unrelated topics or lacks the context needed to interpret a statement, a better ranking model may still produce a poor final context.

A strong pipeline therefore treats chunking as the foundation, retrieval as candidate generation, and reranking as a later precision improvement.

How Chunk Size Affects Context Windows

The final chunks sent to the LLM consume context-window capacity. Very large chunks can quickly fill the context with irrelevant material, while very small chunks may require retrieving many separate pieces to reconstruct an answer.

Good chunking can reduce this problem by creating focused units that contain a high concentration of relevant information.

Chunking and Overlap Trade-Offs

Overlap should be treated as a tuning parameter rather than a requirement. Some structure-aware chunking strategies may need little or no overlap because boundaries already follow meaningful sections.

Adaptive Chunking

Adaptive chunking uses different rules for different parts of a document. For example, ordinary prose may be split into paragraph-based chunks, code blocks may remain intact, and large sections may be recursively divided.

This approach recognizes that documents are heterogeneous. One universal rule may work adequately as a baseline but can be suboptimal for complex sources.

Agentic or Query-Aware Chunking

Some advanced systems consider expected query types when designing document representations. Instead of optimizing chunks only according to the source structure, the system can consider how users are likely to search the content.

For example, an API reference may benefit from small function-level chunks because users frequently ask about individual methods. A policy document may benefit from larger section-level chunks because questions often require several related rules.

Query-aware approaches can be powerful but add complexity. They are most useful when the application has enough real query data to justify specialized optimization.

Document-Level vs Chunk-Level Retrieval

Some RAG systems retrieve entire documents, while others retrieve individual chunks. Chunk-level retrieval generally provides greater precision, but document-level retrieval can preserve more context.

A hybrid approach is often useful. The system can retrieve a relevant chunk and then use its relationship to the parent document or section to provide additional context.

Hierarchical Retrieval Strategy

A hierarchical strategy can use several retrieval stages. The first stage identifies relevant documents or sections, and the next stage searches within those candidates for more precise chunks.

Query
  ↓
Find relevant documents
  ↓
Find relevant sections
  ↓
Find relevant chunks
  ↓
Rerank candidates
  ↓
Build final context

This can improve scalability and context selection for large document collections, although it introduces additional retrieval logic.

How to Choose a Chunking Strategy

The best strategy depends primarily on the structure of your data and the questions users ask. Start with a simple baseline and compare it against more advanced strategies using real evaluation queries.

A Practical Chunking Workflow

A practical ingestion pipeline can apply several techniques in sequence rather than relying on one splitter.

• Extract the primary document content.
• Remove irrelevant boilerplate and duplicates.
• Parse the document structure.
• Preserve titles and heading hierarchy.
• Identify code blocks, tables, lists, and other special structures.
• Split large sections using meaningful boundaries.
• Apply a maximum size when necessary.
• Add controlled overlap where useful.
• Attach metadata to each chunk.
• Generate embeddings for the final chunks.
• Store parent-child or section relationships when useful.
• Evaluate retrieval quality against representative queries.

Example Chunk Metadata

A chunk should usually contain more than just raw text. Metadata can preserve information needed for filtering, ranking, display, citations, and context reconstruction.

{
  "documentId": "docs-123",
  "title": "API Authentication Guide",
  "section": "API Keys",
  "version": "v2",
  "language": "en",
  "chunkIndex": 7,
  "parentId": "section-4",
  "text": "Your API key should be stored..."
}

The exact metadata schema depends on the application. The important principle is to preserve enough information to identify where a chunk came from and how it relates to the original source.

How to Test Different Chunking Strategies

Do not judge chunking quality only by inspecting a few examples. Build an evaluation dataset containing realistic questions and the documents or chunks that should answer them.

Run the same queries against different chunking configurations. Compare retrieval metrics such as Recall@K, Precision@K, Mean Reciprocal Rank, and Normalized Discounted Cumulative Gain.

Also evaluate the generated answers. A chunking strategy that improves retrieval metrics but produces worse final answers may not be a useful improvement for the application.

Diagnosing Chunking Problems

When retrieval fails, inspect the actual chunks rather than looking only at the final answer. Many retrieval failures can be traced directly to chunk boundaries.

• The answer is split across two unrelated chunks.
• A chunk contains a reference without the information it refers to.
• A heading is separated from its section.
• A code example is separated from its explanation.
• A table loses its headers.
• Several unrelated topics appear in one chunk.
• Many retrieved chunks contain nearly identical text.
• The relevant passage exists but is too small to retrieve reliably.

Common Chunking Mistakes

• Using one fixed chunk size for every document type.
• Splitting documents without considering their structure.
• Using excessive overlap.
• Ignoring headings and section relationships.
• Splitting code examples arbitrarily.
• Separating table values from their headers.
• Indexing noisy web-page content.
• Discarding useful document metadata.
• Making chunks so small that they lose context.
• Making chunks so large that they contain unrelated topics.
• Changing chunking without evaluating retrieval quality.

Chunking and Retrieval Recall

Chunking has a direct relationship with retrieval recall. If an answer is divided into fragments that are individually too vague, the retrieval system may fail to recognize them as relevant.

Conversely, if an entire document is represented by only a few huge chunks, a query may match the document broadly but fail to surface the exact passage needed for the answer.

A useful chunking strategy therefore aims to make important pieces of information independently retrievable without destroying the relationships between them.

Chunking and Retrieval Precision

Precision can suffer when chunks contain too many unrelated concepts. A query about authentication may retrieve a large document section containing authentication, logging, caching, and deployment simply because one part of the section is relevant.

More focused chunks allow the retrieval system and reranker to distinguish these concepts more effectively.

Chunking and Hallucinations

Poor chunking can indirectly contribute to hallucinations. If the relevant information is split, truncated, or retrieved without necessary context, the LLM may not have enough evidence to produce a reliable answer.

However, chunking is only one part of the problem. Retrieval algorithms, embeddings, reranking, prompts, model behavior, and source quality also influence whether an answer is grounded.

Chunking Performance and Cost

More chunks generally mean more embeddings, more index entries, and potentially more retrieval candidates. Excessive overlap can increase these costs further.

At the same time, extremely large chunks can increase the amount of text sent to rerankers and language models. Good chunking can therefore improve both retrieval quality and resource efficiency.

Should You Use the Same Chunk Size Everywhere?

Usually, there is no strong reason to force every document type into exactly the same chunk size. Different content has different natural structures.

A FAQ answer, API method description, source-code function, legal clause, and long explanatory section may all require different retrieval units. A flexible ingestion pipeline can use different rules while maintaining a consistent metadata and indexing model.

A Recommended Starting Strategy

For many RAG applications, a good starting point is recursive or structure-aware chunking with controlled size limits, limited overlap, and rich metadata. Preserve headings, titles, code blocks, and other meaningful structures whenever possible.

Then evaluate the system using real questions. If retrieval recall is poor, experiment with larger chunks, different boundaries, additional context, or deeper retrieval. If precision is poor, experiment with smaller or more focused chunks, hybrid retrieval, reranking, and better metadata filters.

Advanced Chunking Strategy

For complex RAG applications, a layered strategy can provide both precise retrieval and sufficient context. Documents can first be parsed into structural sections, then large sections can be recursively divided into smaller chunks. Each child chunk can retain its parent section and document metadata.

The retrieval system can search the smaller chunks, rerank the candidates, and then optionally expand selected results with their parent context. This separates the problem of finding the relevant passage from the problem of providing enough context to the LLM.

Source Document
      ↓
Document Structure
      ↓
Sections / Parents
      ↓
Child Chunks
      ↓
Embeddings + Metadata
      ↓
Retrieval
      ↓
Reranking
      ↓
Context Expansion
      ↓
LLM

Best Practices for RAG Chunking

• Choose chunk boundaries based on document structure when possible.
• Keep related information together.
• Use fixed-size splitting as a baseline rather than an automatic final solution.
• Apply controlled overlap only when it provides measurable value.
• Preserve titles and heading hierarchies.
• Keep code examples and explanations associated when appropriate.
• Preserve table headers and structured relationships.
• Attach useful metadata to every chunk.
• Use parent-child relationships for long or hierarchical documents.
• Avoid excessive duplication caused by overlap.
• Use different strategies for substantially different document types.
• Evaluate chunking with real queries.
• Measure both retrieval metrics and final answer quality.
• Consider latency, storage, embedding, and reranking costs.
• Inspect failed retrievals at the chunk level.

Frequently Asked Questions

Conclusion

Chunking is a foundational part of RAG retrieval. The way a document is divided determines what each embedding represents, what the retrieval system can find, how much context reaches the language model, and how efficiently the system uses its storage and compute resources.

Simple fixed-size chunking is useful as a baseline, but structure-aware, recursive, semantic, hierarchical, and parent-child strategies can provide better results for complex documents. The most effective approach is usually one that preserves meaningful boundaries while keeping each retrieval unit focused and sufficiently contextual.

There is no universal chunk size or overlap value that works for every RAG application. The best strategy should be determined by the actual documents and queries in your system. Build a representative evaluation set, inspect retrieval failures, compare different chunking configurations, and measure both retrieval quality and final answer quality.