RAG应用中如何选择最佳文档分块技术？（附静态与动态方法对比）

Introduction: The Evolving Role of Chunking in RAG

Large Language Models (LLMs) possess a finite capacity for input and output generation. In Retrieval-Augmented Generation (RAG) applications, a set of documents is first retrieved and appended to the input alongside an instruction, forming the final prompt. This process is known as in-context learning. A useful analogy can be drawn with computer architecture: the LLM is the CPU, the immediate context is the RAM, and the entire document library is the hard disk.

大型语言模型（LLM）的输入和输出生成能力是有限的。在检索增强生成（RAG）应用中，首先会检索一组文档，并将其与指令一起添加到输入中，从而构成最终的提示词。这个过程被称为上下文学习。一个有用的类比是计算机架构：LLM是CPU，即时上下文是RAM，而整个文档库则是硬盘。

The context length (RAM) has been expanding consistently. In the early days of working with Recurrent Neural Networks (RNNs), reliable predictions could be made with input lengths of around 20 words, though issues like vanishing gradients were prevalent. Today, we have LLMs boasting contexts of up to 200,000 tokens. It's important to note that the unit of processing has also evolved, transitioning from words to subword tokens.

上下文长度（RAM）一直在持续扩展。在早期使用循环神经网络（RNN）时，大约20个单词的输入长度就能进行可靠的预测，尽管梯度消失等问题普遍存在。如今，我们拥有了上下文长度高达20万个令牌的LLM。需要注意的是，处理单元也发生了演变，从单词过渡到了子词令牌。

Chunking, the technique of dividing a document into smaller segments to fit within the model's context window, has played a pivotal role in managing these expanding contexts. With a context size of 200K tokens, it's theoretically possible to process up to 330 pages within a single chunk for prediction.

分块（Chunking）是一种将文档分割成更小片段以适应模型上下文窗口的技术，它在管理这些不断扩展的上下文方面发挥了关键作用。拥有20万个令牌的上下文大小，理论上可以在单个块内处理多达330页的文本进行预测。

Initially, chunking presented a significant challenge for LLM applications. However, as context lengths grew, this problem seemed to recede into the background. Or did it? It turns out that processing vast quantities of text in a single pass introduces substantial latency to the entire system. Consequently, strategic chunking remains a critical technique for developing high-performance, low-latency RAG applications.

最初，分块对LLM应用来说是一个重大挑战。然而，随着上下文长度的增长，这个问题似乎逐渐淡出视野。但事实果真如此吗？ 事实证明，单次处理大量文本会给整个系统带来显著的延迟。因此，策略性的分块仍然是开发高性能、低延迟RAG应用的关键技术。

Key Chunking Strategies: From Static to Dynamic

From practical experience, the optimal chunking strategy is highly dependent on the user's or system's intent. For instance, summarizing an entire document requires consideration of all chunks, while answering a specific question necessitates retrieving only the most relevant segments. Let's explore several chunking techniques.

根据实践经验，最优的分块策略高度依赖于用户或系统的意图。例如，总结整个文档需要考虑所有块，而回答特定问题则只需要检索最相关的片段。让我们探讨几种分块技术。

Static Chunking

One straightforward method involves setting a fixed chunk size aligned with the LLM's context length. A crucial consideration is that if a chunk is to be combined with an instruction in a prompt, the chunk's length must be smaller than the context length minus the instruction length. For example, with a 200-token context and a 20-token instruction, the chunk size should be set to 180 tokens.

一种直接的方法是根据LLM的上下文长度设置固定的块大小。一个关键的考虑因素是，如果一个块需要与指令一起构成提示词，那么块的长度必须小于上下文长度减去指令长度。例如，上下文为200个令牌，指令为20个令牌，则块大小应设置为180个令牌。

In practice, this method often lacks versatility. It appears most suitable for agentic systems where the agent is explicitly aware that a document is segmented and can make decisions about which chunks to retrieve sequentially.

在实践中，这种方法通常缺乏灵活性。它似乎最适合于智能体系统，其中智能体明确知道文档被分割，并可以决定按顺序检索哪些块。

Dynamic Chunking Based on Traditional Information Retrieval (IR)

This method involves segmenting a document based on predefined conditions, framed as an Information Retrieval problem. A document is treated as a dataset where each sentence is an item. Given a query, a list of ranked text fragments is retrieved, with fragment boundaries aligning with natural sentence delimiters. Each retrieved fragment effectively becomes a chunk. This can be efficiently implemented using tools like Elasticsearch with its default BM25一种基于概率信息检索理论的文本检索模型，通过词频和逆文档频率对文档进行排名，考虑词项的相关性和稀有性。 ranker.

这种方法基于预定义条件对文档进行分割，并将其构建为一个信息检索问题。文档被视为一个数据集，其中每个句子是一个条目。给定一个查询，会检索出一个排名的文本片段列表，片段边界与自然的句子分隔符对齐。每个检索到的片段即成为一个块。这可以使用像Elasticsearch及其默认的BM25一种基于概率信息检索理论的文本检索模型，通过词频和逆文档频率对文档进行排名，考虑词项的相关性和稀有性。排序器这样的工具来高效实现。

By treating chunking as an IR problem, we can leverage established IR techniques such as stemming, lemmatization, synonym expansion, and query reformulation. This approach is straightforward, efficient, and less resource-intensive than neural methods. It also avoids potential issues with Out-of-Distribution (OOD) data shifts common in learned models.

通过将分块视为IR问题，我们可以利用成熟的IR技术，如词干提取、词形还原、同义词扩展和查询重构。这种方法直接、高效，并且比神经方法资源消耗更少。它还避免了学习模型中常见的分布外（OOD）数据偏移问题。

Dynamic Chunking Based on Neural IR (Dense Embeddings)

Similar to the previous method, this approach converts document sentences into dense vector embeddings. Relevant chunks are then retrieved based on the similarity (e.g., cosine similarity) between the query embedding and the sentence embeddings. While computationally more demanding, this method can yield higher accuracy, especially when the embedding model is fine-tuned or pre-trained on a domain similar to the application. Most modern vector databases support this indexing and search paradigm.

与之前的方法类似，这种方法将文档句子转换为稠密向量嵌入。然后基于查询嵌入和句子嵌入之间的相似度（例如，余弦相似度）来检索相关块。虽然计算需求更高，但这种方法可以获得更高的准确性，特别是当嵌入模型在与应用相似的领域上进行微调或预训练时。大多数现代向量数据库都支持这种索引和搜索范式。

Dynamic Chunking Based on Neural IR (ColBERT一种神经IR方法，避免池化操作，更有效地保留查询和文档嵌入信号，可以自动识别多句子片段。)

Many vector search systems require pooling query and document tokens into a single vector, which can discard valuable fine-grained signals. ColBERT introduces a late-interaction model that avoids this pooling, preserving more granular signal from both query and document embeddings. Furthermore, it can automatically identify multi-sentence fragments delimited by natural boundaries without requiring pre-segmentation into sentences.

许多向量搜索系统需要将查询和文档令牌池化为单个向量，这可能会丢弃有价值的细粒度信号。ColBERT 引入了一种延迟交互模型，避免了这种池化操作，保留了查询和文档嵌入中更细粒度的信号。此外，它可以自动识别由自然边界分隔的多句子片段，而无需预先分割成句子。

However, this method incurs higher computational and storage costs and requires integration of the ColBERT一种神经IR方法，避免池化操作，更有效地保留查询和文档嵌入信号，可以自动识别多句子片段。 ranker within the database infrastructure, which is not always available out-of-the-box. Like other neural methods, it is also susceptible to performance degradation on OOD data.

然而，这种方法会产生更高的计算和存储成本，并且需要在数据库基础设施中集成ColBERT一种神经IR方法，避免池化操作，更有效地保留查询和文档嵌入信号，可以自动识别多句子片段。排序器，而这并非总是开箱即用的。与其他神经方法一样，它也容易在OOD数据上出现性能下降。

Comparative Analysis of Chunking Techniques

The choice of chunking strategy involves trade-offs between simplicity, performance, accuracy, and computational cost. The following table summarizes the key characteristics of the discussed methods.

分块策略的选择需要在简单性、性能、准确性和计算成本之间进行权衡。下表总结了所讨论方法的关键特征。

Conclusion and Practical Guidance

While LLM context windows continue to expand, strategic document chunking remains essential for building responsive and efficient RAG applications. The primary goal has shifted from merely fitting text into a context window to optimizing for retrieval precision and system latency.

尽管LLM的上下文窗口在持续扩大，但策略性的文档分块将文档分割成较小片段的过程，便于后续处理和检索。对于构建响应迅速且高效的RAG应用仍然至关重要。主要目标已从仅仅将文本塞进上下文窗口，转变为优化检索精度和系统延迟。

There is no universally "best" chunking technique. The optimal choice is dictated by the specific requirements of the end application:

• Prioritize speed and simplicity? Start with Traditional IR-based dynamic chunking.
• Need deep semantic understanding? Consider Dense Embedding or ColBERT一种神经IR方法，避免池化操作，更有效地保留查询和文档嵌入信号，可以自动识别多句子片段。 approaches, acknowledging their higher computational footprint.
• Building an agentic system that controls retrieval? Static chunking might provide the necessary structure.

不存在通用的“最佳”分块技术。最优选择取决于最终应用的具体需求：

• 优先考虑速度和简单性？从基于传统IR的动态分块基于预定义条件或查询动态分割文档的分块方法，包括传统IR、神经IR和ColBERT等变体。开始。
• 需要深度的语义理解？考虑稠密嵌入或ColBERT一种神经IR方法，避免池化操作，更有效地保留查询和文档嵌入信号，可以自动识别多句子片段。方法，但需接受其较高的计算成本。
• 构建一个控制检索的智能体系统？静态分块将分块大小与LLM上下文长度对齐的分块方法，适用于代理系统。可能提供所需的结构。

Ultimately, effective chunking is a foundational component of the RAG pipeline, directly impacting the quality and speed of the generated responses. By carefully selecting and implementing the appropriate strategy, developers can significantly enhance the performance and user experience of their LLM-powered applications.

最终，有效的分块是RAG流水线的基础组成部分，直接影响生成响应的质量和速度。通过仔细选择和实施适当的策略，开发者可以显著提升其LLM驱动应用的性能和用户体验。

常见问题（FAQ）

为什么LLM上下文长度增加了，RAG应用还需要文档分块将文档分割成较小片段的过程，便于后续处理和检索。？

虽然上下文长度增加，但单次处理大量文本会显著增加系统延迟。策略性分块仍是实现高性能、低延迟RAG应用的关键技术。

静态分块将分块大小与LLM上下文长度对齐的分块方法，适用于代理系统。和动态分块基于预定义条件或查询动态分割文档的分块方法，包括传统IR、神经IR和ColBERT等变体。有什么区别？

静态分块将分块大小与LLM上下文长度对齐的分块方法，适用于代理系统。设置固定大小，适合智能体系统；动态分块基于预定义条件或查询动态分割文档的分块方法，包括传统IR、神经IR和ColBERT等变体。根据查询需求灵活划分，包括基于传统IR、神经IR嵌入和ColBERT一种神经IR方法，避免池化操作，更有效地保留查询和文档嵌入信号，可以自动识别多句子片段。等方法。

如何选择最适合的文档分块将文档分割成较小片段的过程，便于后续处理和检索。方法？

最优分块策略取决于具体应用需求。总结文档需考虑所有块，回答特定问题只需检索最相关片段，需根据意图选择相应技术。