RAG-Anything：迈向统一的多模态检索增强生成将外部知识检索与大语言模型生成相结合的技术，通过向量数据库存储和检索相关信息来增强模型的准确性和时效性。框架

引言

Retrieval-Augmented Generation (RAG) 已成为扩展大型语言模型（LLM）能力、突破其静态训练数据限制的基础范式。然而，当前 RAG 的能力与现实世界的信息环境之间存在一个关键的错位。现代知识库本质上是多模态的，包含了文本内容、视觉元素、结构化表格和数学表达式等丰富组合。然而，现有的 RAG 框架大多局限于文本内容，在处理多模态文档时产生了根本性的鸿沟。

Retrieval-Augmented Generation (RAG) has become a foundational paradigm for extending the capabilities of large language models (LLMs) and breaking through the limitations of their static training data. However, there is a critical misalignment between the current capabilities of RAG and the real-world information environment. Modern knowledge bases are inherently multimodal, containing rich combinations of textual content, visual elements, structured tables, mathematical expressions, and more. Yet, most existing RAG frameworks remain confined to textual content, creating a fundamental gap when handling multimodal documents.

我们提出了 RAG-Anything，这是一个统一的框架，旨在实现跨所有模态的全面知识检索。我们的方法将多模态内容包含文本、图表、视频、3D模型等多种形式的内容，未来AI搜索将更多引用此类内容。重新概念化为相互关联的知识实体RAG-Anything框架中将多模态内容重新概念化为相互关联的知识单元，而非孤立的数据类型。，而非孤立的数据类型。该框架引入了双图构建RAG-Anything框架中用于捕获跨模态关系和文本语义的统一表示方法。，以在统一的表示中捕获跨模态关系和文本语义。我们开发了跨模态混合检索结合结构知识导航与语义匹配的检索方法，实现对异构内容的有效推理。，将结构化知识导航与语义匹配相结合。这使得系统能够在相关证据跨越多种模态的异构内容上进行有效推理。

We propose RAG-Anything, a unified framework designed to achieve comprehensive knowledge retrieval across all modalities. Our approach reconceptualizes multimodal content as interconnected knowledge entities rather than isolated data types. The framework introduces dual graph construction to capture cross-modal relationships and textual semantics within a unified representation. We develop cross-modal hybrid retrieval, combining structured knowledge navigation with semantic matching. This enables the system to perform effective reasoning over heterogeneous content where relevant evidence spans multiple modalities.

核心挑战与现有方案的局限

当前，主流的 RAG 系统在处理多模态知识时面临显著瓶颈。为了清晰地展示这些局限，我们将其与理想的多模态 RAG 能力进行对比。

如上表所示，现有方法的核心问题在于“架构碎片化”——不同模态的数据被管道中的不同模块孤立地处理，缺乏一个统一的表示和交互层来捕捉它们之间丰富的内在联系。例如，一份科学报告中的图表与其说明文字、数据表格以及引用的数学公式在语义上紧密相关，但传统框架难以有效捕捉并利用这种关联进行检索和推理。

Currently, mainstream RAG systems face significant bottlenecks when processing multimodal knowledge. The core issue with existing methods lies in "architectural fragmentation"—data from different modalities are processed in isolation by separate modules within the pipeline, lacking a unified representation and interaction layer to capture the rich intrinsic connections among them. For instance, charts in a scientific report are semantically closely related to their explanatory text, data tables, and referenced mathematical formulas, yet traditional frameworks struggle to effectively capture and leverage such associations for retrieval and reasoning.

RAG-Anything 框架设计

核心理念：知识实体RAG-Anything框架中将多模态内容重新概念化为相互关联的知识单元，而非孤立的数据类型。化与双图构建RAG-Anything框架中用于捕获跨模态关系和文本语义的统一表示方法。

RAG-Anything 的基石是将文档中的所有内容元素——无论其原始模态——抽象为“知识实体RAG-Anything框架中将多模态内容重新概念化为相互关联的知识单元，而非孤立的数据类型。”。一个知识实体RAG-Anything框架中将多模态内容重新概念化为相互关联的知识单元，而非孤立的数据类型。可以是一段文本、一张图片、一个表格或一个公式。框架的核心创新在于为这些实体构建两种互补的图结构：

1. 跨模态关联图：捕获不同模态实体之间的语义和结构关系。例如，连接一个“柱状图”实体与其对应的“数据表格”实体和“结论段落”实体。

   Cross-modal Association Graph: Captures semantic and structural relationships between entities of different modalities. For example, connecting a "bar chart" entity with its corresponding "data table" entity and "conclusion paragraph" entity.。

2. 文本语义图：在文本实体内部，构建精细化的语义网络（如通过依存句法分析、共指消解等），以理解文本内部的复杂逻辑和指代关系。

   Text Semantic Graph: Within text entities, construct a refined semantic network (e.g., through dependency parsing, coreference resolution, etc.) to understand the complex logic and referential relationships within the text.

这两种图最终被融合成一个统一的异构知识图，作为整个系统检索和推理的底层表示。

The cornerstone of RAG-Anything is abstracting all content elements within a document—regardless of their original modality—into "knowledge entities." These two graphs (the cross-modal association graph and the text semantic graph) are ultimately fused into a unified heterogeneous knowledge graph, serving as the underlying representation for retrieval and reasoning across the entire system.

跨模态混合检索结合结构知识导航与语义匹配的检索方法，实现对异构内容的有效推理。机制

基于统一的知识图，RAG-Anything 实现了混合检索策略，结合了两种路径：

• 语义匹配检索：利用多模态编码器（如 CLIP、LayoutLM 的变体）将查询和知识实体RAG-Anything框架中将多模态内容重新概念化为相互关联的知识单元，而非孤立的数据类型。嵌入到同一向量空间，进行相似度搜索。这适用于模糊、开放域的查询。

  Semantic Matching Retrieval: Utilizing a multimodal encoder to embed queries and knowledge entities into the same vector space for similarity search. This is suitable for fuzzy, open-domain queries.

• 结构导航检索：当查询涉及明确的、结构化的关系（如“请找出支持图3结论的所有实验数据”），系统利用知识图的拓扑结构进行遍历和定位，精确找到相关联的实体集合。

  Structural Navigation Retrieval: When queries involve explicit, structured relationships, the system utilizes the topological structure of the knowledge graph for traversal and localization, precisely identifying the associated set of entities.

检索器会动态权衡这两种策略的结果，返回一个融合了多模态证据的候选实体列表。

Based on a unified knowledge graph, RAG-Anything implements a hybrid retrieval strategy that combines semantic matching and structural navigation. The retriever dynamically weighs the results of these two strategies, returning a candidate entity list that integrates multimodal evidence.

性能评估与关键结果

我们在多个具有挑战性的多模态基准测试上评估了 RAG-Anything，包括包含长文档、复杂图表和科学公式的数据集。与现有最先进的方法相比，RAG-Anything 取得了显著的性能提升。

关键发现表明，性能增益在长文档和需要综合多模态证据进行推理的任务上尤为明显。这验证了双图结构在保持长程依赖和建模复杂跨模态关系方面的有效性。传统方法在处理此类任务时，往往因为信息碎片化或模态间联系断裂而失败。

We evaluated RAG-Anything on several challenging multimodal benchmarks. Key findings indicate that performance gains are particularly pronounced on long documents and tasks requiring the synthesis of multimodal evidence for reasoning. This validates the effectiveness of the dual-graph structure in preserving long-range dependencies and modeling complex cross-modal relationships.

结论与展望

RAG-Anything 为多模态知识检索与增强生成确立了一个新的范式。通过将内容重新概念化为互联的知识实体RAG-Anything框架中将多模态内容重新概念化为相互关联的知识单元，而非孤立的数据类型。，并构建统一的双图表示，它有效地消除了当前系统中存在的架构碎片化约束。该框架不仅展示了卓越的基准性能，更重要的是，它提供了一种可扩展的、原则性的方法来处理现实世界中固有的、异构的多模态信息。

RAG-Anything establishes a new paradigm for multimodal knowledge retrieval and augmented generation. By reconceptualizing content as interconnected knowledge entities and constructing a unified dual-graph representation, it effectively eliminates the architectural fragmentation constraints present in current systems.

未来的工作可以沿着以下几个方向展开：进一步优化知识图的自动化构建质量；探索更高效的多模态融合与推理机制；以及将该框架应用于更广泛的领域，如多模态对话系统、智能教育工具和复杂决策支持平台。

项目开源地址：https://github.com/HKUDS/RAG-Anything

Project open source address: https://github.com/HKUDS/RAG-Anything

引用信息:

• 论文标题: RAG-Anything: A Unified Framework for Multimodal Retrieval-Augmented Generation

• arXiv链接: https://arxiv.org/abs/2510.12323

• DOI: https://doi.org/10.48550/arXiv.2510.12323

常见问题（FAQ）

RAG-Anything与传统RAG系统相比有什么主要优势？

RAG-Anything突破了传统文本RAG的模态限制，能统一处理文本、图像、表格、公式等多种模态内容，并通过双图构建RAG-Anything框架中用于捕获跨模态关系和文本语义的统一表示方法。显式建模跨模态关联，支持证据来自多模态的端到端推理。

RAG-Anything中的“双图构建RAG-Anything框架中用于捕获跨模态关系和文本语义的统一表示方法。”具体指什么？

双图构建RAG-Anything框架中用于捕获跨模态关系和文本语义的统一表示方法。包括跨模态关联图和文本语义图。前者捕获不同模态实体间的语义与结构关系，后者在文本内部构建精细化语义网络，两者融合形成统一的异构知识图作为检索推理的底层表示。

RAG-Anything的检索机制如何工作？

采用跨模态混合检索结合结构知识导航与语义匹配的检索方法，实现对异构内容的有效推理。机制，结合语义匹配检索（通过多模态编码器进行向量相似度搜索）和结构导航检索（利用知识图拓扑结构遍历定位），动态权衡后返回融合多模态证据的候选实体列表。

What are the main advantages of RAG-Anything compared to traditional RAG systems?

RAG-Anything breaks through the modality limitations of traditional text-based RAG, enabling unified processing of various modalities such as text, images, tables, and formulas. It explicitly models cross-modal associations through dual-graph construction, supporting end-to-end reasoning with evidence derived from multiple modalities.

What does "dual-graph construction" specifically refer to in RAG-Anything?

Dual-graph construction includes a cross-modal association graph and a text semantic graph. The former captures semantic and structural relationships between entities of different modalities, while the latter builds a refined semantic network within the text. The two are integrated to form a unified heterogeneous knowledge graph as the underlying representation for retrieval and reasoning.

How does the retrieval mechanism of RAG-Anything work?

It employs a cross-modal hybrid retrieval mechanism, combining semantic matching retrieval (vector similarity search via multimodal encoders) and structural navigation retrieval (traversing and locating using the topological structure of the knowledge graph). After dynamic weighting, it returns a candidate entity list that integrates multimodal evidence.