PageIndex：开源无向量RAG系统，重塑长文档精准检索

一、什么是 PageIndex？

PageIndex is an open-source, vector-free Retrieval-Augmented Generation (RAG) system developed by VectifyAI. It is designed to address the accuracy issues inherent in traditional vector database-based RAG systems when handling long documents, which often rely on semantic similarity. By constructing a hierarchical tree-like index that mimics human logic in processing documents, PageIndex enables precise retrieval based on reasoning rather than vector matching. It supports features like chunk-free processing and visual retrieval, making it suitable for professional scenarios such as financial reports, academic papers, and legal documents. It can be quickly deployed via self-hosting or cloud services.

PageIndex 是由 VectifyAI 开发的一款开源无向量检索增强生成（RAG）系统，旨在解决传统向量数据库在长文档检索中依赖语义相似性导致的准确性问题。它通过构建层级树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。模拟人类处理文档的逻辑，基于推理而非向量匹配实现精准检索，支持无分块处理、视觉检索PageIndex的功能之一，通过分析PDF页面中的图像特征（如表格结构、图表类型）生成视觉标签并关联到索引，支持用户直接检索图像内容，无需依赖OCR文字识别。等功能，适用于金融报告、学术论文、法律文档等专业场景，可通过自托管PageIndex的一种部署方式，用户可在本地环境（需Python等依赖）克隆项目代码并运行，适合对数据隐私和控制权要求高的技术团队。或云服务快速部署使用。

In the era of information explosion, the efficient retrieval of long documents (such as hundreds-page financial annual reports, academic monographs, and legal manuals) remains a persistent industry challenge. Traditional RAG systems rely on vector databases, which involve chunking documents, converting them into vectors, and then retrieving results based on semantic similarity. However, this approach has significant drawbacks: manual chunking can disrupt document logic, vector similarity does not equate to content relevance (e.g., "profit growth" and "profit decline" are semantically similar but opposite in meaning), and context fragmentation in long documents can lead to retrieval bias.

在信息爆炸的时代，长文档（如数百页的财务年报、学术专著、法律手册）的高效检索始终是行业痛点。传统的检索增强生成（RAG）系统依赖向量数据库，通过将文档分块、转化为向量，再基于语义相似性匹配检索结果。但这种方式存在明显缺陷：人工分块可能破坏文档逻辑、向量相似性不等于内容相关性（如“利润增长”与“利润下降”语义相似但含义相反）、长文档上下文断裂导致检索偏差。

PageIndex is an open-source tool born to solve these problems. Its core logic simulates how human experts process complex documents: first grasping the document structure through a "table of contents," then locating relevant sections layer by layer, and finally extracting information precisely. Unlike traditional vector-based RAG, PageIndex does not rely on vector databases or require manual chunking. Instead, it leverages the natural structure of documents (such as heading hierarchies and page numbers) to generate a tree-like index and uses the reasoning capabilities of Large Language Models (LLMs) to locate relevant content, achieving a "relevance-first" retrieval effect.

PageIndex正是为解决这些问题而生的开源工具。它核心逻辑是模拟人类专家处理复杂文档的方式——先通过“目录”把握文档结构，再逐层定位相关章节，最终精准提取信息。不同于传统向量RAG，PageIndex无需依赖向量数据库，也无需人工分块，而是利用文档自然结构（如标题层级、页码）生成树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。，通过大语言模型（LLM）的推理能力定位相关内容，实现“相关性优先”的检索效果。

Simply put, traditional vector RAG is like "fuzzy search," while PageIndex is like "finding answers with logic." For example, when retrieving "2023 net profit growth rate," it would first locate the "Financial Data" chapter in the document, then find the "Net Profit" subsection, and finally extract the content from the corresponding pages, rather than matching semantically similar fragmented sentences.

简单来说，传统向量RAG像“模糊搜索”，而PageIndex像“带着逻辑找答案”——例如检索“2023年净利润增长率”时，它会先定位文档中的“财务数据”章节，再找到“净利润”子标题，最终提取对应页码的内容，而非匹配语义相似的零散句子。

二、核心功能特色

The core advantages of PageIndex stem from its "anti-vector-dependency" design philosophy. Its specific functional features are as follows:

PageIndex的核心优势源于其“反向量依赖”的设计理念，具体功能特色如下：

1. 无向量、无分块，摆脱技术依赖

Traditional RAG requires deploying vector databases (e.g., Pinecone, Milvus) and manually setting chunking rules (e.g., 200 words per chunk). PageIndex completely eliminates these steps. It directly reads the native structure of PDF documents (titles, paragraphs, page numbers) and uses LLMs to automatically identify hierarchical relationships (e.g., "chapter-section-subsection") to generate a structured index. This feature offers two major benefits:

• 降低技术门槛 (Lower Technical Barrier): No need to learn vector database operations or chunking strategies, allowing even novice users to get started quickly.
• 保留文档完整性 (Preserve Document Integrity): Avoids context fragmentation caused by chunking (e.g., a formula split into two chunks might be missed by traditional RAG).

传统RAG需部署向量数据库（如Pinecone、Milvus），并人工设置分块规则（如每200字一块），而PageIndex完全无需这些步骤。它直接读取PDF文档的原生结构（标题、段落、页码），通过LLM自动识别层级关系（如“章-节-小节”），生成结构化索引。这一特性带来两大好处：

• 降低技术门槛：无需学习向量数据库运维或分块策略，小白用户也能快速上手；
• 保留文档完整性：避免分块导致的上下文断裂（如某公式被拆分为两块，传统RAG可能漏检）。

2. 树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。+推理检索，模拟人类逻辑

The core innovation of PageIndex is the combination of a "Tree-like Index" and "Reasoning-based Retrieval":

• 树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。 (Tree-like Index): Similar to an optimized "smart table of contents," it includes each node's title, page range, and content summary (generated by an LLM). For example, the tree index for an academic paper might be: Root Node (Paper Title) → Chapter Nodes ("Introduction," "Method," "Conclusion") → Section Nodes ("Dataset Introduction," "Experimental Steps") → Leaf Nodes (specific paragraph content).
• 推理检索 (Reasoning-based Retrieval): When a user asks a question, the LLM performs multi-step reasoning based on the tree index rather than directly matching vectors. For example, for the query "What is the model's accuracy on the ImageNet dataset?", the system would first reason that "the experimental results section needs to be found," then locate the "sub-chapter for the ImageNet dataset," and finally extract the content from the corresponding pages.

Compared to traditional vector RAG, this method significantly improves retrieval accuracy. In the financial benchmark FinanceBench, a system based on PageIndex achieved an accuracy rate of 98.7%, far surpassing the 72.3% of traditional vector RAG (data from the project's test report).

PageIndex的核心创新是“树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。”和“推理检索”的结合：

• 树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。：类似优化后的“智能目录”，包含每个节点的标题、页码范围、内容摘要（由LLM生成）。例如，一本学术论文的树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。可能是：根节点（论文标题）→ 章节点（“引言”“方法”“结论”）→ 节节点（“数据集介绍”“实验步骤”）→ 叶子节点（具体段落内容）。
• 推理检索：当用户提问时，LLM会基于树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。进行多步推理，而非直接匹配向量。例如，用户问“该模型在ImageNet数据集上的准确率是多少”，系统会先推理“需要找实验结果部分”，再定位“数据集为ImageNet的子章节”，最终提取对应页码的内容。

这种方式相比传统向量RAG，检索准确率提升显著——在金融基准测试FinanceBench中，基于PageIndex的系统准确率达98.7%，远超传统向量RAG的72.3%（数据源自项目测试报告）。

3. 透明可追溯，解决“黑箱检索”问题

The retrieval results of traditional vector RAG are often questioned—"Why was this content found?"—because the vector similarity calculation process is difficult to explain. In contrast, PageIndex's retrieval process is fully transparent: every step of reasoning is traceable (e.g., "from the 'Financial Summary' node → 'Net Profit' child node → pages 15-16"). Users can clearly see the system's "thought process," making it easier to verify the rationality of the results. This is crucial for fields requiring rigor, such as law and healthcare.

传统向量RAG的检索结果常被质疑“为什么找到这段内容”，因向量相似性计算过程难以解释。而PageIndex的检索过程完全透明：每一步推理都可追溯（如“从‘财务摘要’节点→‘净利润’子节点→页码15-16”），用户能清晰看到系统的“思考路径”，便于验证结果合理性。这对法律、医疗等需严谨性的领域至关重要。

4. 长文档适配，突破上下文限制

LLMs typically have context window limitations (e.g., GPT-4 has 8k-128k tokens). Traditional RAG needs to split long documents into small chunks for processing. PageIndex, however, "compresses" document information through its tree index—the node summaries in the tree index retain only core information (e.g., a chapter summary might be only 200 words), enabling the LLM to process documents hundreds of pages long within a limited context. For example, a 500-page legal manual might require only 5k tokens for its tree index, supporting precise retrieval across the entire document.

LLM通常有上下文窗口限制（如GPT-4为8k-128k tokens），传统RAG需将长文档拆分为小块才能处理，而PageIndex通过树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。“压缩”文档信息——树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。的节点摘要仅保留核心信息（如一章的摘要可能仅200字），使LLM能在有限上下文内处理数百页文档。例如，一本500页的法律手册，其树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。可能仅需5k tokens，即可支持全文档精准检索。

5. 支持视觉检索PageIndex的功能之一，通过分析PDF页面中的图像特征（如表格结构、图表类型）生成视觉标签并关联到索引，支持用户直接检索图像内容，无需依赖OCR文字识别。，无需OCR也能找图

Some professional documents (e.g., engineering drawings, financial report charts) contain大量图像信息. Traditional RAG relies on OCR for text recognition, which can easily lead to errors due to blurry images. PageIndex provides visual retrieval capabilities based on page images: by analyzing image features (e.g., table structure, chart type) to generate "visual summaries," which are incorporated into the tree index. For example, when a user asks for the "2023 quarterly revenue bar chart," the system can directly locate the page containing that chart without relying on OCR text recognition.

部分专业文档（如工程图纸、财报图表）包含大量图像信息，传统RAG依赖OCR识别文字，易因图像模糊导致错误。PageIndex提供基于页面图像的视觉检索PageIndex的功能之一，通过分析PDF页面中的图像特征（如表格结构、图表类型）生成视觉标签并关联到索引，支持用户直接检索图像内容，无需依赖OCR文字识别。能力：通过分析图像特征（如表格结构、图表类型）生成“视觉摘要”，并纳入树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。。例如，用户问“2023年季度营收柱状图”，系统可直接定位包含该图表的页码，无需依赖OCR文字识别。

三、技术架构与流程

The core technology of PageIndex can be broken down into three parts: "Tree Index Generation," "Reasoning-based Retrieval," and "Visual Retrieval Support." The differences between its technical workflow and that of traditional vector RAG are shown in the following table:

PageIndex的核心技术可拆解为“树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。生成”“推理式检索PageIndex 采用的检索方法，让大模型基于文档树结构进行逻辑推理，逐步缩小检索范围，而非依赖简单的向量相似度匹配。”“视觉检索PageIndex的功能之一，通过分析PDF页面中的图像特征（如表格结构、图表类型）生成视觉标签并关联到索引，支持用户直接检索图像内容，无需依赖OCR文字识别。支持”三部分，其技术流程与传统向量RAG的差异如下表所示：

技术环节	传统向量RAG	PageIndex
文档处理	人工分块→转化为向量	自动识别自然结构→生成树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。
检索依据	向量语义相似性	LLM推理+树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。层级关系
上下文处理	依赖分块拼接，易断裂	基于索引节点定位，保留完整上下文
可解释性	低（向量计算过程黑箱）	高（推理路径可追溯）
长文档适配	需复杂分块策略，效果有限	索引压缩文档信息，天然支持长文档

1. 树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。生成流程

The tree-like index is the "core engine" of PageIndex. Its generation process can be divided into 3 steps:

1. 结构解析 (Structure Parsing): The system reads the PDF document's metadata (such as title styles, page numbers, font sizes) to identify the natural hierarchical structure. For example, "Level 1 Heading (e.g., 1. Introduction)" and "Level 2 Heading (e.g., 1.1 Research Background)" form the initial hierarchical framework.
2. 内容摘要 (Content Summarization): For each hierarchical node (e.g., "1.1 Research Background"), the LLM extracts the core information of the content covered by that node (e.g., "This section introduces the research status and 3 existing problems in the XX field"), generates a summary, and associates it with a page range (e.g., "pages 3-5").
3. 索引优化 (Index Optimization): The system optimizes the index structure through cross-validation (e.g., checking if child node content aligns with the parent node's theme) to ensure consistent hierarchical logic. The final generated index is in JSON format (examples can be found in the PRML_structure.json file in the project's tests/results directory) and can be directly used for retrieval.

树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。是PageIndex的“核心引擎”，其生成过程可分为3步：

1. 步骤1：结构解析：系统读取PDF文档的元数据（如标题样式、页码、字体大小），识别自然层级结构。例如，“一级标题（如1. 引言）”“二级标题（如1.1 研究背景）”等，形成初始层级框架。
2. 步骤2：内容摘要：对每个层级节点（如“1.1 研究背景”），LLM会提取该节点涵盖内容的核心信息（如“本节介绍了XX领域的研究现状及存在的3个问题”），生成摘要并关联页码范围（如“页码3-5”）。
3. 步骤3：索引优化：系统通过交叉验证（如检查子节点内容是否符合父节点主题）优化索引结构，确保层级逻辑一致。最终生成的索引为JSON格式（示例见项目tests/results目录下的PRML_structure.json），可直接用于检索。

2. 推理式检索PageIndex 采用的检索方法，让大模型基于文档树结构进行逻辑推理，逐步缩小检索范围，而非依赖简单的向量相似度匹配。原理

The core of reasoning-based retrieval is "multi-step reasoning based on the tree index." The specific process is as follows:

• 问题解析 (Query Parsing): The LLM transforms the user's question into a retrieval target (e.g., "Find 2023 Q4 net profit data" → Target: "net profit," "2023Q4").
• 根节点匹配 (Root Node Matching): Starting from the root node of the tree index (the document's main title), determine which first-level node (e.g., "Financial Data") is relevant to the target.
• 层级定位 (Hierarchical Localization): Drill down layer by layer into child nodes (e.g., "Financial Data" → "Quarterly Data" → "2023Q4"), filtering the most relevant leaf nodes using node summaries.
• 内容提取 (Content Extraction): Locate the page range corresponding to the leaf node, extract the original text content as the retrieval result, and return the reasoning path (e.g., "Financial Data → Quarterly Data → 2023Q4 → Page 28").

推理式检索PageIndex 采用的检索方法，让大模型基于文档树结构进行逻辑推理，逐步缩小检索范围，而非依赖简单的向量相似度匹配。的核心是“基于树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。的多步推理”，具体流程如下：

• 问题解析：LLM将用户问题转化为检索目标（如“找到2023年第四季度净利润数据”→目标：“净利润”“2023Q4”）；
• 根节点匹配：从树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。的根节点（文档总标题）出发，判断哪个一级节点（如“财务数据”）与目标相关；
• 层级定位：逐层深入子节点（如“财务数据”→“季度数据”→“2023Q4”），通过节点摘要筛选最相关的叶子节点；
• 内容提取：定位叶子节点对应的页码范围，提取原文内容作为检索结果，并返回推理路径（如“财务数据→季度数据→2023Q4→页码28”）。

3. 视觉检索PageIndex的功能之一，通过分析PDF页面中的图像特征（如表格结构、图表类型）生成视觉标签并关联到索引，支持用户直接检索图像内容，无需依赖OCR文字识别。实现

Visual retrieval is achieved through "image feature extraction + index association":

• The system performs feature analysis on each page image in the PDF (such as charts, formulas, diagrams) to generate "visual tags" (e.g., "bar chart, horizontal axis: quarter, vertical axis: revenue").
• These visual tags are associated with the corresponding page numbers and incorporated into the leaf nodes of the tree index.
• When a user's query involves visual content (e.g., "find the revenue trend line chart"), the LLM matches the visual tags to locate the corresponding page.

视觉检索PageIndex的功能之一，通过分析PDF页面中的图像特征（如表格结构、图表类型）生成视觉标签并关联到索引，支持用户直接检索图像内容，无需依赖OCR文字识别。通过“图像特征提取+索引关联”实现：

• 系统对PDF中的每一页图像（如图表、公式、示意图）进行特征分析，生成“视觉标签”（如“柱状图，横轴为季度，纵轴为营收”）；
• 将视觉标签与对应页码关联，纳入树状索引PageIndex的核心数据结构，通过自动识别文档的自然层级结构（如标题、段落、页码）生成，模拟人类阅读文档时先看目录再定位章节的逻辑。的叶子节点；
• 当用户提问涉及视觉内容时（如“找营收趋势折线图”），LLM会匹配视觉标签，定位对应页码。

四、典型应用场景

The "precise, transparent, and long-document-adapted" characteristics of PageIndex give it irreplaceable value in multiple professional fields. Typical scenarios include:

PageIndex的“精准、透明、长文档适配”特性使其在多个专业领域具备不可替代的价值，典型场景如下：

1. 金融领域：年报与研报分析 (Finance: Annual Report and Research Report Analysis)
   Financial professionals need to extract key data (e.g., net profit, asset-liability ratio) from hundreds-page-long上市公司 annual reports and industry research reports. Traditional RAG might mis-retrieve due to the semantic similarity between "profit growth" and "profit decline." PageIndex, however, directly locates the "Financial Statements" chapter through its tree index and accurately extracts target data combined with reasoning. For example, the Mafin 2.5 system based on PageIndex achieved a 98.7% accuracy rate in answering questions like "Calculate the 2023 gross margin" and "Explain the reason for changes in inventory turnover ratio" in the FinanceBench financial Q&A benchmark, far surpassing traditional tools.

   金融从业者需从数百页的上市公司年报、行业研报中提取关键数据（如净利润、资产负债率）。传统RAG可能因“利润增长”与“利润下滑”语义相似