Vectorless RAG
Vectorless RAG is a retrieval method without embeddings: an LLM builds a table-of-contents tree of a document once, and at question time an LLM reads that tree to choose which sections to read and answer from.
Last updated: 08 Oct, 2026 · LangChain 1.4
Vector RAG cuts a document into fixed-size chunks and searches their vectors. That can split a section in the middle and returns a similarity score that explains little. The video's second approach, built around the PageIndex library, keeps whole sections and lets a model navigate them the way a person uses a table of contents.
How the tree index works
The video starts from a PDF that has a table of contents: chapters with page numbers, subsections under them. An LLM tree builder turns that into a hierarchy of nodes saved as a JSON tree index. Each node stands for a section and holds its title and page range, and a short summary of its content written by the LLM. The PageIndex tree also keeps the section text on the node, so a search prompt can show the model the summaries and the answer step can read the full text.
At question time the model is given the whole tree as context, decides which nodes are likely to hold the answer, and the text of those sections is passed to a model that writes the answer with section and page citations. The video's human analogy fits: open the book at the table of contents, find the page, read it. No database server is involved, and the tree can live in a file, an object store or a document database.
When the PDF has no table of contents
Many PDFs have no table of contents. The video's answer is that an LLM reads the pages, infers the headings and splits at logical boundaries, not at a token count, so a section is not cut into pieces the way a fixed-size chunker would. The sections still correspond to page ranges, so this is chunking by structure rather than no splitting at all. The video then lists the retrieval loop: read the tree, reason about which nodes to select, extract their content, and check whether it is enough to answer; if it is not, go back to selecting. The code below makes one pass; the sufficiency check can be added as one more model call.
The video's run with the PageIndex library
The video runs the PageIndex cloud service on a long syllabus PDF with an OpenAI model: it uploads the file, waits for the tree, prints it and counts the nodes, then writes a tree-search prompt, selects the nodes for a question and generates a cited answer. Shown as it ran in the video, not run here: it needs a PageIndex key and an OpenAI key. The current library names a node's pages start_index and end_index, where the video's notebook reads page_index.
from pageindex import PageIndexClient # pip install pageindex
client = PageIndexClient(api_key="YOUR_PAGEINDEX_KEY")
doc_id = client.submit_document("report.pdf")["doc_id"] # upload; the tree is built in the background
# poll client.get_document(doc_id)["status"] until it is "completed"
tree = client.get_tree(doc_id, node_summary=True)["result"] # nodes with titles, ids, summaries, pagesThe same three steps work with only the course keys if the tree is built by the chat model. The example builds a small tree of the Transformer paper, asks the model which sections answer a question, fetches those pages and asks for a cited answer.
Building a tree with the chat model
What a section looks like
from pydantic import BaseModel
class Section(BaseModel):
node_id: str
title: str
start_page: int
end_page: int
summary: str
class Tree(BaseModel):
sections: list[Section]Showing the model each page
A real tree builder reads whole pages. To keep the request small, the model here is shown, for every page, its first 150 characters and any numbered heading lines that a regular expression finds, such as 3.2.1 Scaled Dot-Product Attention. Opening text alone gives the model little sign of where the paper's sections begin; the headings let it start a section where the paper does, and the last try-it removes them so that you can compare.
import re
def digest_line(i, text):
headings = re.findall(r"^\d+(?:\.\d+)* [A-Z][^\n]{3,70}$", text, flags=re.M)
return f"[page {i + 1}] starts: {text[:150]!r} headings: {headings}"
digest = "\n".join(digest_line(i, p) for i, p in enumerate(pages))Asking the model for the tree
The model returns a table of contents as the Tree schema. method="json_schema" makes Groq return JSON that fits it.
tree = model.with_structured_output(Tree, method="json_schema").invoke(
"Build a table of contents of 8 to 12 sections. Each section starts on the page where its heading "
"appears and ends on the page where the next section starts...\n\n" + digest
).model_dump()Searching the tree and answering with pages
pick = model.with_structured_output(Pick, method="json_schema").invoke(
"Question: " + question + "\nTable of contents:\n" + json.dumps(tree["sections"])
+ "\nWhich sections most likely contain the answer? Return their ids."
)
chosen = [s for s in tree["sections"] if s["node_id"] in pick.node_ids]A tree of the Transformer paper, from build to cited answer
import os, re, json, urllib.request
from pydantic import BaseModel
from langchain_community.document_loaders import PyPDFLoader
from langchain.chat_models import init_chat_model
if not os.path.exists("attention.pdf"):
urllib.request.urlretrieve("https://arxiv.org/pdf/1706.03762", "attention.pdf")
pages = [d.page_content for d in PyPDFLoader("attention.pdf").load()]
model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
class Section(BaseModel):
node_id: str
title: str
start_page: int
end_page: int
summary: str
class Tree(BaseModel):
sections: list[Section]
class Pick(BaseModel):
thinking: str
node_ids: list[str]
def digest_line(i, text):
headings = re.findall(r"^\d+(?:\.\d+)* [A-Z][^\n]{3,70}$", text, flags=re.M) # lines like "3.2.1 Scaled Dot-Product Attention"
return f"[page {i + 1}] starts: {text[:150]!r} headings: {headings}"
digest = "\n".join(digest_line(i, p) for i, p in enumerate(pages))
tree = model.with_structured_output(Tree, method="json_schema").invoke(
"Below is the opening text and the numbered headings found on each page of a paper. Build a table of "
"contents of 8 to 12 sections. Each section starts on the page where its heading appears and ends on the "
"page where the next section starts; use the heading numbers in the titles. Give each a one-sentence summary "
"and ids 0001, 0002, ...\n\n" + digest
).model_dump()
for s in tree["sections"]:
print(s["node_id"], f"p.{s['start_page']}-{s['end_page']}", s["title"], "|", s["summary"])
question = "Why does the Transformer divide the dot products by the square root of the key dimension?"
pick = model.with_structured_output(Pick, method="json_schema").invoke(
"Question: " + question + "\nTable of contents:\n" + json.dumps(tree["sections"])
+ "\nWhich sections most likely contain the answer? Return their ids."
)
print(pick)
chosen = [s for s in tree["sections"] if s["node_id"] in pick.node_ids]
text = "\n".join(f"[page {n}]\n{pages[n - 1]}" for s in chosen for n in range(s["start_page"], s["end_page"] + 1))
print("pages read:", sorted({n for s in chosen for n in range(s["start_page"], s["end_page"] + 1)}), "characters:", len(text))
reply = model.invoke("Answer the question using only the pages below. Cite page numbers like (p.N). Max three sentences."
"\n\nQuestion: " + question + "\n\n" + text)
print(reply.content)0001 p.2-2 1 Introduction | Introduces the problem domain and outlines the paper's contributions.
0002 p.2-2 2 Background | Reviews prior work on recurrent networks and sets context for the Transformer.
0003 p.2-3 3 Model Architecture | Describes the overall Transformer architecture and its components.
0004 p.3-3 3.1 Encoder and Decoder Stacks | Details the stacked encoder and decoder layers used in the model.
0005 p.3-6 3.2 Attention | Explains the attention mechanism central to the Transformer.
0006 p.6-7 4 Why Self-Attention | Motivates the use of self‑attention over recurrent approaches.
0007 p.7-8 5 Training | Covers the training data, batching strategy, hardware, optimizer, and regularization.
0008 p.8-10 6 Results | Presents experimental results on machine translation and parsing tasks.
0009 p.10-15 7 Conclusion | Summarizes findings and suggests future directions.
thinking='The question asks about the scaling factor in the dot‑product attention, which is explained in the paper’s description of the attention mechanism. The relevant discussion is in the section titled “3.2 Attention”. The broader model architecture section may mention it, but the detailed rationale is in the attention subsection. Therefore the node ID for that section (0005) is the most likely source of the answer.' node_ids=['0005']
pages read: [3, 4, 5, 6] characters: 10984
The dot‑product of a query and a key grows in magnitude as the key dimension \(d_k\) increases, which can push the softmax into regions where its gradients become extremely small and learning stalls. To keep the variance of the dot products stable, the authors scale them by \(1/\sqrt{d_k}\) before applying the softmax, thereby preventing this saturation effect (p.4). This scaling thus ensures more effective gradient flow for larger key dimensions (p.4).Reading the tree, the choice and the answer
- The tree has nine sections, each with an id, a page range and a one-sentence summary. The titles carry the heading numbers (
3.2 Attention) because the digest handed them to the model. - The ranges are approximate. They overlap where a section begins in the middle of a page:
0003covers pages 2 to 3,0004page 3,0005pages 3 to 6. A tree built from a page-by-page digest is a guide to where to read, not an exact map. - The model chose one node,
0005, and itsthinkingfield says why: the scaling factor is explained in the attention description, section 3.2. The choice and the reasoning are visible, which a similarity score does not give. - Pages 3 to 6 were read, 10,984 characters, the whole section and not a slice of it.
- The answer cites page 4 twice. That is the page of the PDF where the paper explains the scaling, so the citation can be checked by opening it.
Choosing between vector and vectorless RAG
The video's guidance, in a table: traditional RAG for very large and mixed collections, tight latency and short factual questions; vectorless for long structured documents such as annual reports and contracts, where explaining the path matters. It ends with the hybrid view: scale from vectors plus structure from the tree.
| Vector RAG | Vectorless RAG | |
|---|---|---|
| What is indexed | Fixed-size chunks and their embeddings | A tree of sections with titles, page ranges and summaries |
| Work at question time | One embedding and one search | At least one model call to read the tree, one more to answer |
| Collection size | Millions of mixed documents | One to thousands of long documents |
| Why a result was chosen | A similarity score | The model's stated reasoning and a section path |
| Weak spot | A section split across chunks loses context | More latency and cost per question; needs documents with structure |
| Typical use | Support articles, tickets, web pages | Annual reports, legal contracts, long manuals |
Where you use vectorless RAG
- Annual reports and filings. A question about one section of a 200-page document, where the answer needs the whole section.
- Contracts and policies. The reader wants the clause and its page, and the path to it.
- Long manuals. The structure is already a table of contents, so the tree is cheap to build.
Related
- Previous: Agentic RAG
- Next: Evaluating RAG
- Reference: PageIndex repository
- Change the question to "How many attention heads does the model use?" and read which sections the model picks.
- Remove
headingsfromdigest_lineand compare the section titles and page ranges with the first tree. - Ask for 14 to 18 sections instead of 8 to 12 and see how the page ranges narrow.
Slow is fine. Stopping is the only problem.