Building a RAG pipeline
A RAG pipeline is three steps in code: a retriever fetches the closest chunks, a prompt template puts them next to the question, and a chat model writes the answer from them.
Last updated: 08 Oct, 2026 · LangChain 1.4
With a vector store in place (Embeddings and vector stores), the query side of the pipeline is left. The examples in this lesson start from the same four pages of two papers, with the store already built; the setup lines are the ones shown in that lesson.
Creating a retriever from the store
The video writes a RAGRetriever class by hand: it embeds the query with the same model that embedded the chunks, asks the collection for the closest vectors and returns the documents with their metadata. LangChain gives the same behaviour in one call. store.as_retriever() returns a retriever, an object with invoke(question) that returns a list of Documents.
Choosing how many chunks to fetch
The number of chunks is k, and it goes inside search_kwargs. Writing as_retriever(k=2) looks right and is silently ignored, so the retriever returns its default instead. The next cell shows both.
retriever = store.as_retriever(search_kwargs={"k": 3}) # the three closest chunks
docs = retriever.invoke("Why are the dot products scaled?")print(len(store.as_retriever(k=2).invoke("attention")))
retriever = store.as_retriever(search_kwargs={"k": 2})
print(len(retriever.invoke("attention")))4 2
Reading the two retrievers
The first line prints 4: k=2 was ignored, and the retriever returned its default number of chunks. The second prints 2: search_kwargs={"k": 2} is the form that works.
Augmentation and generation
The video names the two steps that follow retrieval. Augmentation takes the retrieved context and combines it with a prompt that says how to behave. Generation sends the result to the model. In code both are a prompt template and a model call.
Writing the prompt template
A ChatPromptTemplate holds messages with {placeholders} that are filled at call time. The system message carries the rules, including the one that lets the model decline; the human message carries the context and the question. The context goes in as a value, so braces inside retrieved text, such as the formulas in these papers, cannot break it.
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_messages([
("system", "Answer the question using only the context. If the context does not contain "
"the answer, say you do not know. Cite the source like [file p.N]."),
("human", "Context:\n{context}\n\nQuestion: {question}"),
])Putting the chunks into the context
def format_docs(docs):
return "\n\n".join(
f"[{d.metadata['source']} p.{d.metadata['page']}]\n{d.page_content}" for d in docs
)The whole pipeline in one function
The video wraps retrieval, prompt and model call in one function, rag_simple. The version below does the same with the pieces above and joins them with the | operator: prompt | model is a chain that fills the template and sends the result to the model. The video's model is Groq's gemma2-9b-it, which Groq has retired; the example runs on openai/gpt-oss-120b.
The video then adds sources, the page of each chunk and a confidence value to the return. The function below returns the answer, the sources taken from chunk metadata and the top similarity score. The third question is not in the papers; it shows what the pipeline does when retrieval has nothing to offer.
from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
retriever = store.as_retriever(search_kwargs={"k": 2})
prompt = ChatPromptTemplate.from_messages([
("system", "Answer the question using only the context. If the context does not contain "
"the answer, say you do not know. Cite the source like [file p.N]."),
("human", "Context:\n{context}\n\nQuestion: {question}"),
])
def format_docs(docs):
return "\n\n".join(f"[{d.metadata['source']} p.{d.metadata['page']}]\n{d.page_content}" for d in docs)
def rag(question):
docs = retriever.invoke(question)
reply = (prompt | model).invoke({"context": format_docs(docs), "question": question})
top = store.similarity_search_with_score(question, k=1)[0][1]
sources = sorted({(d.metadata["source"], d.metadata["page"]) for d in docs})
return {"answer": reply.content, "sources": sources, "top_score": round(top, 3)}
for q in ["Why are the dot products divided by the square root of dk?",
"Which model is QZhou-Embedding built on?",
"Who won the 2022 football World Cup?"]:
out = rag(q)
print("Q:", q)
print("A:", out["answer"])
print("sources:", out["sources"], "top score:", out["top_score"])
print()Q: Why are the dot products divided by the square root of dk?
A: The dot‑product values become very large when the dimensionality \(d_k\) is big, which drives the softmax into regions where its gradients are tiny. Dividing the dot products by \(\sqrt{d_k}\) rescales them to a more moderate range, keeping the softmax stable and preserving useful gradients. [attention.pdf p.3]
sources: [('attention.pdf', 3)] top score: 0.686
Q: Which model is QZhou-Embedding built on?
A: QZhou‑Embedding is built on the **Qwen2.5‑7B‑Instruct** foundation model【qzhou.pdf p.0】.
sources: [('qzhou.pdf', 0)] top score: 0.797
Q: Who won the 2022 football World Cup?
A: I do not know.
sources: [('attention.pdf', 4), ('qzhou.pdf', 0)] top score: 0.532Reading the three answers
- The first answer explains the scaling and ends with the tag
[attention.pdf p.3]. The sources list has one entry because both chunks come from the same page; the page number counts from 0, as in the loaders. - The second answer names Qwen2.5-7B-Instruct and tags it with
【qzhou.pdf p.0】, in different brackets from the ones the prompt asked for. A model does not follow a format rule every time, so a program that needs the sources should read thesourcesvalue, which is built from metadata, and not parse the model's text. - The third question is not in the papers. The answer is
I do not know.The sources list still names two files because it records what was retrieved, not what the answer used, and the top score is 0.532, lower than 0.686 and 0.797 for the two answerable questions but not near zero. The prompt rule, not the score, produced the refusal.
Choosing k
| k = 1 | k = 3 or 4 | k = 10 | |
|---|---|---|---|
| Chance the right chunk is included | Lowest | Usually enough | Highest |
| Noise sent to the model | None | A little | Much, and the model can lose the point |
| Tokens and cost per question | Smallest | Moderate | Largest |
| Good when | Each fact sits in one chunk | A topic spans neighbouring chunks | You rerank down to a few before the prompt |
Where you use a RAG pipeline
- Policy and manual bots. The same function over HR, finance or product documents, with the file and page next to every answer.
- Paper and report assistants. Questions over a folder of PDFs, as this lesson does over two papers.
- The base for richer systems. Better retrieval, conversation history and agents all wrap this retrieve, augment, generate core.
I do not know. Check answers against the cited page, and measure the pipeline as in Evaluating RAG.Related
- Previous: Embeddings and vector stores
- Next: Better retrieval (hybrid search, reranking, query rewriting)
- Go deeper: Retrieval tool
- Change
kfrom 2 to 1 and ask the dot-product question again; read whether the answer still has what it needs. - Remove the sentence about saying you do not know from the system message and ask the World Cup question again.
- Print
format_docs(retriever.invoke(q))for the QZhou question to see exactly what the model was given.
This is what real progress feels like.