Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Building a RAG pipeline

A RAG pipeline is three steps in code: a retriever fetches the closest chunks, a prompt template puts them next to the question, and a chat model writes the answer from them.

Last updated: 08 Oct, 2026 · LangChain 1.4

With a vector store in place (Embeddings and vector stores), the query side of the pipeline is left. The examples in this lesson start from the same four pages of two papers, with the store already built; the setup lines are the ones shown in that lesson.

A question goes into retrieval, which returns the closest chunks. Augmentation fills a prompt template with rules, the chunks and the question. Generation sends that prompt to the model and returns an answer together with the source file and page of each chunk.

Creating a retriever from the store

The retriever: embed the query, search the vector store · from the Complete Agentic AI Course video · 6:21:54 to 6:25:40

The video writes a RAGRetriever class by hand: it embeds the query with the same model that embedded the chunks, asks the collection for the closest vectors and returns the documents with their metadata. LangChain gives the same behaviour in one call. store.as_retriever() returns a retriever, an object with invoke(question) that returns a list of Documents.

Choosing how many chunks to fetch

The number of chunks is k, and it goes inside search_kwargs. Writing as_retriever(k=2) looks right and is silently ignored, so the retriever returns its default instead. The next cell shows both.

python
retriever = store.as_retriever(search_kwargs={"k": 3})   # the three closest chunks
docs = retriever.invoke("Why are the dot products scaled?")
ExampleAPI keyStarts from the four-page store of the previous lesson; run on langchain-core 1.6.7
print(len(store.as_retriever(k=2).invoke("attention")))
retriever = store.as_retriever(search_kwargs={"k": 2})
print(len(retriever.invoke("attention")))

Reading the two retrievers

The first line prints 4: k=2 was ignored, and the retriever returned its default number of chunks. The second prints 2: search_kwargs={"k": 2} is the form that works.

Augmentation and generation

Augmentation and generation explained · from the Complete Agentic AI Course video · 6:32:44 to 6:35:48

The video names the two steps that follow retrieval. Augmentation takes the retrieved context and combines it with a prompt that says how to behave. Generation sends the result to the model. In code both are a prompt template and a model call.

Writing the prompt template

A ChatPromptTemplate holds messages with {placeholders} that are filled at call time. The system message carries the rules, including the one that lets the model decline; the human message carries the context and the question. The context goes in as a value, so braces inside retrieved text, such as the formulas in these papers, cannot break it.

python
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer the question using only the context. If the context does not contain "
               "the answer, say you do not know. Cite the source like [file p.N]."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])

Putting the chunks into the context

python
def format_docs(docs):
    return "\n\n".join(
        f"[{d.metadata['source']} p.{d.metadata['page']}]\n{d.page_content}" for d in docs
    )

The whole pipeline in one function

A simple RAG function: retrieve, build the prompt, call the model · from the Complete Agentic AI Course video · 6:37:59 to 6:42:29

The video wraps retrieval, prompt and model call in one function, rag_simple. The version below does the same with the pieces above and joins them with the | operator: prompt | model is a chain that fills the template and sends the result to the model. The video's model is Groq's gemma2-9b-it, which Groq has retired; the example runs on openai/gpt-oss-120b.

Returning an answer together with its sources and a score · from the Complete Agentic AI Course video · 6:43:08 to 6:46:27

The video then adds sources, the page of each chunk and a confidence value to the return. The function below returns the answer, the sources taken from chunk metadata and the top similarity score. The third question is not in the papers; it shows what the pipeline does when retrieval has nothing to offer.

ExampleAPI keyStarts from the four-page store of the previous lesson; run on Groq openai/gpt-oss-120b
from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
retriever = store.as_retriever(search_kwargs={"k": 2})
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer the question using only the context. If the context does not contain "
               "the answer, say you do not know. Cite the source like [file p.N]."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])

def format_docs(docs):
    return "\n\n".join(f"[{d.metadata['source']} p.{d.metadata['page']}]\n{d.page_content}" for d in docs)

def rag(question):
    docs = retriever.invoke(question)
    reply = (prompt | model).invoke({"context": format_docs(docs), "question": question})
    top = store.similarity_search_with_score(question, k=1)[0][1]
    sources = sorted({(d.metadata["source"], d.metadata["page"]) for d in docs})
    return {"answer": reply.content, "sources": sources, "top_score": round(top, 3)}

for q in ["Why are the dot products divided by the square root of dk?",
          "Which model is QZhou-Embedding built on?",
          "Who won the 2022 football World Cup?"]:
    out = rag(q)
    print("Q:", q)
    print("A:", out["answer"])
    print("sources:", out["sources"], "top score:", out["top_score"])
    print()

Reading the three answers

  • The first answer explains the scaling and ends with the tag [attention.pdf p.3]. The sources list has one entry because both chunks come from the same page; the page number counts from 0, as in the loaders.
  • The second answer names Qwen2.5-7B-Instruct and tags it with 【qzhou.pdf p.0】, in different brackets from the ones the prompt asked for. A model does not follow a format rule every time, so a program that needs the sources should read the sources value, which is built from metadata, and not parse the model's text.
  • The third question is not in the papers. The answer is I do not know. The sources list still names two files because it records what was retrieved, not what the answer used, and the top score is 0.532, lower than 0.686 and 0.797 for the two answerable questions but not near zero. The prompt rule, not the score, produced the refusal.

Choosing k

k = 1k = 3 or 4k = 10
Chance the right chunk is includedLowestUsually enoughHighest
Noise sent to the modelNoneA littleMuch, and the model can lose the point
Tokens and cost per questionSmallestModerateLargest
Good whenEach fact sits in one chunkA topic spans neighbouring chunksYou rerank down to a few before the prompt

Where you use a RAG pipeline

  • Policy and manual bots. The same function over HR, finance or product documents, with the file and page next to every answer.
  • Paper and report assistants. Questions over a folder of PDFs, as this lesson does over two papers.
  • The base for richer systems. Better retrieval, conversation history and agents all wrap this retrieve, augment, generate core.
Watch out. A source list shows which chunks the model was given, not that the answer is correct or that the model used them: in the run above the list names two files next to the answer I do not know. Check answers against the cited page, and measure the pipeline as in Evaluating RAG.
Try it yourself
  • Change k from 2 to 1 and ask the dot-product question again; read whether the answer still has what it needs.
  • Remove the sentence about saying you do not know from the system message and ask the World Cup question again.
  • Print format_docs(retriever.invoke(q)) for the QZhou question to see exactly what the model was given.

This is what real progress feels like.