Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Conversational RAG

Conversational RAG is a RAG pipeline that answers follow-up questions by first rewriting each new question, with the chat history, into a standalone question that the retriever can search.

Last updated: 08 Oct, 2026 · LangChain 1.4

People do not ask in complete sentences. After "What is scaled dot-product attention?" the next message is "Why is that needed?", and "that" means nothing to a vector search. This lesson builds the step that fixes it: condense the question with the history, then retrieve. It starts from the same four-page store as Building a RAG pipeline.

The history and the new question go to an LLM that condenses them into one standalone question. The retriever searches with that question. The answer prompt receives the history, the retrieved chunks and the new question, and the finished turn is appended to the history.

Why a follow-up fails on its own

A retriever sees one string. The string "Why is that needed?" carries no topic, so its embedding lands wherever vague questions land. The first run below prints what the retriever returns for each message with and without the history, so the difference is visible in the chunk ids.

Keeping the history

The history is a list of messages: a HumanMessage for each user turn and an AIMessage for each reply. They come from langchain_core.messages and can be dropped into a prompt template at a placeholder. For storage between sessions, see A chatbot with memory; here the list lives in a variable.

python
from langchain_core.messages import HumanMessage, AIMessage

history = []
history.extend([HumanMessage("What is scaled dot-product attention?"), AIMessage("It is an attention function that ...")])

Condensing the question

One model call turns the history and the new question into one question that makes sense alone. The instruction must forbid answering it: a model asked to "rewrite" a question with a long history in view will sometimes answer instead.

python
from langchain_core.prompts import ChatPromptTemplate

condense = ChatPromptTemplate.from_messages([
    ("system", "Rewrite the final question as one standalone question that makes sense without "
               "the conversation. Do not answer it. Return only the rewritten question."),
    ("placeholder", "{history}"),     # the earlier messages go here
    ("human", "{question}"),
])
standalone = (condense | model).invoke({"history": history, "question": "Why is that needed?"}).content

Answering with history and context

The answer prompt gets the history as well, so the model can use earlier turns, and the retrieved chunks as context. Only the search uses the condensed question; the model is shown the user's real words.

python
answer = ChatPromptTemplate.from_messages([
    ("system", "Answer in at most three sentences, using only the context. "
               "If the answer is not there, say so.\n\nContext:\n{context}"),
    ("placeholder", "{history}"),
    ("human", "{question}"),
])

A three-turn conversation

ExampleAPI keyStarts from the four-page store of the previous lesson; run on Groq openai/gpt-oss-120b
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import HumanMessage, AIMessage
from langchain.chat_models import init_chat_model

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
retriever = store.as_retriever(search_kwargs={"k": 2})
condense = ChatPromptTemplate.from_messages([
    ("system", "Rewrite the final question as one standalone question that makes sense without "
               "the conversation. Do not answer it. Return only the rewritten question."),
    ("placeholder", "{history}"),
    ("human", "{question}"),
])
answer = ChatPromptTemplate.from_messages([
    ("system", "Answer in at most three sentences, using only the context. "
               "If the answer is not there, say so.\n\nContext:\n{context}"),
    ("placeholder", "{history}"),
    ("human", "{question}"),
])
history = []

def chat(question):
    standalone = (condense | model).invoke({"history": history, "question": question}).content if history else question
    docs = retriever.invoke(standalone)
    context = "\n\n".join(d.page_content for d in docs)
    reply = (answer | model).invoke({"context": context, "history": history, "question": question}).content
    history.extend([HumanMessage(question), AIMessage(reply)])
    return standalone, [d.id for d in docs], reply

for q in ["What is scaled dot-product attention?", "Why is that needed?", "How does that change with several heads?"]:
    standalone, ids, reply = chat(q)
    print("YOU:", q)
    print("  searched for:", standalone)
    print("  chunks:", ids)
    print("  AI:", reply)
    print("  without rewriting:", [d.id for d in retriever.invoke(q)])

Reading the three turns

  • Turn 1 has no history, so the question is searched as typed and the chunks are the same with or without the rewrite step.
  • Turn 2, "Why is that needed?", became Why is scaled dot-product attention needed?. The retriever returned c2 and c1, the chunks about scaling. Without the rewrite it returned c14 first, a chunk of the QZhou paper, then c2.
  • Turn 3 became How does the scaling of dot-product attention change when using multiple attention heads.... The rewrite read "that" as the scaling, a reasonable but narrower reading, and retrieved c2 and c0; the raw question retrieved c3 and c2. Both lists contain c2, the chunk that introduces the heads, and the answer describes them. A rewrite can narrow a question, which is why the example prints the line searched for.
  • The answer prompt asked for at most three sentences, and each reply keeps to that. The history also goes into that prompt, so the model can build on the earlier turns.

Condensing vs sending the whole history to the retriever

Condense, then searchSearch with the whole history as text
What the retriever seesOne clear questionEvery earlier turn as well as the new one
Extra costOne short model call per turnNone, but a longer query to embed
RiskA bad rewrite searches for the wrong thingOld topics pull in unrelated chunks
Good whenUsers ask short follow-upsThe conversation stays on one topic

Where you use conversational RAG

  • Support chat over manuals. "And what about the warranty?" only makes sense with the product from the turn before.
  • Reading a paper with a chatbot. A reader keeps asking about "it", "that term" or "the second method".
  • Internal help desks. Employees ask a chain of short questions about one policy.
Watch out. The history list grows with every turn, and the condense call and the answer call both receive it. Keep only the last few turns (history[-6:]) or summarise old ones, and give each user their own list; a list shared between users leaks one person's questions into another's search.
Try it yourself
  • Print history after the loop and count the messages: two for each turn.
  • Remove the sentence "Do not answer it." from the condense prompt and read what the "searched for" line becomes.
  • Add a fourth turn, "What was my first question?", and read how the answer prompt uses the history.

Little by little, you're building something great.