Conversational RAG
Conversational RAG is a RAG pipeline that answers follow-up questions by first rewriting each new question, with the chat history, into a standalone question that the retriever can search.
Last updated: 08 Oct, 2026 · LangChain 1.4
People do not ask in complete sentences. After "What is scaled dot-product attention?" the next message is "Why is that needed?", and "that" means nothing to a vector search. This lesson builds the step that fixes it: condense the question with the history, then retrieve. It starts from the same four-page store as Building a RAG pipeline.
Why a follow-up fails on its own
A retriever sees one string. The string "Why is that needed?" carries no topic, so its embedding lands wherever vague questions land. The first run below prints what the retriever returns for each message with and without the history, so the difference is visible in the chunk ids.
Keeping the history
The history is a list of messages: a HumanMessage for each user turn and an AIMessage for each reply. They come from langchain_core.messages and can be dropped into a prompt template at a placeholder. For storage between sessions, see A chatbot with memory; here the list lives in a variable.
from langchain_core.messages import HumanMessage, AIMessage
history = []
history.extend([HumanMessage("What is scaled dot-product attention?"), AIMessage("It is an attention function that ...")])Condensing the question
One model call turns the history and the new question into one question that makes sense alone. The instruction must forbid answering it: a model asked to "rewrite" a question with a long history in view will sometimes answer instead.
from langchain_core.prompts import ChatPromptTemplate
condense = ChatPromptTemplate.from_messages([
("system", "Rewrite the final question as one standalone question that makes sense without "
"the conversation. Do not answer it. Return only the rewritten question."),
("placeholder", "{history}"), # the earlier messages go here
("human", "{question}"),
])
standalone = (condense | model).invoke({"history": history, "question": "Why is that needed?"}).contentAnswering with history and context
The answer prompt gets the history as well, so the model can use earlier turns, and the retrieved chunks as context. Only the search uses the condensed question; the model is shown the user's real words.
answer = ChatPromptTemplate.from_messages([
("system", "Answer in at most three sentences, using only the context. "
"If the answer is not there, say so.\n\nContext:\n{context}"),
("placeholder", "{history}"),
("human", "{question}"),
])A three-turn conversation
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import HumanMessage, AIMessage
from langchain.chat_models import init_chat_model
model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
retriever = store.as_retriever(search_kwargs={"k": 2})
condense = ChatPromptTemplate.from_messages([
("system", "Rewrite the final question as one standalone question that makes sense without "
"the conversation. Do not answer it. Return only the rewritten question."),
("placeholder", "{history}"),
("human", "{question}"),
])
answer = ChatPromptTemplate.from_messages([
("system", "Answer in at most three sentences, using only the context. "
"If the answer is not there, say so.\n\nContext:\n{context}"),
("placeholder", "{history}"),
("human", "{question}"),
])
history = []
def chat(question):
standalone = (condense | model).invoke({"history": history, "question": question}).content if history else question
docs = retriever.invoke(standalone)
context = "\n\n".join(d.page_content for d in docs)
reply = (answer | model).invoke({"context": context, "history": history, "question": question}).content
history.extend([HumanMessage(question), AIMessage(reply)])
return standalone, [d.id for d in docs], reply
for q in ["What is scaled dot-product attention?", "Why is that needed?", "How does that change with several heads?"]:
standalone, ids, reply = chat(q)
print("YOU:", q)
print(" searched for:", standalone)
print(" chunks:", ids)
print(" AI:", reply)
print(" without rewriting:", [d.id for d in retriever.invoke(q)])YOU: What is scaled dot-product attention? searched for: What is scaled dot-product attention? chunks: ['c0', 'c1'] AI: Scaled Dot‑Product Attention takes queries Q, keys K, and values V, computes the dot products QKᵀ, scales them by 1/√dₖ, applies a softmax to obtain attention weights, and then multiplies by V to produce the output. This operation can be performed on batches of queries, keys, and values packed as matrices. It is the core attention mechanism used in the Transformer architecture. without rewriting: ['c0', 'c1'] YOU: Why is that needed? searched for: Why is scaled dot‑product attention needed? chunks: ['c2', 'c1'] AI: Because the dot products between queries and keys grow with the dimension dₖ, they can become so large that the softmax falls into regions with extremely small gradients, making training unstable. Scaling the dot products by 1/√dₖ keeps their magnitude in a range where the softmax retains useful gradients. This stabilizes and speeds up the attention computation. without rewriting: ['c14', 'c2'] YOU: How does that change with several heads? searched for: How does the scaling of dot‑product attention change when using multiple attention heads in a multi‑head attention mechanism? chunks: ['c2', 'c0'] AI: In Multi‑Head Attention each head first linearly projects the original queries, keys and values into its own dₖ‑ and dᵥ‑dimensional subspaces, then applies the same Scaled Dot‑Product Attention (dot‑product, divide by √dₖ, softmax, multiply by V) in parallel on those projections. The resulting dᵥ‑dimensional outputs from all heads are computed independently and then combined (e.g., concatenated) to form the final multi‑head result. This lets the model attend to information from multiple representation subspaces simultaneously. without rewriting: ['c3', 'c2']
Reading the three turns
- Turn 1 has no history, so the question is searched as typed and the chunks are the same with or without the rewrite step.
- Turn 2, "Why is that needed?", became
Why is scaled dot-product attention needed?. The retriever returned c2 and c1, the chunks about scaling. Without the rewrite it returned c14 first, a chunk of the QZhou paper, then c2. - Turn 3 became
How does the scaling of dot-product attention change when using multiple attention heads.... The rewrite read "that" as the scaling, a reasonable but narrower reading, and retrieved c2 and c0; the raw question retrieved c3 and c2. Both lists contain c2, the chunk that introduces the heads, and the answer describes them. A rewrite can narrow a question, which is why the example prints the linesearched for. - The answer prompt asked for at most three sentences, and each reply keeps to that. The history also goes into that prompt, so the model can build on the earlier turns.
Condensing vs sending the whole history to the retriever
| Condense, then search | Search with the whole history as text | |
|---|---|---|
| What the retriever sees | One clear question | Every earlier turn as well as the new one |
| Extra cost | One short model call per turn | None, but a longer query to embed |
| Risk | A bad rewrite searches for the wrong thing | Old topics pull in unrelated chunks |
| Good when | Users ask short follow-ups | The conversation stays on one topic |
Where you use conversational RAG
- Support chat over manuals. "And what about the warranty?" only makes sense with the product from the turn before.
- Reading a paper with a chatbot. A reader keeps asking about "it", "that term" or "the second method".
- Internal help desks. Employees ask a chain of short questions about one policy.
history[-6:]) or summarise old ones, and give each user their own list; a list shared between users leaks one person's questions into another's search.Related
- Previous: Better retrieval (hybrid search, reranking, query rewriting)
- Next: Agentic RAG
- Go deeper: RAG
- Print
historyafter the loop and count the messages: two for each turn. - Remove the sentence "Do not answer it." from the condense prompt and read what the "searched for" line becomes.
- Add a fourth turn, "What was my first question?", and read how the answer prompt uses the history.
Little by little, you're building something great.