How do you handle follow-up questions in a conversational RAG chatbot?

30-second answerSay your answer out loud first, then reveal.

Follow-ups like "what about for contractors?" are meaningless on their own to a retriever. Before retrieval, use an LLM to condense the chat history plus the follow-up into a standalone query ("What is the leave policy for contractors?"), retrieve with that, then answer using the retrieved context plus recent history. Also decide when retrieval isn't needed at all ("thanks!", or rephrasing the previous answer).

The chat history and the follow-up go to a condense LLM that writes a standalone question, which is used for retrieval, and generation uses the retrieved context plus the follow-up and recent history.

Design points

Query condensation: cheap, fast model; few-shot examples; keep the user's language.
Retrieval decision: classify whether new retrieval is needed (new information need) or the turn is chit-chat or a formatting request on the previous answer.
History management: keep the last N turns verbatim plus a summary of older turns; don't stuff full past contexts back in.
Context carry-over: sometimes reuse previously retrieved chunks plus new ones (e.g. "explain point 3 more").
Entity tracking: "he", "that product" resolved via condensation.

Failure modes

Condensation drops important constraints ("for India" from three turns ago).
Topic switches incorrectly blended with old context.
Evaluate on multi-turn test conversations, not only single questions.

PreviousHow do you handle tables and structured data in RAG?

This is what real progress feels like.

How do you handle follow-up questions in a conversational RAG chatbot?

Design points

Failure modes

Related