RAG Interview Questions
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →

Q45HardScenario

You switched to a new embedding model and a new chunking strategy, and production quality dropped. How should you have managed this migration, and what now?

30-second answerSay your answer out loud first, then reveal.

Immediate actions

  1. Roll back: point traffic to the previous index and pipeline version. This requires that the old index wasn't deleted. (If it was, that's lesson #1.)
  2. Root-cause: evaluate four combinations: (old/new chunking) × (old/new embedder). Find which change hurt, and on which query types.
  3. Common culprits: the new model needs query/passage prefixes or a different similarity metric; the new chunk sizes exceed the model's token limit (truncation); different normalisation; BM25 tokenisation changed; metadata lost in the new pipeline.

Proper migration process (blue-green indexing)

Blue-green index migration: while index v1 keeps serving, build v2 in parallel and evaluate it offline; if worse, stop and investigate; if better, run shadow traffic, then a 5–10% canary, and when metrics are OK switch the alias to v2 while keeping v1 for a rollback window.

Principles

  • One change at a time (or a factorial test), so you can attribute effects.
  • Version everything: parser, chunker, embedder, index config. Store the pipeline version on each chunk.
  • Index aliases to switch atomically.
  • Cost and time planning: re-embedding millions of chunks takes time and money; run it in the background with checkpoints.
  • Segment-level evaluation: a change can improve the average but hurt critical segments (e.g. Hindi queries, tables).
PreviousDesign a documentation assistant that answers coding questions using the latest library docs and source code (version-aware).

Every expert started right here.