When and how would you fine-tune an embedding model or reranker for your domain?

30-second answerSay your answer out loud first, then reveal.

Fine-tune when evaluation shows that off-the-shelf models miss domain-specific relevance (jargon, acronyms, internal product names, specialised language) and simpler fixes (hybrid, rewriting, contextual chunks) don't close the gap. Build (query, relevant passage) pairs from logs, click data or synthetic LLM-generated questions, mine hard negatives, train with a contrastive loss, and verify gains on a held-out set. Remember that changing the embedding model means re-indexing.

Training data

Positive pairs: (query, relevant chunk) from search logs or clicks, support tickets with linked KB articles, expert labels, or synthetic questions generated per chunk by an LLM (cheap and effective).
Hard negatives: passages that look similar but aren't relevant (e.g. retrieved by BM25 or the base model but not correct). They teach fine distinctions, and they matter more than the dataset's size.

Training

Contrastive loss, such as MultipleNegativesRankingLoss with in-batch negatives (Sentence-Transformers).
Rerankers: train a cross-encoder on (query, passage, label) with hard negatives.
Start from a strong base model; a few thousand quality pairs can give meaningful gains.

Evaluation

Held-out queries (no overlap with training docs if possible).
Recall@k, nDCG before and after; also check that general queries didn't regress.

Reranker vs embedder fine-tuning

	Fine-tune reranker	Fine-tune embedder
Re-indexing needed	No	Yes (all vectors)
Impact	Top-of-list precision	Recall of the first stage
Risk	Low	Index migration effort

Good order: fine-tune the reranker first (no re-index), and the embedder if first-stage recall is the bottleneck.

PreviousAn attacker uploads a document containing hidden instructions that get retrieved into the prompt. What are the risks, and how do you defend?

You understood something today that you didn't yesterday.

When and how would you fine-tune an embedding model or reranker for your domain?

Training data

Training

Evaluation

Reranker vs embedder fine-tuning

Related