How do you prompt the LLM so it answers only from retrieved context and cites sources?

Example prompt structure

System: You are a support assistant for Acme. Answer using ONLY the
documents below. If they don't contain the answer, say you don't know
and suggest contacting support. Cite sources as [doc_id] after each
claim. If documents conflict, prefer the most recent and mention it.

<documents>
<doc id="D12" title="Refund Policy" updated="2026-08-01">...</doc>
<doc id="D7"  title="Shipping FAQ" updated="2026-05-10">...</doc>
</documents>

Question: Can I return an opened item?

Techniques

1. Explicit abstention ("I don't know") reduces made-up answers. Test that it actually triggers on unanswerable questions.
2. Citations with IDs the model copies, then validate them: does the cited doc ID exist in the context? Does the cited passage support the claim? (An NLI model or LLM judge can check.)
3. Quote extraction first: for high-stakes domains, ask the model to extract relevant quotes and then answer from the quotes.
4. Structured output: {answer, citations: [{doc_id, quote}], confidence} for easier verification and UI rendering.
5. Conflict handling instructions (see Q32).

Common mistakes

• Not testing unanswerable questions. Most eval sets only contain answerable ones, so over-confident systems look fine.
• Trusting citations without validation. Models sometimes cite the wrong chunk.

Related

• Previous: Q10. How do metadata and filtering improve retrieval?
• Next: Q12. RAG vs fine-tuning vs long context: when do you use each?
• LangChain
• LLM Fundamentals