Curious builder0 XP earned · 300 to level 2

0 daysFinish a lesson to begin

Badge collection0 of 6 unlocked

28 small wins to finish your pathNext lesson →

Faithfulness

Faithfulness is a RAGAS metric that measures how much of an answer is supported by the retrieved context: the share of the answer's claims that the chunks back up, from 0 to 1.

Last updated: 29 Sep, 2026 · RAGAS 0.4.3

A RAG bot that adds a detail its documents never said is hallucinating, and for a support bot that can mean a promise the company never made. Faithfulness is the metric that catches it.

Faithfulness claim by claim · from the Production RAG Live Marathon · 224:53 to 229:59

Checking each claim against the chunks

The video keeps four things in view for any RAG metric: the question, the retrieved context, the generated answer, and the reference. Faithfulness uses two of them: the retrieved context and the answer. It does not look at the reference.

The judge breaks the answer into atomic claims, then asks one question about each: can this claim be fully inferred from the retrieved context? In the video's bank example, the answer makes four claims. The urban minimum balance, the non-maintenance fee and the rural minimum are all in the chunks: grounded. "You can also transfer funds online at no extra charge" is in no chunk: hallucinated. Three grounded out of four gives 0.75. With a pass mark of 0.8, this answer fails.

The Faithfulness API

python

from ragas.metrics.collections import Faithfulness

faithfulness = Faithfulness(llm=judge)
result = faithfulness.score(user_input=..., response=..., retrieved_contexts=[...])
result.value  # supported claims / all claims

The bank chunks

python

chunks = [
    "Minimum balance is ₹10,000 for urban branches. Non-maintenance fee is ₹350 + GST if balance falls below.",
    "Semi-urban branch minimum balance is ₹5,000. Rural branch minimum balance is ₹2,500.",
]

The answer with one hallucinated claim

python

response = ("The minimum balance for urban branches is ₹10,000. The non-maintenance fee charged is ₹350 + GST. "
            "You can also transfer funds online at no extra charge. For rural branches the minimum is ₹2,500.")

Scoring against a threshold

python

result = Faithfulness(llm=judge).score(user_input=question, response=response, retrieved_contexts=chunks)
print("PASS" if result.value >= 0.8 else "FAIL")

Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.

written in LLM as a judge
written in TechNest RAG app
written in TechNest RAG app

View the code here

judge.py

import os

from google import genai
from openai import AsyncOpenAI
from ragas.embeddings import GoogleEmbeddings
from ragas.llms import llm_factory

groq = AsyncOpenAI(
    api_key=os.environ.get("JUDGE_GROQ", os.environ["GROQ_API_KEY"]),
    base_url="https://api.groq.com/openai/v1",
)
judge = llm_factory("openai/gpt-oss-20b", provider="openai", client=groq)


class OneTextPerCall(GoogleEmbeddings):
    """gemini-embedding-2 turns a list into one embedding, so embed each text on its own."""

    def embed_texts(self, texts, **kwargs):
        return [self.embed_text(text) for text in texts]

    async def aembed_texts(self, texts, **kwargs):
        return [await self.aembed_text(text) for text in texts]


embeddings = OneTextPerCall(client=genai.Client(), model="gemini-embedding-2")

technest.py

import json
import os

import numpy as np
from google import genai
from openai import OpenAI

gemini = genai.Client()  # reads GOOGLE_API_KEY
groq = OpenAI(api_key=os.environ["GROQ_API_KEY"], base_url="https://api.groq.com/openai/v1")
EMBED_MODEL = "gemini-embedding-2"
CHAT_MODEL = "qwen/qwen3.8-27b"

SYSTEM_PROMPT = """You are a helpful customer support assistant for TechNest, an online electronics store.
Answer the customer's question using ONLY the information provided in the context below.
If the context does not contain enough information to answer fully, say so honestly.
Keep your answer concise, factual, and friendly. Do not invent any details not present in the context.
Reply in two or three plain sentences, with no lists or tables."""

with open("catalog.json", encoding="utf-8") as f:
    CATALOG = json.load(f)


def embed(texts):
    # gemini-embedding-2 turns everything in one call into one embedding, so send one text per call
    vectors = np.array([gemini.models.embed_content(model=EMBED_MODEL, contents=t).embeddings[0].values for t in texts])
    return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)


DOC_VECTORS = embed([f"{item['title']}. {item['content']}" for item in CATALOG])


def retrieve(question, top_k=3):
    scores = DOC_VECTORS @ embed([question])[0]
    best = np.argsort(scores)[::-1][:top_k]
    return [CATALOG[i]["content"] for i in best]


def generate(question, contexts):
    context_block = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(contexts))
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context_block}\n\nCustomer question: {question}"},
    ]
    response = groq.chat.completions.create(model=CHAT_MODEL, messages=messages, temperature=0)
    return response.choices[0].message.content.strip()


def answer(question, top_k=3):
    contexts = retrieve(question, top_k)
    return generate(question, contexts), contexts

catalog.json

[
  {
    "id": "prod_001",
    "category": "product",
    "title": "ProBook X1 Laptop",
    "content": "The TechNest ProBook X1 is a 14-inch laptop featuring an Intel Core i7-13th Gen processor, 16GB DDR5 RAM, and a 512GB NVMe SSD. It has a battery life of 12 hours, weighs 1.4kg, and comes with a backlit keyboard. Price: $1,299. Includes a 2-year manufacturer warranty."
  },
  {
    "id": "prod_002",
    "category": "product",
    "title": "PixelPhone 15",
    "content": "The TechNest PixelPhone 15 is a 6.7-inch AMOLED smartphone with a 50MP triple camera system, 8GB RAM, 256GB storage, and a 5,000mAh battery supporting 65W fast charging. Available in Midnight Black and Arctic White. Price: $899. Includes a 1-year warranty."
  },
  {
    "id": "prod_003",
    "category": "product",
    "title": "SoundPods Pro",
    "content": "The TechNest SoundPods Pro are true wireless earbuds with active noise cancellation (ANC), 8 hours of playback per charge plus 24 hours with the case, and IPX4 water resistance. They connect via Bluetooth 5.3 and support multipoint pairing with two devices simultaneously. Price: $149."
  },
  {
    "id": "prod_004",
    "category": "product",
    "title": "UltraTab S2 Tablet",
    "content": "The TechNest UltraTab S2 is a 11-inch tablet powered by a Snapdragon 870 processor with 8GB RAM and 128GB storage expandable via microSD. It features a 120Hz display, a 7,500mAh battery, and supports the TechNest Stylus Pen sold separately. Price: $549. Includes a 1-year warranty."
  },
  {
    "id": "prod_005",
    "category": "product",
    "title": "SmartWatch X",
    "content": "The TechNest SmartWatch X features continuous heart rate monitoring, SpO2 tracking, GPS, and 7-day battery life. It is water-resistant up to 50 metres. Compatible with both Android and iOS. Price: $299. Includes a 1-year warranty and a free extra silicone band."
  },
  {
    "id": "prod_006",
    "category": "product",
    "title": "ProCam 4K Action Camera",
    "content": "The TechNest ProCam 4K shoots 4K video at 60fps and 20MP photos. It is waterproof up to 10 metres without a case, has built-in image stabilisation (EIS), and includes a touch screen. Battery life is 90 minutes of 4K recording. Price: $229. Includes a 1-year warranty."
  },
  {
    "id": "prod_007",
    "category": "product",
    "title": "BassBuds Max Headphones",
    "content": "The TechNest BassBuds Max are over-ear wireless headphones with 40-hour battery life, hybrid active noise cancellation, and a premium 40mm driver for deep bass. They fold flat for travel and include a carrying case. Price: $199. Compatible with all Bluetooth devices."
  },
  {
    "id": "prod_008",
    "category": "product",
    "title": "SoundBar 360",
    "content": "The TechNest SoundBar 360 is a 2.1 soundbar with a 120W output, built-in subwoofer, Dolby Atmos support, and HDMI ARC connectivity. It also supports Bluetooth streaming and has an optical audio input. Dimensions: 90cm wide. Price: $349. Includes a 2-year warranty."
  },
  {
    "id": "policy_001",
    "category": "policy",
    "title": "Return Policy",
    "content": "TechNest accepts returns within 30 days of the original purchase date. Items must be in their original packaging with all accessories included. Customers are responsible for return shipping costs unless the item arrives defective or damaged. Refunds are processed within 5 to 7 business days of receiving the returned item. Digital downloads and opened software are non-refundable."
  },
  {
    "id": "policy_002",
    "category": "policy",
    "title": "Shipping Policy",
    "content": "TechNest offers free standard shipping on all orders over $50 within the continental US. Standard shipping takes 3 to 5 business days. Expedited shipping (1 to 2 business days) is available for $9.99. Same-day delivery is available in select cities for $19.99. Orders placed before 2pm local time are dispatched the same day."
  },
  {
    "id": "policy_003",
    "category": "policy",
    "title": "Warranty Policy",
    "content": "All TechNest products include a minimum 1-year manufacturer warranty covering defects in materials and workmanship. The ProBook X1 and SoundBar 360 include a 2-year warranty. Warranty does not cover physical damage, water damage (unless the product is rated waterproof), or damage from unauthorised modifications. To make a warranty claim, contact support@technest.com with your order number and a description of the issue."
  },
  {
    "id": "policy_004",
    "category": "policy",
    "title": "Payment Policy",
    "content": "TechNest accepts Visa, Mastercard, American Express, PayPal, and Apple Pay. All transactions are encrypted using 256-bit SSL. Buy Now Pay Later is available via Klarna for orders over $100, with 0% interest for 3 monthly instalments. TechNest does not store full card details — payments are processed securely by Stripe."
  },
  {
    "id": "faq_001",
    "category": "faq",
    "title": "Order Tracking",
    "content": "To track your order, visit technest.com/orders and enter your order number and email address. A shipping confirmation email with a tracking link is sent within 24 hours of dispatch. If you have not received your tracking email after 48 hours, check your spam folder or contact support@technest.com."
  },
  {
    "id": "faq_002",
    "category": "faq",
    "title": "International Shipping",
    "content": "TechNest ships to over 40 countries. International shipping rates start at $14.99 and delivery takes 7 to 14 business days. Import duties and taxes are the responsibility of the customer and are not included in the product price. Free shipping promotions apply to US orders only."
  },
  {
    "id": "faq_003",
    "category": "faq",
    "title": "Bulk and Business Orders",
    "content": "TechNest offers volume discounts for businesses purchasing 10 or more units of any single product. Discounts range from 10% for 10 to 49 units up to 25% for 100 or more units. Contact business@technest.com with your requirements for a custom quote. A dedicated account manager is assigned for orders over $10,000."
  }
]

The video's bank example

ExampleAPI keyFrom the video, run on Groq

from judge import judge
from ragas.metrics.collections import Faithfulness

question = "What is the minimum balance for my savings account?"
chunks = [
    "Minimum balance is ₹10,000 for urban branches. Non-maintenance fee is ₹350 + GST if balance falls below.",
    "Semi-urban branch minimum balance is ₹5,000. Rural branch minimum balance is ₹2,500.",
]
response = ("The minimum balance for urban branches is ₹10,000. The non-maintenance fee charged is ₹350 + GST. "
            "You can also transfer funds online at no extra charge. For rural branches the minimum is ₹2,500.")

result = Faithfulness(llm=judge).score(user_input=question, response=response, retrieved_contexts=chunks)
print("faithfulness:", result.value)
print("PASS" if result.value >= 0.8 else "FAIL", "at a threshold of 0.8")

Output

faithfulness: 0.75
FAIL at a threshold of 0.8

The judge reaches the slide's number, 0.75: three of the four claims are in the chunks, so the answer fails the 0.8 mark. The next run applies the same check to the TechNest bot, with its real answer and a short answer that invents a promise.

Faithfulness of the TechNest bot

ExampleAPI key

from judge import judge
from ragas.metrics.collections import Faithfulness
from technest import answer

question = "What is TechNest's return policy?"
response, contexts = answer(question)
invented = "TechNest accepts returns within 30 days of purchase. Return shipping is free on every order."

faithfulness = Faithfulness(llm=judge)
for label, text in [("app answer", response), ("invented promise", invented)]:
    result = faithfulness.score(user_input=question, response=text, retrieved_contexts=contexts)
    print(f"{result.value:.2f}  {label}")
print()
print(response)

Output

1.00  app answer
0.50  invented promise

TechNest accepts returns within 30 days of the original purchase date, provided the items are in their original packaging with all accessories included. Customers are responsible for return shipping costs unless the item arrives defective or damaged, and refunds are processed within 5 to 7 business days of receiving the returned item.

Reading the two scores

1.00, the app's answer. Every claim in it, the 30 days, the packaging, who pays return shipping and the refund time, is in the return-policy chunk.
0.50, the short answer with a promise. Its first claim is in the chunk; "Return shipping is free on every order" contradicts it, since the customer pays unless the item is defective, the same mistake the video's notebook uses as its bad example. One of its two claims is supported, so it scores 0.50.
The pass mark decides what counts. At the video's 0.8 threshold the app's answer passes and the invented one does not; where you set the line is a choice you make per metric.

Faithfulness vs answer correctness

	Faithfulness	Answer correctness
Compares the answer with	The retrieved chunks	The golden's reference
Needs a reference?	No	Yes
Catches	Claims the chunks do not support	Facts that differ from the truth
Misses	A wrong chunk copied faithfully	Whether the chunks backed the answer

When to use faithfulness

Any bot whose answers carry promises, prices or policies, where an invented detail costs money or trust.
After a prompt change, to check the model still stays inside the chunks.
On live traffic: it needs no reference, only the chunks the app retrieved.

Watch out. Faithfulness cannot tell whether an answer is right, only whether it stayed inside the chunks. An answer that faithfully copies an out-of-date chunk scores 1. Pair it with context recall or answer correctness.

Try it yourself

Remove the online-transfer sentence from the bank answer and check that the score reaches 1.
Change "₹2,500" in the bank answer to "₹3,000" and see which claim the judge rejects.
Ask the TechNest bot about the SoundPods Pro battery and score its answer.

PreviousRate limits and cooldowns

Little by little, you're building something great.

Faithfulness

Checking each claim against the chunks

The Faithfulness API

The bank chunks

The answer with one hallucinated claim

Scoring against a threshold

The video's bank example

Faithfulness of the TechNest bot

Reading the two scores

Faithfulness vs answer correctness

When to use faithfulness

Related