High Answer Relevance but Low Faithfulness: Diagnose Wrong RAG Answers

A high relevance low faithfulness RAG result is not contradictory. It describes an answer that addresses the user's topic and intent but includes claims that the retrieved context does not support. Fluency, specificity, and directness can raise relevance while a remembered date, invented exception, merged policy, or unjustified inference lowers faithfulness. Diagnose the mismatch by splitting the answer into atomic claims and tracing every claim to the exact context the model received.

The metric relationships are mapped in the complete RAG evaluation metrics guide. Use the retrieval-versus-generation failure workflow, the context precision, recall, and relevancy comparison, and the synthetic RAG testset tutorial for adjacent work. The broader LLM application testing guide explains evaluator design, while the existing Ragas faithfulness test guide provides a direct implementation reference. QA teams can browse /skills and use the Playwright CLI skill when the answer and citations must be verified through a browser.

Relevance and Faithfulness Ask Different Questions

Answer relevance asks whether the response addresses the user input. Faithfulness asks whether the response's claims are supported by retrieved context. Neither property implies the other.

Answer pattern	Relevance	Faithfulness	Example
Direct and fully supported	High	High	Gives the requested effective date found in the supplied policy
Direct but unsupported	High	Low	Gives a plausible date absent from every supplied passage
Supported but off-target	Low	High	Accurately summarizes policy history instead of answering the date question
Off-target and unsupported	Low	Low	Discusses unrelated benefits and invents an approval rule

The dangerous quadrant is high relevance with low faithfulness because it feels useful. The answer repeats the user's vocabulary, adopts the expected format, and may include convincing citations. A user has fewer surface cues that the factual basis is missing.

Ragas' current AnswerRelevancy metric compares the user input with artificial questions generated from the response through embeddings. Its documentation says the metric focuses on alignment with intent and does not assess factual accuracy. The resulting score usually falls between zero and one, but the docs note that cosine similarity is mathematically capable of negative values. Do not describe it as a factuality probability.

Ragas Faithfulness extracts claims from the response, judges whether each can be inferred from retrieved context, and reports the supported proportion. It does not compare the context with external truth. A response can faithfully repeat a stale, malicious, or factually incorrect source. Faithfulness is evidence adherence, not source correctness.

Version Baseline and Required Inputs

This guide was verified on July 14, 2026 against Ragas 0.4.3. New code should use collections metrics from ragas.metrics.collections, direct keyword arguments, and the MetricResult returned by .score() or .ascore(). Read the numeric value from .value and retain .reason when available. The migration documentation discourages new dependencies on the legacy sample-based APIs.

The two metrics require different inputs:

Metric	Minimum conceptual inputs	What must be preserved	Major dependency
Answer relevancy	User input and response	Resolved query, exact answer, evaluator and embeddings	Question generation and embedding similarity
Faithfulness	Response and retrieved contexts	Exact packed context in order, exact answer, evaluator	Claim extraction and context attribution

Do not feed the full candidate retrieval list to faithfulness if the generator saw only the packed top-k context. That would credit evidence unavailable during generation. Conversely, do not evaluate a post-processed answer if the user saw another version. The test must use the actual boundary artifacts.

Reconstruct the User-Visible Failure

Start with a case that a domain reviewer has marked wrong or unsupported. Capture:

1. The user input after conversation resolution.
2. Query rewrites and filters.
3. The final context payload delivered to the model.
4. Source IDs, revisions, titles, and character or token spans.
5. System and answer prompts.
6. Model identifier and decoding settings.
7. Raw answer before and after output processing.
8. Citations and the mapping from citation labels to sources.
9. Relevance and faithfulness evaluator identities and repeated results.

Save the packed context, not just links. Documents can change after the run, and a URL does not prove which revision or excerpt the model received. Apply access controls and redaction appropriate to the source data; evaluation traces can contain confidential text and user queries.

Turn the Answer into an Atomic Claim Ledger

The most useful diagnostic artifact is a claim ledger. Split compound statements so each row can be supported or rejected independently. Preserve qualifiers such as dates, populations, regions, exceptions, quantities, and modal verbs. "Employees receive 16 weeks from September" contains at least a duration claim and an effective-date claim.

{
  "case_id": "leave-duration-031",
  "user_input": "How much parental leave applies from September?",
  "response": "All employees receive 16 paid weeks from 1 September 2026.",
  "claims": [
    {
      "id": "c1",
      "claim": "The policy grants 16 paid weeks.",
      "supporting_spans": ["policy-2026#p12:l4-l6"],
      "verdict": "supported"
    },
    {
      "id": "c2",
      "claim": "The policy applies to all employees.",
      "supporting_spans": [],
      "verdict": "unsupported"
    },
    {
      "id": "c3",
      "claim": "The effective date is 1 September 2026.",
      "supporting_spans": ["policy-2026#p2:l1-l2"],
      "verdict": "supported"
    }
  ]
}

The ledger explains a low faithfulness result better than a scalar. It shows that the answer is on topic and mostly plausible, but overgeneralizes eligibility. It also supports remediation: improve the prompt's qualification rule, retrieve the eligibility section, or require abstention on the population clause.

Use three support labels at minimum: supported, contradicted, and not found. A contradicted claim is stronger evidence than missing support. Add ambiguous when the passage requires domain interpretation, but require adjudication rather than letting ambiguous become an automatic pass.

Trace Claims to Context, Not to Model Memory

For each claim, ask whether a reasonable reviewer can infer it from the supplied context without outside knowledge. Exact wording is unnecessary; valid paraphrase and straightforward entailment count. Do not allow the grader to fill missing facts from common knowledge.

Claim condition	Faithfulness label	Additional check
Explicitly stated in current supplied source	Supported	Verify citation points to that source
Directly entailed by multiple supplied statements	Supported with rationale	Record all spans and inference step
Topically plausible but absent	Unsupported	Search packed context, then full corpus separately
Conflicts with supplied source	Contradicted	Mark severity and source authority
Present only in a source retrieved but not packed	Unsupported for generation	Investigate context packing
Present in stale or unauthorized source	May be faithful to supplied text	Fail freshness or authorization rule separately

This distinction prevents a common mistake: searching the entire corpus after the answer and declaring a claim grounded because some document contains it. Faithfulness concerns the context used for that response. Corpus truth, source authority, and retrieval coverage are separate tests.

Run Controlled Context Variants

After the ledger is reviewed, rerun the same query and generator with controlled context changes.

Gold-context run: Supply a minimal verified bundle containing every required fact. If unsupported claims persist, generation instructions or model behavior are implicated.

Evidence-removal run: Remove the span supporting one fact. The answer should omit that fact, qualify uncertainty, or abstain according to product policy. If the same claim remains, the model is likely relying on prior knowledge or pattern completion.

Contradiction run: Insert two clearly versioned passages with conflicting dates and metadata naming the current source. Expect conflict handling or preference for the authoritative revision, not silent selection.

Distractor run: Add topically similar but irrelevant material while preserving the valid evidence. Unsupported claims should not appear merely because nearby concepts are mentioned.

Citation-swap run: Keep answer text fixed in a test double and alter the cited source mapping. Citation alignment must fail even if answer relevance remains high.

Variant outcome	Evidence	Leading hypothesis	Next action
Claim disappears when evidence is removed	Generation responds to context	Original support mapping may be valid	Check source truth and citations
Claim remains without evidence	Prior knowledge or prompt-induced completion	Generation faithfulness defect	Tighten evidence rule and add abstention test
Gold context fixes claim	Missing or noisy production context	Retrieval or packing defect	Use component localization workflow
Gold context does not fix claim	Generator ignores or misreads evidence	Prompt/model/output defect	Inspect instructions and claim complexity
Contradictory source selected silently	Authority metadata unused	Prompt and source-policy defect	Make revision precedence explicit

Keep all other variables fixed. A new model plus new prompt plus gold context is a candidate solution test, not a causal diagnosis.

Score with Current Ragas Collections Metrics

The example below follows current Ragas 0.4.3 documentation. It uses the same answer for both metrics but supplies only the inputs each metric needs. The OpenAI model names mirror current documentation examples; production suites should pin approved evaluator and embedding versions and record cost and latency.

import asyncio
from openai import AsyncOpenAI
from ragas.embeddings.base import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerRelevancy, Faithfulness


async def evaluate_answer() -> None:
    client = AsyncOpenAI()
    llm = llm_factory("gpt-4o-mini", client=client)
    embeddings = embedding_factory(
        "openai",
        model="text-embedding-3-small",
        client=client,
    )

    user_input = "How much parental leave applies from September?"
    response = "All employees receive 16 paid weeks from 1 September 2026."
    retrieved_contexts = [
        "Eligible primary caregivers receive 16 paid weeks.",
        "The revised policy takes effect on 1 September 2026.",
    ]

    relevancy = await AnswerRelevancy(
        llm=llm,
        embeddings=embeddings,
    ).ascore(user_input=user_input, response=response)

    faithfulness = await Faithfulness(llm=llm).ascore(
        response=response,
        retrieved_contexts=retrieved_contexts,
    )

    print({
        "answer_relevancy": relevancy.value,
        "faithfulness": faithfulness.value,
        "faithfulness_reason": faithfulness.reason,
    })


asyncio.run(evaluate_answer())

The expected qualitative finding is that the response directly answers the question but overstates eligibility. Do not hard-code an expected numeric score from this article: evaluator models and prompts are probabilistic, and Ragas does not define one universal release threshold.

Run repeated evaluations for borderline or high-impact cases. If claim-level human labels remain stable while scores vary widely, treat the metric as an investigation aid rather than a blocking gate until the evaluator is improved.

Add Promptfoo Assertions at the System Boundary

Promptfoo's current RAG documentation distinguishes answer relevance from context faithfulness and allows contextTransform to extract evidence from a structured response. The following configuration assumes the provider returns answer and documents.

tests:
  - description: leave answer remains within retrieved evidence
    vars:
      query: How much parental leave applies from September?
    options:
      transform: output.answer
    assert:
      - type: answer-relevance
        threshold: 0.8
      - type: context-faithfulness
        contextTransform: output.documents.map((doc) => doc.text).join('\n\n')
        threshold: 0.9
      - type: llm-rubric
        value: >
          Every population qualifier in the answer must be explicitly supported
          by the supplied policy context. Do not infer that "eligible primary
          caregivers" means "all employees".
      - type: javascript
        metric: citations-resolve
        value: |
          return output.citations.every((citation) =>
            output.documents.some((doc) => doc.id === citation.source_id)
          );

The example values demonstrate syntax only. Calibrate answer-relevance and faithfulness thresholds with independently reviewed cases. The targeted rubric and deterministic citation check make the release rule more specific than one aggregate metric.

Find the Actual Source of Unsupported Claims

Unsupported claims can enter at several points:

Retrieval omission. The generator needs an eligibility qualifier, but the packed context contains only duration and date. This is a retrieval or packing coverage failure followed by unsafe generation behavior. Track both defects.

Noisy context fusion. One chunk says "all employees" about another benefit, and the answer incorrectly attaches it to parental leave. Improve chunk boundaries, metadata, ordering, and prompt source separation.

Stale source preference. An old policy supports the answer, while the current policy contradicts it. Faithfulness may look high if stale text was supplied. Add version authority and deletion tests.

Prompt pressure. Instructions demand a definitive answer even when evidence is incomplete. Replace unconditional helpfulness with explicit partial-answer, clarification, and abstention rules.

Model prior. The model completes a familiar pattern from training. Evidence removal tests expose this because the claim persists when its supporting text is absent.

Citation post-processing. The answer is generated without citation alignment, then a separate component attaches the nearest source. Evaluate claim support and citation alignment independently.

Judge error. The evaluator overlooks a qualifier or treats topical similarity as entailment. Keep claim ledgers and adjudicated anchor cases to audit the grader.

Design CI and Release Gates

Create separate gates for separate risks:

1. Required-claim coverage: the answer contains mandatory fields when evidence exists.
2. Unsupported-claim limit: zero unsupported high-risk claims; calibrated tolerance for low-risk explanatory text if the product allows it.
3. Contradiction block: no answer claim may contradict authoritative supplied context.
4. Citation alignment: each citation resolves to a delivered source and supports its associated claim.
5. Abstention behavior: missing evidence produces a bounded response rather than invention.
6. Answer relevance: the response still addresses the user after faithfulness controls are tightened.

Do not optimize faithfulness by returning empty or evasive answers. A response with no factual claims can avoid unsupported claims while failing the user. Pair faithfulness with answer completeness, relevance, and task-specific required fields.

Compare candidate and baseline by slice. Unsupported claims about greetings and unsupported claims about eligibility, dosage, payment, or legal obligations do not carry equal harm. High-risk slices need deterministic review and stricter blocking behavior.

Store claim-level diffs in CI artifacts. Reviewers should see which claim changed, what context was supplied, and why the grader labeled it unsupported. A dashboard average cannot support a safe release decision by itself.

Common Mistakes

Equating on-topic with true. Relevance rewards addressing the question. It does not validate facts, evidence, or source authority.

Evaluating against the whole corpus. Faithfulness must use the context delivered for that answer. Searching later changes the question from "was it grounded?" to "could it be supported somewhere?"

Treating citations as proof. A citation can exist, resolve, and still fail to support the adjacent claim. Verify semantic alignment at claim level.

Ignoring qualifiers. Dates, regions, eligibility groups, exceptions, and confidence language often contain the actual defect. Split them into separate claims.

Using one judge run as ground truth. Repeat model-assisted grading, retain reasons, and adjudicate important disagreements.

Raising faithfulness by deleting useful content. Pair faithfulness with relevance, completeness, and required-answer checks so empty answers do not win.

Letting stale context pass. Faithfulness is relative to supplied evidence. Add separate checks for source revision, authorization, and truth.

Limits

Claim extraction is not perfectly objective. Reviewers can disagree on whether a sentence contains one claim or several, and whether an inference is sufficiently direct. Define annotation rules and measure agreement on high-impact fixtures.

Ragas and Promptfoo metrics depend on evaluator models and prompts. Scores can drift when providers update models, even if the application is unchanged. Pin when possible, maintain anchor examples, and rerun calibration after upgrades.

Faithfulness cannot detect a source that is itself false. Answer relevancy cannot detect an unauthorized disclosure. Neither metric establishes completeness unless the fixture defines required content. Treat them as parts of a quality system, not a replacement for domain truth and security testing.

Investigation Checklist

• Capture the exact query, packed context, prompt, model, answer, and citations.
• Split every factual statement into atomic claims with qualifiers preserved.
• Map each claim to exact supplied spans as supported, contradicted, absent, or ambiguous.
• Verify source authority, revision, and access separately from faithfulness.
• Repeat generation with gold context, removed evidence, conflicts, and distractors.
• Run current collections-based answer relevancy and faithfulness metrics.
• Calibrate evaluator thresholds and retain reasons and repeated outcomes.
• Test citation resolution and claim-to-citation alignment deterministically.
• Gate unsupported high-risk claims independently of aggregate relevance.
• Add the reviewed failure and perturbations to the regression suite.

FAQ: Relevance and Faithfulness

How can an answer be highly relevant but unfaithful?

It can directly answer the user's question while adding a date, amount, exception, or population claim that no supplied passage supports. Relevance concerns intent alignment; faithfulness concerns evidence support.

Does low faithfulness prove the model hallucinated?

Not by itself. The context extraction may be incomplete, the claim may be ambiguous, or the evaluator may be wrong. Review atomic claims against the exact packed context before assigning a generation defect.

Can a faithful answer still be wrong?

Yes. If the supplied source is stale, incorrect, poisoned, or unauthorized, a response can accurately repeat it and remain faithful. Source truth and governance require separate checks.

Should citations automatically increase faithfulness?

No. Citation presence does not prove support. Resolve each citation, confirm it was supplied to the model, and verify that the cited span entails the associated claim.

What should happen when context lacks one required fact?

The product policy should define whether to give a partial answer, ask for clarification, or abstain. The model should not fill the gap from memory when the application promises evidence-grounded answers.

Is a perfect faithfulness score enough for release?

No. The answer can be faithful but irrelevant, incomplete, unsafe, or based on a stale source. Keep independent gates for relevance, required content, correctness, authorization, freshness, and citations.

Why keep a manual claim ledger if Ragas extracts claims?

The ledger is an auditable ground truth for important cases and helps evaluate the evaluator. Automated extraction scales review, but it should be compared with adjudicated examples before it blocks releases.

How should stochastic grading be handled in CI?

Pin evaluator configuration, repeat borderline cases, store every result and reason, and use deterministic invariants for high-risk claims. Do not retry silently until a desired score appears.

Conclusion

High relevance with low faithfulness is a warning that usefulness and evidence have diverged. Preserve the exact context, break the answer into atomic claims, trace support span by span, and rerun controlled context variants. Pair current Ragas or Promptfoo metrics with deterministic citation and source rules. The goal is not to maximize one score; it is to ensure every important claim is both responsive to the user and defensible from authorized, current evidence.