How to Tell Whether a RAG Failure Comes from Retrieval or Generation

A RAG retrieval vs generation failure cannot be localized from a bad final answer alone. The answer is downstream evidence: it may be wrong because the retriever omitted the governing passage, ranked noise above it, applied the wrong metadata filter, or supplied correct evidence that the generator ignored, distorted, or supplemented from unsupported memory. A reliable diagnosis changes one stage at a time, preserves every input and version, and compares the resulting claims against the same evidence contract.

Start with the complete RAG evaluation metrics guide, then use the companion guides on context precision, recall, and relevancy, high relevance with low faithfulness, and synthetic RAG testset generation. For the broader LLM quality model, read testing LLM applications; for an existing production pattern, use the RAG regression testing guide. Teams can also browse reusable QA instructions in /skills and automate browser evidence collection with the Playwright CLI skill.

The Diagnostic Question Is Causal, Not Merely Statistical

A low aggregate metric tells you that a system missed an expectation. It does not prove which component caused the miss. Retrieval and generation are coupled: generation quality depends on what retrieval supplied, while retrieval may be judged through a generated reference or an LLM evaluator. If the observation crosses both stages, it is not yet a component diagnosis.

The practical question is counterfactual: would the same generator answer correctly if it received a verified evidence bundle, and would the production retriever return that evidence without help? Those are separate experiments. The first holds generation configuration constant and substitutes context. The second bypasses generation and evaluates ranked retrieval output against labeled evidence.

Observation	Plausible cause	Evidence still needed	Do not conclude yet
Correct answer with gold context, wrong answer end to end	Retrieval path is implicated	Production trace, relevant document IDs, filter and ranking output	That the vector database itself is broken
Wrong answer with gold context	Generation path or fixture is implicated	Claim trace, prompt, model settings, gold-context review	That retrieval is healthy in production
Required document retrieved below the context cutoff	Ranking or packing can cause omission	Full ranked list and final prompt context	That embedding quality alone caused it
Correct document and passage reach the prompt, unsupported claim appears	Generation is strongly implicated	Exact prompt, response, claim support labels	That every answer defect is hallucination
Both isolated stages pass but end-to-end fails	Integration or transformation defect	Serialized prompt, truncation, deduplication, ordering	That metrics are inconsistent

This table is a triage map, not a theorem. A mislabeled reference can make a healthy retriever look wrong. A stale source can make a faithful generator return an answer that is no longer operationally correct. A grader can disagree with human reviewers. Diagnosis requires raw artifacts, not only scores.

Version and Reproducibility Baseline

This guide was checked on July 14, 2026. The current PyPI release is Ragas 0.4.3. Its modern metrics live under ragas.metrics.collections, accept direct keyword arguments, and return a MetricResult whose numeric result is available through .value. The Ragas migration guide recommends experiment-oriented workflows instead of new uses of the legacy evaluate() and sample-based metric methods.

Promptfoo's current RAG guide also recommends testing document retrieval separately from generated output. Its context assertions can read static test context or extract context from a provider response with contextTransform. These tools support the workflow below, but neither library knows your authoritative document set, access policy, freshness rule, harm model, or release tolerance. Pin tool, model, prompt, corpus, index, and fixture versions in every result.

Record at least this run identity:

{
  "case_id": "leave-policy-effective-date-014",
  "query": "When does the revised parental leave policy apply?",
  "corpus_revision": "hr-handbook-2026-07-01",
  "index_revision": "hr-prod-shadow-2026-07-13.2",
  "retriever_config": "hybrid-v7-top20-rerank5-pack3",
  "prompt_revision": "answer-with-citations-v12",
  "generator": "provider/model-version",
  "grader": "provider/model-version",
  "expected_source_ids": ["hr-leave-2026-section-2"],
  "run_id": "ci-8421-case-014"
}

Do not replace an immutable model identifier with a marketing alias if the provider exposes a pinned version. If only an alias is available, save the date, provider response metadata, and repeated-run distribution. Reproducibility means another reviewer can reconstruct the conditions and evidence, not that a probabilistic output must be byte-identical.

Define an Evidence Contract Before Running Metrics

Each diagnostic fixture needs more than a question and an expected sentence. Define the facts that must be supported, the sources that are allowed to support them, the sources that are relevant but insufficient, and the response behavior when evidence is absent or contradictory.

For a policy question, a useful fixture includes:

A stable case ID and user-visible query.
The authoritative source document IDs, revisions, and relevant spans.
Required atomic facts, such as effective date, eligible population, and exception.
Acceptable alternate wording, without requiring one exact generated sentence.
Distractor documents that are topically close but superseded, regional, or incomplete.
Expected filters, permissions, and tenant boundaries.
The desired abstention or clarification behavior when required evidence is unavailable.
A risk label that determines whether a miss blocks a release or starts an investigation.

Fixture field	Retrieval use	Generation use	Failure it exposes
Relevant document IDs	Calculate deterministic top-k inclusion and rank	Confirm citations point to allowed evidence	Missing, misranked, or fabricated sources
Required claim list	Identify passages needed for coverage	Check every answer claim and required answer element	Partial context or incomplete answer
Superseded distractors	Test ranking and freshness filters	Test resistance to conflicting old text	Stale-source preference
Access labels	Test metadata and authorization filters	Prevent use of unauthorized context	Cross-tenant or role leakage
No-answer expectation	Verify empty or insufficient retrieval	Verify abstention instead of invention	Confident answer without evidence

Human reviewers should validate the evidence contract independently of the system under test. If the reference answer was generated from the same RAG pipeline, it can preserve the same omission or false claim and create circular approval.

Capture the End-to-End Trace First

Run the failing case once without modifying the system. Capture every boundary so later substitutions are comparable:

Original user input after conversation resolution.
Query rewrites, decomposed subqueries, and filters.
Candidate IDs with raw retrieval scores.
Reranked IDs and scores, if a reranker is used.
Chunks after deduplication, packing, and token-budget truncation.
Exact context delivered to the generator, in order.
System instruction, answer prompt, tool output, and generation settings.
Final answer, citations, latency, token usage, and evaluator results.

Do not log restricted text indiscriminately. Store access-controlled references or approved redacted snippets when the corpus contains personal, regulated, or confidential data. The requirement is traceability, not uncontrolled duplication.

Separate the candidate list from the final prompt context. A relevant chunk can appear at rank three and still disappear because a parent-document expansion, deduplicator, tokenizer estimate, or context packer removed it. Calling that a retrieval success hides the integration defect that the generator actually experienced.

Run the Controlled Substitution Matrix

Use four runs with the same fixture. Keep temperature, seed when supported, prompt revision, model identifier, output parser, and evaluator configuration fixed.

Run	Context source	Generator	What it isolates
A: production	Production retrieval and packing	Production generator	Reproduces user-visible behavior
B: gold-context substitution	Human-verified minimal evidence	Same generator and prompt	Whether generation can use sufficient evidence
C: retrieval-only	Production ranked output	No answer generation	Rank, coverage, filters, freshness, authorization
D: context perturbation	Gold context plus controlled distractor or missing span	Same generator and prompt	Sensitivity to noise, conflicts, and absent evidence

Run B must replace the exact final context payload, not merely insert a desired document earlier in the pipeline. Otherwise a later transform may still remove it. Keep the gold bundle minimal but sufficient: include every required supporting span and the metadata the prompt normally receives, without leaking the expected final wording into an artificial instruction.

Run C should use deterministic labels where possible. Check whether every required source ID appears within the operational cutoff, its rank, whether an explicitly irrelevant source precedes it, and whether filters removed an authorized source. An LLM relevance grader may complement those labels for semantic edge cases, but a known document ID does not need a probabilistic judge.

Run D tests the response contract. Remove one required span and expect abstention or a bounded partial answer. Add a superseded but topically similar passage and expect the system to prefer the current source or expose the conflict. Add irrelevant text without changing relevant evidence and observe whether the answer acquires unsupported claims.

Implement a Framework-Neutral Diagnostic Harness

The component boundary should return structured data. The following Python uses dependency injection rather than a library-specific RAG API, so adapters can wrap any retriever and generator. It records both the production run and a gold-context substitution under the same configuration.

from dataclasses import dataclass
from typing import Callable, Sequence


@dataclass(frozen=True)
class Chunk:
    source_id: str
    text: str
    rank: int


@dataclass(frozen=True)
class Case:
    case_id: str
    query: str
    required_source_ids: frozenset[str]
    gold_context: tuple[Chunk, ...]


def diagnose(
    case: Case,
    retrieve: Callable[[str], Sequence[Chunk]],
    generate: Callable[[str, Sequence[Chunk]], str],
) -> dict:
    retrieved = tuple(retrieve(case.query))
    production_answer = generate(case.query, retrieved)
    gold_answer = generate(case.query, case.gold_context)
    retrieved_ids = {chunk.source_id for chunk in retrieved}

    return {
        "case_id": case.case_id,
        "required_sources_retrieved": sorted(
            case.required_source_ids.intersection(retrieved_ids)
        ),
        "missing_required_sources": sorted(
            case.required_source_ids.difference(retrieved_ids)
        ),
        "retrieved": [chunk.__dict__ for chunk in retrieved],
        "production_answer": production_answer,
        "gold_context_answer": gold_answer,
    }

This harness deliberately does not assign a root cause. It creates evidence for a reviewer or a separate assertion layer. Add claim-support labels, required-answer checks, and run identity outside the adapter. Do not infer that gold_context_answer is correct merely because it differs from production; evaluate it against the same claim contract.

For stochastic generators, run each condition several times and retain every output. Compare failure rates or score distributions rather than selecting the best answer. The number of repeats should follow observed variance and release risk, not a universal constant.

Add Metric Evidence without Letting It Replace the Experiment

Ragas context precision assesses whether relevant chunks are ranked ahead of irrelevant chunks. Context recall evaluates whether retrieved context supports the claims in a reference. Faithfulness evaluates whether claims in the response are supported by retrieved context. These metrics answer different questions, and their model-assisted variants inherit evaluator variance.

A useful evidence bundle includes both deterministic and model-assisted results:

Evidence	Preferred method	Why it matters	Limitation
Required source in top k	ID comparison	Directly tests known retrieval expectation	Requires curated source labels
Required claim represented in context	Human label or context recall	Detects evidence coverage gaps	Depends on reference completeness
Relevant chunks ranked early	Context precision plus rank inspection	Detects noisy ordering	Relevance labels or judge can be wrong
Answer claims supported	Faithfulness plus claim ledger	Detects unsupported generation	Does not prove source truth or freshness
Answer addresses user intent	Answer relevancy or rubric	Detects evasive or incomplete answers	Does not prove factual correctness

Promptfoo can keep retrieval and answer checks in one reviewable configuration while still treating them separately. This example assumes the provider returns an object with answer and documents; adapt paths to the real response shape and calibrate thresholds on labeled local data.

tests:
  - description: revised leave policy uses current evidence
    vars:
      query: When does the revised parental leave policy apply?
    assert:
      - type: javascript
        value: |
          const ids = output.documents.map((doc) => doc.id);
          return ids.includes('hr-leave-2026-section-2');
      - type: context-recall
        value: The revised policy applies from 1 September 2026.
        contextTransform: output.documents.map((doc) => doc.text).join('\n\n')
        threshold: 0.9
      - type: context-faithfulness
        contextTransform: output.documents.map((doc) => doc.text).join('\n\n')
        threshold: 0.9
      - type: answer-relevance
        threshold: 0.8

The numbers above illustrate configuration syntax, not recommended defaults. Establish release values by comparing metric outcomes with independent human labels, measuring evaluator repeatability, and assigning different tolerances to different risk slices. A single global threshold can conceal a serious failure in a small but regulated slice.

Interpret Controlled Outcomes

If production retrieval misses required evidence and the same generator succeeds with gold context, the retrieval path is the leading cause. Inspect query rewriting, embedding input, lexical and vector fusion, metadata filters, tenant scope, freshness, reranking, deduplication, and context packing in that order. Preserve the failing trace before rebuilding an index, because rebuilding can destroy evidence.

If gold context still produces an unsupported or incomplete answer, inspect generation. Verify that the context is actually serialized into the prompt, the instruction defines how to handle conflicts and absent facts, the output parser retains citations, and the model is not asked to combine incompatible sources. Create an atomic claim ledger and identify exactly which required claim was omitted or which answer claim lacked support.

If retrieval-only labels pass and gold-context generation passes but end-to-end fails, focus on the seam. Common defects include escaping that empties a context field, token truncation, wrong chunk ordering, a stale cache, mismatch between retrieved IDs and fetched bodies, a prompt variable bound to another case, or an authorization layer replacing content after evaluation.

If both isolated stages fail, log two defects unless evidence shows a shared cause. A stale corpus can create both missing retrieval labels and an incorrect reference. A malformed query can impair retrieval and misstate the user's intent to generation. Fixing only the visible answer may leave the retrieval regression intact.

Use the Diagnosis in CI and Release Decisions

CI should run a small deterministic localization set on every relevant change and a larger distributional suite on scheduled or release jobs. Tag fixtures by component, source, language, role, query complexity, freshness, and harm. Run impacted slices when a chunker, embedding model, reranker, filter, prompt, generator, or corpus revision changes.

Block immediately on hard invariants such as unauthorized source retrieval, missing required source IDs for critical policy cases, fabricated citations, or unsupported high-risk claims. Treat model-assisted score movement as an investigation signal until calibrated. Require the candidate to beat or match the baseline by slice, not only in a global average.

Store the substitution result with the failure. A CI message that says "faithfulness 0.62" forces the next engineer to reconstruct the experiment. A useful message says that the required source disappeared after reranking, gold context restored all required claims in four of five repeated generations, and the index revision changed in the candidate build.

Common Diagnostic Mistakes

Judging retrieval from the final answer. A generator may answer from prior knowledge even when retrieval failed, or ignore good context. Inspect ranked sources directly.

Calling any relevant chunk a retrieval pass. Retrieval must provide enough evidence inside the final context budget, not merely one topically related paragraph somewhere in the candidate list.

Using the expected answer as hidden context. Gold context should contain source evidence, not a sentence crafted to force the expected wording. Otherwise the substitution tests prompt leakage rather than generation ability.

Changing multiple variables during substitution. Switching the model, prompt, temperature, and context together destroys causal value. Change one boundary at a time.

Treating faithfulness as correctness. A response can faithfully repeat an outdated or incorrect source. Validate source authority, revision, and business truth separately.

Assuming a score names the defect. Context recall can fall because a reference has unsupported claims. Context precision can vary with relevance judgments. Read the reasons and raw examples.

Ignoring permissions. Gold evidence that the production identity is not allowed to retrieve is not a valid expected source for that user. Authorization belongs in fixture design.

Limits of This Method

Controlled substitution localizes a failure to an operational boundary; it does not automatically find the underlying line of code. It is weaker when retrieval and generation are jointly trained, when generation invokes tools or additional retrieval, or when conversation memory silently changes the query. Expand the trace to every evidence-producing step in those systems.

Gold context also has limits. Human reviewers may omit a necessary passage, disagree about source authority, or encode hindsight into the fixture. Review high-impact fixtures with domain owners and version them like code. Model-based metrics can reduce review effort but cannot establish the ground truth they are asked to judge.

Diagnostic Checklist

FAQ: RAG Failure Diagnosis

Can answer correctness alone distinguish retrieval from generation failure?

No. A correct answer can come from model memory despite failed retrieval, and an incorrect answer can appear despite perfect evidence. Inspect retrieval output directly and rerun the same generator with verified context.

What is the strongest evidence that retrieval caused the failure?

The production trace lacks required authorized evidence, retrieval-only checks confirm the omission or misranking, and the unchanged generator reliably succeeds when the final context is replaced with validated evidence. This implicates the retrieval path but still requires inspection of rewriting, filtering, reranking, and packing to find the implementation cause.

What if the required document was retrieved but the answer is wrong?

Confirm that the relevant span survived fetching, deduplication, ordering, and token packing into the exact prompt. If it did, trace answer claims against that span and inspect generation. A document ID in an early candidate list is not proof that the model received usable evidence.

Does low faithfulness always mean a generation defect?

It means the evaluator found answer claims unsupported by the supplied context. The cause may be generation, a malformed context extraction, an incomplete trace, ambiguous claims, or grader error. Review the claim-level evidence before assigning ownership.

Should gold context contain only one chunk?

Only if one chunk supports every required fact and preserves necessary qualifiers. Prefer the smallest sufficient bundle, but do not remove definitions, exceptions, dates, or provenance that a reasonable answer needs.

How many repeated generations are enough?

There is no universal count. Measure variance for the model and case type, then choose repetitions that can detect a release-relevant change within budget. Retain all outputs and avoid selecting only the best run.

Can Promptfoo or Ragas identify the root cause automatically?

They can produce retrieval, context, response, and faithfulness evidence. Root-cause localization still depends on controlled substitutions, trustworthy fixtures, raw traces, and knowledge of the system boundary. A metric name is not a causal diagnosis.

When should this become a blocking CI test?

Make deterministic safety, authorization, required-source, and citation invariants blocking as soon as they are stable. Promote model-assisted thresholds only after calibration shows acceptable agreement and variance for the relevant slice.

Conclusion

Diagnosing a RAG failure is an experiment-design problem. Capture the full trace, evaluate ranked retrieval against labeled evidence, replace the final context with a verified bundle, and hold generation constant. Then trace each answer claim back to supplied text. That sequence separates missing or mispacked evidence from a generator that fails to use evidence, while preserving enough detail to find the real defect and prevent its return.