Contextual Precision vs Recall vs Relevancy for RAG Testing

RAG context precision recall relevancy testing needs three separate questions: are useful chunks ranked early, is the necessary evidence present, and is the delivered context focused on the user's request? Precision describes ordering quality, recall describes coverage against a reference, and relevancy describes pertinence or focus. Combining them into one vague "retrieval quality" score removes the information a QA team needs to diagnose a regression.

Use the complete RAG evaluation metrics guide as the cluster reference, then compare the retrieval-versus-generation diagnostic workflow, the high-relevance, low-faithfulness investigation, and the Ragas synthetic testset tutorial. The LLM application testing guide provides the broader quality model, while the existing top-k context precision test guide goes deeper on rank cutoffs. Reusable QA workflows live in /skills, including the Playwright CLI skill for browser-based evidence.

The Short Comparison

The terms overlap in ordinary language, but the metrics do not.

Dimension	Operational question	Data it needs	A low result suggests	It cannot prove
Context precision	Are relevant chunks placed ahead of irrelevant chunks?	Ordered retrieved contexts plus a relevance basis	Useful evidence is buried among noise	That all required evidence was retrieved
Context recall	Does retrieved context support the required reference claims?	Retrieved contexts plus a reference or reference contexts	Necessary evidence is missing	That the returned list is concise or well ordered
Context relevance	Is supplied context pertinent to the user input?	User input plus retrieved contexts	Context is off-topic, mixed, or weakly focused	That every reference fact is covered

A result can have high precision and low recall. Imagine two returned chunks, both highly relevant, but a third necessary policy exception is absent. The ranking is clean; coverage is incomplete. Another result can have high recall and low precision: all required facts are present, but dozens of irrelevant chunks precede or surround them. The evidence exists, yet noise may consume the prompt budget or distract generation.

High contextual relevance does not guarantee either metric. Every retrieved chunk may discuss parental leave, so the context is topically focused, while none contains the newly effective exception required by the reference. Conversely, a context can include the exact required paragraph and several unrelated passages, yielding strong coverage with poor focus.

Terminology and Current Ragas Baseline

Searchers often say "contextual precision," but current Ragas documentation names the metric Context Precision. In Ragas 0.4.3, the collections-based ContextPrecision evaluates whether contexts useful for answering the query are ranked higher. The documented reference-based scorer receives user_input, reference, and ordered retrieved_contexts.

Ragas ContextRecall asks whether claims in a reference can be attributed to retrieved context. A reference is required for that LLM-based calculation because recall needs a denominator: the information that should have been present. The documentation also describes alternatives based on reference contexts or document IDs. If a fixture has authoritative document IDs, deterministic ID recall is often easier to explain than an LLM judgment.

The current Ragas collections catalog also documents ContextRelevance in its NVIDIA metric group. It uses independent judge calls to assess how pertinent retrieved contexts are to user_input. That is not the same calculation as Context Precision and should not be renamed precision in reports.

This article was verified on July 14, 2026 against Ragas 0.4.3. Modern metrics are imported from ragas.metrics.collections; .score() and .ascore() return MetricResult, with the numeric value in .value and an optional explanation in .reason. The v0.4 migration documentation recommends these collections APIs rather than legacy SingleTurnSample examples. Pin Ragas and evaluator versions because imports, result structures, prompts, and model behavior can change.

Design a Fixture That Can Support All Three Questions

A query-and-answer pair is not enough. Build a retrieval fixture containing source-level truth and claim-level truth:

• user_input: the exact resolved query sent to retrieval.
• reference: a reviewed answer or set of required claims.
• reference_context_ids: documents or chunks that legitimately support those claims.
• required_claims: atomic facts that the returned evidence must cover.
• retrieved_contexts: ordered text exactly as the generator receives it.
• retrieved_context_ids: stable IDs in the same order.
• relevance_labels: independent labels for each result, ideally with reviewer rationale.
• slice_tags: language, source type, role, freshness, query class, and risk.

Do not create a reference by copying the current generated answer. That makes recall circular and can legitimize the same omission being tested. The reference should come from approved source material and domain review. When policy permits multiple correct answers, store required facts and acceptable alternatives instead of one fragile exact string.

The relevance label also needs a clear rule. A chunk may be topically related but not useful for answering the specific question. For "When does the revised policy begin?", a history of parental leave is related to the topic but irrelevant to the requested effective date. Define relevance at the task level.

{
  "case_id": "benefits-effective-date-022",
  "user_input": "When does the revised parental leave policy begin?",
  "reference": "The revised policy begins on 1 September 2026.",
  "required_claims": [
    {"id": "effective-date", "text": "Start date is 1 September 2026"}
  ],
  "reference_context_ids": ["leave-2026-section-2"],
  "retrieved_context_ids": [
    "leave-history",
    "leave-2026-section-2",
    "benefits-index"
  ],
  "relevance_labels": [0, 1, 0],
  "slice_tags": ["hr-policy", "date", "high-impact"]
}

This fixture gives precision an ordered relevance sequence, recall a required claim or source set, and relevance the query-context relationship. It also exposes why one score is insufficient: the required source is present, but it is not first and the list contains noise.

Measure Ranking with Context Precision

Context Precision is sensitive to order. Moving the same relevant item upward should improve or preserve ranking quality; adding irrelevant chunks ahead of it should not improve the result. This makes the metric useful for comparing retrievers, rerankers, fusion weights, and top-k policies.

Before using an LLM-based scorer, calculate transparent rank evidence:

1. Mark each retrieved context relevant or irrelevant according to the fixture rule.
2. Record the first relevant rank.
3. Record the ranks of every required source.
4. Calculate a deterministic ranking measure suitable for your product, such as precision at an operational cutoff or reciprocal rank for single-target lookups.
5. Inspect cases individually when an aggregate changes.

Ragas Context Precision can complement this record by judging usefulness against a reference. It cannot replace source labels where exact IDs matter, and it does not establish whether every required fact was retrieved. A perfectly ordered list containing only one of two necessary sources may still be incomplete.

Be explicit about the cutoff. Users and generators experience only the contexts actually packed into the prompt. A metric over top 50 candidates may look healthy while the context packer keeps only top 4. Evaluate the candidate ranking for retriever engineering and the packed ranking for end-to-end risk.

Measure Coverage with Context Recall

Recall asks what is missing. Ragas' LLM-based Context Recall breaks the reference into claims and checks whether each can be attributed to retrieved context. That makes reference quality decisive. If the reference omits an exception, the metric has no basis for requiring that exception. If it includes a disputed claim, a good retriever may be penalized.

Use source-ID recall when the authoritative support set is known. Use claim attribution when multiple passages can support the same fact or when stable chunk IDs change after reindexing. Keep both for high-impact fixtures: IDs reveal source-level regressions, while claims tolerate legitimate rechunking.

Recall approach	Best fit	Strength	Main risk
Required document or chunk IDs	Stable curated corpus	Deterministic and explainable	Rechunking or equivalent sources can create false failures
Reference-context comparison	Reviewed evidence passages	Tests evidence coverage directly	Requires costly reference curation
Reference-answer claim attribution	Reviewed answers with atomic facts	Works when exact source spans vary	Judge and reference errors affect the result
Required-field assertions	Structured policy or product facts	Clear release invariants	Covers only facts explicitly modeled

Do not interpret low recall as poor ranking. A missing source might have been excluded by permissions, absent from the index, filtered by date, or not requested by query decomposition. Ranking cannot promote a candidate that never entered the set. Use traces to locate the stage.

Measure Focus with Context Relevance

Context relevance asks whether the context is pertinent to the user input. It is useful when retrieval returns semantically related but unhelpful material, when query rewriting drifts, or when broad parent documents flood the prompt with tangential content.

Ragas' current Context Relevance implementation uses discrete LLM judgments normalized into a score. Promptfoo's current RAG guide similarly describes context relevance as how much retrieved context is necessary for the query. These are model-assisted judgments, so record the evaluator, run repeated grading on borderline cases, and retain explanations.

Relevance is not coverage. A short passage can be perfectly focused yet omit the requested exception. Relevance is not faithfulness either: it evaluates context against the query, not answer claims against context. A generator can invent a date even when every supplied paragraph is on topic.

Use direct labels for obvious distractors and LLM judgments for ambiguous semantic cases. A navigation footer, duplicate header, or unrelated product page does not need an expensive judge. A technical passage that supports a multi-hop inference may need domain review.

Run Current Ragas Metrics without Mixing Inputs

The following example follows the documented Ragas 0.4 collections pattern. It assumes OPENAI_API_KEY is configured for the AsyncOpenAI client and uses an evaluator model shown in current Ragas examples. Pin an approved model in CI rather than relying on an alias indefinitely.

import asyncio
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision, ContextRecall, ContextRelevance


async def score_retrieval() -> None:
    client = AsyncOpenAI()
    llm = llm_factory("gpt-4o-mini", client=client)

    user_input = "When does the revised parental leave policy begin?"
    reference = "The revised policy begins on 1 September 2026."
    retrieved_contexts = [
        "The organization introduced parental leave in 2018.",
        "The revised parental leave policy applies from 1 September 2026.",
        "Benefits documents are listed on the employee portal.",
    ]

    precision = await ContextPrecision(llm=llm).ascore(
        user_input=user_input,
        reference=reference,
        retrieved_contexts=retrieved_contexts,
    )
    recall = await ContextRecall(llm=llm).ascore(
        user_input=user_input,
        reference=reference,
        retrieved_contexts=retrieved_contexts,
    )
    relevance = await ContextRelevance(llm=llm).ascore(
        user_input=user_input,
        retrieved_contexts=retrieved_contexts,
    )

    print({
        "context_precision": precision.value,
        "context_recall": recall.value,
        "context_relevance": relevance.value,
    })


asyncio.run(score_retrieval())

Do not copy the resulting values into a universal pass rule. First run reviewed examples, compare scores with labels, repeat grading to estimate variance, and choose release behavior by risk slice. Preserve .reason when the metric supplies it; disagreement is easier to investigate with rationale than with a number alone.

Cross-Check with Promptfoo

Promptfoo's current context assertions can extract documents from a structured provider response. This is useful when the system under test returns answer plus retrieved documents. The contextTransform expression receives the provider output directly, even if another transform extracts only the answer for output assertions.

tests:
  - description: leave effective date retrieval coverage and focus
    vars:
      query: When does the revised parental leave policy begin?
    options:
      transform: output.answer
    assert:
      - type: javascript
        metric: required-source-present
        value: |
          return output.documents.some(
            (doc) => doc.id === 'leave-2026-section-2'
          );
      - type: context-recall
        value: The revised policy begins on 1 September 2026.
        contextTransform: output.documents.map((doc) => doc.text).join('\n\n')
        threshold: 0.9
      - type: context-relevance
        contextTransform: output.documents.map((doc) => doc.text).join('\n\n')
        threshold: 0.8

These thresholds are syntax examples. Promptfoo requires threshold values for context-based assertions, but the correct values are empirical. Also retain the deterministic source assertion; an LLM judge should not be the only evidence that a required regulated source was returned.

Interpret Metric Combinations as Hypotheses

Precision	Recall	Relevance	Likely pattern	Next inspection
High	High	High	Clean, sufficient, focused context	Check generation faithfulness and correctness
High	Low	High	Few focused results but missing required evidence	Candidate generation, filters, corpus coverage
Low	High	Low	Evidence present amid noise	Ranking, reranking, deduplication, top-k
Low	Low	High	Topically focused but insufficient and poorly ordered	Query decomposition and source coverage
High	High	Low	Required evidence present, but context contains broad tangents	Packing, parent expansion, duplicate removal
Low	High	High	Relevant set covers facts but the best items are buried	Ordering and operational cutoff

These patterns narrow investigation; they do not assign root cause automatically. Inspect actual examples. A high recall value can be produced by one dense chunk containing all reference claims, even if the source is unauthorized or stale. A high relevance value can come from multiple passages repeating the same fact while another required fact is absent.

Build Metamorphic Tests Around Expected Direction

Metamorphic tests alter one retrieval property and assert the expected direction rather than one exact model-assisted score.

• Move a labeled relevant chunk above an irrelevant chunk; precision should not worsen.
• Remove the only passage supporting a required reference claim; recall should not improve.
• Add clearly unrelated text ahead of the same evidence; relevance should not improve.
• Replace a current source with a superseded source; freshness and source-policy assertions should fail even if topic relevance remains high.
• Increase top-k while preserving required evidence; recall may improve, while precision or relevance may fall.
• Apply the correct tenant filter; unauthorized-source count must fall to zero without losing authorized required evidence.

Directional checks are still subject to evaluator noise. For deterministic labels, make them hard assertions. For LLM metrics, repeat the comparison and investigate inconsistent direction rather than automatically retrying until it passes.

Use Separate Release Gates

Do not average precision, recall, and relevance into a single score unless the product has explicitly accepted compensation among them. A high ranking score should not cancel missing evidence for a critical answer. Strong recall should not excuse unauthorized or massively noisy context.

A safer release policy has independent gates:

1. Hard retrieval invariants: required source IDs for critical cases, zero unauthorized sources, no deleted documents, valid revisions.
2. Coverage gate: required claims or reference evidence represented by slice.
3. Ranking gate: no material regression at the operational context cutoff.
4. Focus gate: bounded irrelevant-context rate or calibrated relevance result.
5. End-to-end gate: answer faithfulness, required fields, citations, abstention, and safety.

Compare candidates with a fixed baseline corpus snapshot before comparing production traffic. If both corpus and retriever change, keep a factorial test that runs old and new retrieval configurations against old and new snapshots. That reveals whether a score moved because the system improved or because the evidence set changed.

Common Mistakes

Calling precision "percentage of relevant text." Ragas Context Precision is rank-aware. A separate context relevance or utilization measure may assess focus, but the names are not interchangeable.

Calculating recall without a reference. Recall requires a target set or target information. Without one, you can measure similarity or relevance, not what was missed.

Using answer relevance as context relevance. Answer relevance compares response to user intent. Context relevance compares retrieved context to user input. A fluent answer can be relevant despite poor evidence.

Ignoring order. Passing an unordered set to a rank-aware metric discards the behavior being tested. Preserve retriever and packed-context order.

Treating chunks as independent. Parent-child retrieval, overlapping chunks, and duplicates can inflate apparent evidence. Track source lineage and deduplicate for analysis without changing the actual prompt trace.

Publishing one threshold for every slice. Short factual queries, multi-hop questions, multilingual corpora, and scanned documents have different variance and risk. Calibrate and report by slice.

Letting an LLM judge define authority. A judge can decide that a passage sounds relevant; it cannot determine which internal policy revision is legally authoritative without reliable metadata and fixture rules.

Limits

Metric definitions are implementation-specific. A paper, Ragas, Promptfoo, and an internal dashboard may use similar names for different calculations. Document the tool, class or assertion type, inputs, evaluator, version, and cutoff beside every result.

Model-assisted metrics are probabilistic and can be sensitive to language, prompt, answer length, and evaluator changes. Deterministic source labels are expensive to maintain and can become stale after rechunking. Use both, review disagreements, and keep a small adjudicated anchor set for detecting evaluator drift.

No retrieval metric proves that the underlying source is true, current, authorized, or safe to reveal. Add source governance and access tests. No retrieval metric proves the final answer uses context faithfully. Add claim-level generation checks.

Precision, Recall, and Relevancy Checklist

• Define each metric in the repository using its exact tool and version.
• Preserve ordered candidates and ordered final prompt contexts separately.
• Curate references, required claims, source IDs, distractors, and access labels.
• Use deterministic rank and ID checks where source truth is known.
• Use current Ragas collections APIs and retain MetricResult values and reasons.
• Calibrate Promptfoo or Ragas thresholds against independent human labels.
• Report precision, recall, and relevance independently by risk slice.
• Run directional perturbations for ordering, omission, noise, freshness, and filters.
• Keep retrieval gates separate from faithfulness and answer-correctness gates.
• Revalidate fixtures and evaluators after corpus, chunking, model, or package changes.

FAQ: Context Metrics

Can context precision be high while context recall is low?

Yes. The retriever can return a short, perfectly ordered list of relevant chunks while omitting another source required to answer fully. Precision rewards the clean ranking; recall exposes missing evidence.

Does high context recall mean retrieval is good?

It means the tested context covers the chosen reference or target set. The list may still be noisy, poorly ordered, unauthorized, stale, or too large for the generator. Inspect the other dimensions and governance rules.

Is context relevancy the same as context precision?

No. Relevancy asks whether context is pertinent to the query. Precision is rank-aware and asks whether relevant items are placed ahead of irrelevant ones. Tool vendors may use terms differently, so cite the exact implementation.

Why does Context Recall require a reference?

Recall measures how much of a target set was recovered. A reference answer, reference contexts, required claims, or source IDs provide that target. Without a denominator, the system cannot know what it missed.

Should I use reference text or source IDs?

Use IDs for stable, curated retrieval expectations and reference claims when equivalent sources or rechunking make exact IDs brittle. High-impact suites often retain both because they reveal different regressions.

Is 0.8 a safe universal threshold?

No. A score depends on the metric implementation, evaluator, fixture distribution, language, and risk. Calibrate thresholds against reviewed outcomes and recheck them when any evaluator component changes.

Can I average all three metrics into one KPI?

You can calculate an internal summary for observation, but do not let it hide blocking failures. Missing critical evidence, unauthorized retrieval, or severe ranking regressions should remain independent gates rather than being offset by another high score.

Which metric should run first during triage?

Start with deterministic source inclusion and the raw ranked list. Then inspect recall for missing evidence, precision for ordering, and relevance for focus. Finally evaluate whether generation used the delivered context faithfully.

Conclusion

Context precision, recall, and relevancy form three diagnostic axes. Precision explains ranking, recall explains evidence coverage, and relevance explains focus. Preserve their distinct inputs and limitations, combine model-assisted results with source labels, and gate releases on independent risks. The result is not merely a better dashboard; it is a retrieval test suite that tells engineers where to look next.