Vector Database Recall Testing Guide

The chatbot did not hallucinate because the model became worse. It answered from the wrong document because retrieval failed quietly. A vector database can accept writes, return results quickly, and still lose the passage that should have been ranked first. Recall testing is the guardrail that tells you whether the right items are still findable after index tuning, embedding model changes, metadata filter changes, chunking updates, or a database upgrade.

Vector database QA is different from ordinary API testing. You are not only asking whether query returns 200. You are asking whether the known relevant neighbors appear in the top k results, whether filters preserve those neighbors, whether ranking remains stable enough for the product, and whether latency optimizations have traded away too much retrieval quality.

This guide builds a practical recall test strategy for QA engineers working on search, recommendations, and RAG systems. It connects naturally to broader vector search testing and the retrieval-specific checks in RAG retrieval testing best practices.

Define relevance before measuring recall

Recall needs a ground truth set. For a query, which documents should be considered relevant? Without that, teams end up eyeballing search results and arguing after every release. Ground truth can come from human labeling, production click logs with careful cleaning, synthetic benchmark queries written by domain experts, or deterministic fixtures for narrow product rules.

Do not start with thousands of queries. Start with a small, reviewable set that covers the retrieval risks your product actually has: exact policy lookup, synonym matching, tenant isolation, freshness, language variants, SKU codes, support ticket similarity, or legal clause retrieval. Each query should have expected document IDs and a reason.

Retrieval scenario	Ground truth source	Risk being tested
Policy Q&A	Compliance-reviewed query to passage pairs	Wrong clause retrieved for regulated answer
Support deflection	Historical ticket and accepted help article pairs	Similar but irrelevant article outranks fix
Product search	Merchandising labels and click-cleaned examples	Synonyms, SKU tokens, filter interaction
Code search	Maintainer-labeled symbol and file matches	Chunking drops local context
Multi-tenant RAG	Fixture documents partitioned by tenant	Filter leakage or over-filtering

The labels do not have to be perfect, but they must be owned. A stale benchmark can punish good changes and bless bad ones. Review the set whenever the corpus, product taxonomy, or embedding model changes significantly.

Metrics that matter for vector database tests

Recall@k is the usual starting point: of the expected relevant documents, how many appear in the first k results? If a query has one known answer and it appears in the top five, recall@5 is 1.0. If it does not, recall@5 is 0.0. For multiple expected documents, recall is the fraction found.

Recall alone does not tell you whether the best answer is ranked first. Add mean reciprocal rank or nDCG when ordering matters. For RAG, top-k recall is often more important than exact position because a downstream reranker or model may see several passages. For direct search UI, top positions matter more.

Metric	What it answers	When to use
Recall@k	Did the relevant item appear in the candidate set?	RAG retrieval, recommendations, safety-critical lookup
Precision@k	How much of the candidate set is relevant?	Search result quality, limited context windows
MRR	How early did the first relevant result appear?	Known-answer lookup and support article retrieval
nDCG	Did graded relevance rank well?	Search with excellent, acceptable, and weak results
Filter pass rate	Did metadata constraints preserve expected hits?	Tenant, region, language, permission filters

Set thresholds by query group, not only globally. A 0.92 overall recall can hide a 0.55 recall for legal questions if product search dominates the benchmark. Break results down by tenant, language, content type, and filter shape.

A small runnable recall test with Chroma

The example below uses Chroma with deterministic hand-written embeddings. In a production suite, you would load embeddings generated by the same model and chunking pipeline used by the application. For a unit-level recall test, fixed vectors make the expected ranking easy to understand and remove network dependence.

import chromadb


def recall_at_k(actual_ids: list[str], expected_ids: set[str], k: int) -> float:
    top_k = set(actual_ids[:k])
    return len(top_k & expected_ids) / len(expected_ids)


def test_policy_recall_at_three():
    client = chromadb.Client()
    collection = client.create_collection("policy_recall_smoke")

    collection.add(
        ids=["refund-policy", "password-reset", "invoice-export", "shipping-delay"],
        documents=[
            "Refunds are available for annual plans within thirty days.",
            "Password reset links expire after fifteen minutes.",
            "Invoice exports are available to billing administrators.",
            "Shipping delays are reported by warehouse region.",
        ],
        embeddings=[
            [0.95, 0.05, 0.01],
            [0.02, 0.90, 0.05],
            [0.88, 0.12, 0.02],
            [0.01, 0.10, 0.92],
        ],
        metadatas=[
            {"domain": "billing"},
            {"domain": "security"},
            {"domain": "billing"},
            {"domain": "logistics"},
        ],
    )

    result = collection.query(query_embeddings=[[0.90, 0.10, 0.01]], n_results=3)
    ranked_ids = result["ids"][0]

    assert recall_at_k(ranked_ids, {"refund-policy", "invoice-export"}, 3) == 1.0

This test is intentionally tiny. It proves the harness mechanics: create a collection, insert documents with embeddings, issue a query vector, compute recall. Larger suites should load benchmark cases from JSON or a database and report failures with query text, expected IDs, actual IDs, scores, and applied filters.

Use deterministic tests like this for smoke coverage around client integration and metric calculation. Use a larger offline benchmark for serious model or index changes.

Testing metadata filters without confusing them with ranking

Vector retrieval often combines approximate nearest neighbor search with metadata filters: tenant, language, region, permission, content type, date range, or product line. Filter defects are high impact because they either leak data or hide relevant documents. Test filters separately and together with recall.

import chromadb


def test_tenant_filter_blocks_cross_tenant_neighbors():
    client = chromadb.Client()
    collection = client.create_collection("tenant_filter_contract")

    collection.add(
        ids=["acme-runbook", "globex-runbook", "acme-release-notes"],
        documents=[
            "Acme database failover runbook",
            "Globex database failover runbook",
            "Acme release notes for billing service",
        ],
        embeddings=[
            [0.99, 0.01],
            [0.98, 0.02],
            [0.40, 0.60],
        ],
        metadatas=[
            {"tenant": "acme", "type": "runbook"},
            {"tenant": "globex", "type": "runbook"},
            {"tenant": "acme", "type": "release"},
        ],
    )

    result = collection.query(
        query_embeddings=[[1.0, 0.0]],
        n_results=2,
        where={"tenant": "acme"},
    )

    assert result["ids"][0] == ["acme-runbook", "acme-release-notes"]
    assert "globex-runbook" not in result["ids"][0]

The nearest vector is almost tied across tenants, which is exactly the point. The test would catch a missing tenant filter because globex-runbook is highly similar. A weak fixture where cross-tenant data is not similar would pass even if the filter were broken.

Build adversarial fixtures for filters. Same title across tenants, same document in different languages, same SKU in two catalogs, and same policy in two regions are better than random examples. Filter tests should be designed to fail loudly when the filter is omitted or applied in the wrong stage.

Approximate indexes and recall loss

Most production vector databases use approximate nearest neighbor indexes because exact search is expensive at scale. HNSW, IVF, product quantization, and disk-backed variants can trade recall for speed and memory. That tradeoff may be acceptable, but it must be measured.

When changing index parameters, compare against an exact or high-recall baseline if the database supports it, or compare against a previously approved configuration. Run the same query set through both configurations and report recall deltas by segment. A global average hides the cases users notice.

Important parameters vary by database, so avoid cargo-cult values. HNSW settings that improve recall may increase memory. Quantization may reduce memory but hurt close semantic distinctions. Filtering can interact with indexing in surprising ways if the engine searches first and filters later, or filters first and then searches a smaller candidate set.

Embedding model and chunking regression tests

An embedding model upgrade can change retrieval quality even when the vector database is untouched. The same is true for chunk size, overlap, markdown cleaning, PDF extraction, HTML boilerplate removal, and title prefixing. Treat the embedding pipeline as part of the retrieval system.

For model changes, freeze a benchmark corpus and queries, generate embeddings with the old and new pipelines, and compare metrics. Look at query groups. A new model may improve natural language questions while hurting exact error code lookup. That is not automatically good or bad. It depends on the product.

For chunking changes, evaluate both retrieval and answer assembly. A relevant document ID may still appear, but the chunk may no longer include the sentence needed by the model. Track expected chunk IDs or expected passage text for critical queries. In RAG systems, document-level recall can look healthy while passage-level recall regresses.

Score thresholds and calibration

Many applications apply score thresholds before showing or passing results to an LLM. Thresholds are product decisions, not universal truths. A cosine similarity of 0.78 can mean different things across embedding models, normalization choices, and domains. Test thresholds against labeled queries.

Create cases where no result should be returned. These are as important as positive recall cases. A support bot should not answer a payroll question from an unrelated shipping policy just because it must return something. Measure false positives separately from recall. If the product has an abstain path, test that path with out-of-domain queries.

Do not compare raw scores across model families unless you have calibrated them. After an embedding upgrade, threshold tests may need adjustment. The review should show precision and recall movement, not only a changed number.

Reporting failures developers can act on

A recall failure should include query text, query group, filters, expected IDs, actual IDs, scores if available, and corpus version. Without that context, developers have to reproduce the run before they can diagnose whether the issue is indexing, filtering, embedding, or labels.

For CI, fail on severe regressions and publish a report for smaller movements. Example policies: block if any safety-critical query loses all expected documents at k=5, block if tenant filter tests fail, warn if global recall drops less than an agreed tolerance, require review if a query group drops below threshold. Make the policy visible in the repository.

Store benchmark results over time. Trend lines help catch slow degradation from content growth. Vector systems can regress when the corpus expands because similar documents crowd the top k. A test that passed with 10,000 chunks may fail with 1,000,000 chunks even if the code did not change.

Freshness, deletes, and reindexing

Recall tests should include lifecycle operations, not only initial indexing. Insert a document, query for it, update its text or metadata, query again, delete it, and verify it disappears. Stale vectors are common in systems with asynchronous indexing queues.

If your architecture uses batch indexing, define the freshness contract. Is a document searchable within one minute, five minutes, or only after a nightly job? Tests should match that contract. For event-driven indexing, include idempotency checks: duplicate events should not create duplicate search results, and delete events should remove or tombstone vectors consistently.

Reindex migrations need special care. During a rebuild, the application may read from old and new indexes, dual-write, or switch aliases. Add tests that compare recall before and after the cutover. For multi-tenant data, verify that alias changes do not point a tenant to the wrong collection.

Rerankers and hybrid search in recall tests

Many production systems do not rely on vector similarity alone. They combine lexical search, metadata boosts, business rules, and rerankers. Test the stages separately before judging the combined output. A vector database recall test should answer whether the candidate set contains the relevant items. A reranker test should answer whether those candidates are ordered well. A hybrid search test should answer whether keyword-heavy and semantic-heavy queries both work.

If a reranker sees only the top 20 vector candidates, recall@20 becomes a hard ceiling. The reranker cannot rescue a document that never enters the candidate set. That is why retrieval tests should measure recall at the handoff point, not only final answer quality. For RAG, store the candidate IDs sent to the model. When an answer fails, you need to know whether retrieval missed the evidence or generation ignored it.

Hybrid search adds another risk: one stage can dominate the other. A keyword boost may bury semantically correct passages. A vector score may outrank exact SKU matches. Create benchmark groups for exact identifiers, synonyms, natural language questions, and mixed queries. Look at each group after ranking changes.

Permissions and personalization

Vector systems often sit behind authorization logic. A result can be relevant and still forbidden. Recall tests should therefore distinguish global relevance from permitted relevance. For a given user, the expected set should include only documents that user is allowed to see. For admin or support roles, create separate expectations.

Permission-aware retrieval is easy to break during optimization. A team may precompute nearest neighbors globally, then apply permissions too late and return too few results. Another team may filter first and accidentally remove shared documents. Both defects show up as recall drops for permissioned query groups.

Personalization needs the same discipline. If search boosts a user's recent projects, test whether canonical answers remain findable. Personalization should improve relevance, not erase essential documents. Keep non-personalized benchmark runs as a control so you can tell whether a regression came from the base index or the personalization layer.

Data freshness benchmarks for RAG operations

RAG users often notice freshness failures before they notice small ranking changes. They ask about a newly published policy and receive last quarter's answer. Add benchmark cases for newly inserted, updated, and deleted documents. The expected result should include timing: searchable within the documented indexing window, updated metadata reflected, deleted content unavailable.

For asynchronous pipelines, record the indexing event ID and completion time. A failed freshness test should tell you whether the document never entered the queue, failed embedding, failed upsert, or was written to the wrong collection. Without pipeline observability, recall testing becomes a black box.

Freshness tests should run against a controlled environment, not an eventually changing production corpus. Use a test collection or tenant where the suite owns all documents. That keeps failures attributable to the pipeline rather than unrelated content edits.

Golden queries are product assets

Treat the recall benchmark as a product artifact, not a QA side file. Each golden query should have an owner, intent category, expected relevant IDs, and a note explaining why it matters. That context helps reviewers decide whether a changed result is an improvement, a label problem, or a regression.

Refresh the benchmark when the corpus changes meaningfully. New product lines, new policy sections, retired documents, and changed terminology can make old labels misleading. Keep retired queries when they represent supported historical user language, but remove them when they point to content that should no longer be retrieved.

Use failures to improve labels too. Sometimes the database returns a document that is clearly relevant but missing from the expected set. That is not a retrieval defect. It is a benchmark defect. Allow label updates, but require review so teams do not quietly bless every candidate output.

That discipline keeps recall metrics credible during model and corpus evolution reviews.

Frequently Asked Questions

What is a good recall@k threshold for a vector database?

There is no universal threshold. Set targets by product risk and query group. A legal lookup or permissions assistant may require near-perfect recall for curated queries, while exploratory search may tolerate lower recall if precision and user controls are strong.

Should recall tests use real embeddings or fixed vectors?

Use both. Fixed vectors are good for fast integration and filter tests because they are deterministic. Real embeddings are necessary for measuring model, chunking, and corpus quality. Keep those benchmark runs versioned and reproducible.

How do I test metadata filters in vector search?

Create adversarial fixtures where the most similar document is outside the allowed filter. Then query with the filter and assert that only permitted IDs appear. Random fixtures often fail to expose missing or incorrectly applied filters.

Can approximate nearest neighbor indexes be tested deterministically?

You can make the evaluation deterministic enough by pinning data, index parameters, database version, and query set. Some engines may still have build-order effects, so compare metrics over a benchmark rather than asserting every large-corpus rank exactly.

Why did recall drop after adding more documents?

New chunks can crowd the top k, especially when they are semantically similar to older relevant chunks. Review chunking, metadata filters, reranking, and whether k is large enough for the downstream model or UI.