Generate Synthetic RAG Testsets with Ragas and Your Documents

RAG synthetic testset generation turns a reviewed document collection into candidate questions, reference material, and scenario metadata that can exercise retrieval and generation. Ragas can build those candidates from documents through a knowledge graph, transforms, scenarios, and query synthesizers. The generated rows are not ground truth by default: validate them, remove leakage, measure sampling bias, protect document privacy, and reserve a held-out portion before using results for release decisions.

Use the complete RAG evaluation metrics guide to choose scorers, then connect the generated set to the retrieval-versus-generation diagnostic workflow, the context precision, recall, and relevancy guide, and the high-relevance, low-faithfulness investigation. The LLM application testing guide covers evaluator reliability, while the existing RAG benchmark dataset guide helps frame benchmark governance. Browse reusable workflows in /skills and use the Playwright CLI skill when test cases must be replayed through a browser UI.

What Ragas Generates and What It Does Not

Ragas describes synthetic test data generation as a way to reduce the manual work of curating diverse evaluation samples. Its current RAG testset documentation presents two main operations:

Build a KnowledgeGraph from source documents and enrich it with transforms.
Generate scenarios and test samples from that graph with query synthesizers.

The documented default synthesizer family includes single-hop specific, multi-hop abstract, and multi-hop specific queries. These categories help vary reasoning structure. They do not guarantee representation of your production languages, user roles, security boundaries, rare policy exceptions, malformed queries, or real distribution.

Output	Useful for	Must still be validated	Not guaranteed
Synthetic user query	Expanding scenario coverage	Naturalness, answerability, persona fit	Production frequency
Reference answer or expected information	Context recall and answer checks	Accuracy, completeness, qualifiers	Independent ground truth
Reference contexts or source lineage	Retrieval labels and provenance	Correct revision, authorization, exact support	Safe disclosure
Synthesizer or scenario metadata	Slice analysis	Correct classification	Balanced sampling
Knowledge graph relationships	Multi-hop scenario construction	Domain validity and privacy	Business-authoritative ontology

A generated testset is best treated as a candidate pool. Human-reviewed production failures and domain-authored critical cases remain essential. Synthetic generation is valuable because it can broaden combinations and expose neglected document relationships, not because it replaces subject-matter review.

July 2026 Version Baseline

This tutorial was verified on July 14, 2026. PyPI lists Ragas 0.4.3 as the current release and requires Python 3.9 or newer. Pin ragas==0.4.3 in a reproducible environment for these examples, then review migration and testset documentation before upgrading.

Ragas 0.4 changed metric APIs substantially: collections metrics return structured MetricResult objects and experiment-oriented workflows replace new uses of the legacy evaluation path. Testset generation uses a different surface. Current official testset documentation still shows TestsetGenerator, generate_with_langchain_docs, KnowledgeGraph, default_transforms, and default_query_distribution. Do not infer that metrics migration names apply to testset classes.

There is also a documentation detail worth making explicit: a rendered getting-started example shows one default query distribution, while the current synthesizer reference implementation builds an equal distribution across compatible default synthesizers and can filter them when a knowledge graph is supplied. Do not hard-code weights copied from a screenshot or prose example. Inspect the distribution returned by the pinned version, choose an intentional distribution for your dataset, and save it with the run manifest.

Prerequisites and Data Authorization

Before installing a package or sending a document to a generator model, answer these questions:

Which corpus snapshot is approved for evaluation generation?
Does the selected LLM or embedding provider permit this data classification and region?
May document text leave the environment, and what retention terms apply?
Which fields contain personal, customer, credential, contractual, or regulated information?
Which user roles are allowed to see each source?
Who can validate generated questions and references?
Which production traffic slices must the candidate pool represent?
Which split will remain held out from prompt, retriever, and threshold tuning?

Use synthetic or redacted documents for initial pipeline development. A local-looking Python process can still send text to remote LLM and embedding providers. Review every adapter, endpoint, logging path, cache, trace, notebook, and exported dataframe.

Create a clean virtual environment and pin dependencies in a lock file. The minimal Ragas command is:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install "ragas==0.4.3"
python -m pip freeze > requirements.lock.txt

Provider and document-loader packages depend on your approved stack. Install only the integration used by the code, and pin it too. Never place provider keys in notebooks, generated datasets, test prompts, or source documents.

Build a Corpus Manifest Before Loading Documents

Every document needs lineage. At minimum, record source ID, revision, checksum, classification, authorization labels, language, document type, effective dates, and inclusion reason. Exclude duplicate, superseded, unauthorized, malformed, and out-of-scope material before generation.

{
  "corpus_id": "support-policies-2026-07-01",
  "documents": [
    {
      "source_id": "refund-policy-us-v4",
      "revision": "2026-06-28",
      "sha256": "replace-with-real-content-checksum",
      "language": "en-US",
      "classification": "internal-test-approved",
      "allowed_roles": ["support-agent"],
      "effective_from": "2026-07-01",
      "supersedes": "refund-policy-us-v3"
    }
  ],
  "generation_policy": {
    "remote_processing_approved": false,
    "redaction_profile": "policy-text-no-personal-data-v2"
  }
}

The placeholder checksum must be replaced by the pipeline; it is not a valid content hash. A manifest makes it possible to regenerate a candidate set, remove cases derived from a withdrawn source, and prove that a held-out benchmark was not silently refreshed with tuned material.

Avoid using live production chat logs as source documents without a separate privacy and consent process. Even when names are removed, combinations of account details, issue descriptions, and timestamps can identify users. Production intent can be represented through approved aggregate labels and separately governed examples.

Use the Document Convenience Workflow

The current Ragas getting-started guide documents generate_with_langchain_docs for LangChain documents and a corresponding method for LlamaIndex documents. The following wrapper accepts already approved and configured generator objects rather than assuming one provider. That keeps credentials and data-routing policy outside the tutorial.

from pathlib import Path
from ragas.testset import TestsetGenerator


def generate_candidate_pool(
    docs,
    generator_llm,
    generator_embeddings,
    testset_size: int,
    output_path: Path,
):
    generator = TestsetGenerator(
        llm=generator_llm,
        embedding_model=generator_embeddings,
    )
    testset = generator.generate_with_langchain_docs(
        docs,
        testset_size=testset_size,
    )
    frame = testset.to_pandas()
    frame.to_json(output_path, orient="records", lines=True)
    return frame

The wrapper is reproducible only if the caller records the document manifest, Ragas version, provider adapter versions, generator model, embedding model, prompt or synthesizer configuration, and random controls exposed by those components. The exported rows still need validation before entering an evaluation suite.

Start with a small, non-sensitive sample. Inspect every field and several source-to-question traces. Confirm that the generated question is answerable from its assigned evidence and that the reference does not add facts from model memory.

Understand the Knowledge Graph Workflow

For more control, the official guide constructs a KnowledgeGraph, adds document nodes, applies default_transforms, saves and reloads the graph, then creates a TestsetGenerator with that graph. This exposes the intermediate representation for inspection.

from ragas.testset import TestsetGenerator
from ragas.testset.graph import KnowledgeGraph, Node, NodeType
from ragas.testset.synthesizers import default_query_distribution
from ragas.testset.transforms import apply_transforms, default_transforms


def generate_from_graph(docs, generator_llm, generator_embeddings, size: int):
    graph = KnowledgeGraph()
    for doc in docs:
        graph.nodes.append(
            Node(
                type=NodeType.DOCUMENT,
                properties={
                    "page_content": doc.page_content,
                    "document_metadata": doc.metadata,
                },
            )
        )

    transforms = default_transforms(
        documents=docs,
        llm=generator_llm,
        embedding_model=generator_embeddings,
    )
    apply_transforms(graph, transforms)

    distribution = default_query_distribution(generator_llm, kg=graph)
    generator = TestsetGenerator(
        llm=generator_llm,
        embedding_model=generator_embeddings,
        knowledge_graph=graph,
    )
    return generator.generate(
        testset_size=size,
        query_distribution=distribution,
    )

This follows the current reference signature that accepts an optional knowledge graph when selecting compatible synthesizers. Before operational use, save the graph to an access-controlled artifact, print and record the returned distribution, and validate that the selected synthesizers match the graph. If a version raises because no synthesizer is compatible, fix corpus transforms or choose an explicit supported synthesizer; do not catch the error and silently generate another dataset.

Knowledge graph persistence can increase privacy exposure because enriched nodes and relationships may reveal information not obvious in individual documents. Apply the same or stronger classification, access, and retention controls as the source corpus.

Choose a Query Distribution from Risk and Traffic

A default distribution is a starting point, not a production model. Design the final mix from two views:

Traffic representation: common query types, languages, channels, roles, and source families.
Risk representation: rare but harmful failures, access boundaries, superseded policies, conflicts, unanswerable questions, and multi-hop evidence.

Slice	Why include it	Sampling risk	Validation focus
Single-hop specific	Tests direct retrieval of one fact	Can dominate because it is easy to generate	Source and qualifier correctness
Multi-hop specific	Tests joining explicit evidence	Synthetic joins may be unnatural	Every hop and relationship validity
Multi-hop abstract	Tests conceptual synthesis	Judge may reward vague references	Answerability and reference specificity
No-answer or missing evidence	Tests abstention	Generators tend to invent an answer	Confirm evidence is genuinely absent
Conflicting revisions	Tests authority handling	Both sources may look equally valid	Effective date and supersession labels
Permission boundary	Tests authorization-aware retrieval	Generated query may imply forbidden role	User role and allowed-source contract

Do not infer production performance from an intentionally balanced risk set without reweighting or separate reporting. Balanced datasets are useful for finding defects; traffic-weighted datasets estimate user exposure. Publish both views and label them.

Prevent Training and Evaluation Leakage

Leakage occurs when information from the test set influences the system or decision rule being evaluated. Synthetic generation does not prevent it.

Common leakage paths include:

Generating questions from the same chunks used to tune chunk size and selecting only cases that improve the chosen configuration.
Including reference answers in prompts, retrieval indexes, few-shot examples, or reranker training data.
Repeatedly editing prompts against held-out failures until they pass.
Choosing thresholds after inspecting the final benchmark.
Deduplicating by exact text while near-duplicate documents span train and held-out sets.
Regenerating a failed held-out case and replacing it with an easier variant.

Split by source lineage or topic group, not only random rows. Questions derived from adjacent overlapping chunks can be near duplicates. If one appears in tuning and another in held-out evaluation, the apparent independence is weak.

Use three named partitions:

Partition	May tune system?	May tune thresholds?	Primary use
Development	Yes	Preliminary only	Debug prompts, retrievers, fixtures, evaluators
Validation	Limited model selection	Yes, with recorded procedure	Compare candidates and calibrate gates
Held-out test	No	No	Final release estimate and audit

After a held-out set materially influences a fix, retire or demote affected cases to regression and create a newly governed held-out set. Keep the failure as a deterministic regression; do not continue describing it as unseen evidence.

Detect Sampling Bias

Synthetic generators reflect the input corpus, transform behavior, model preferences, and synthesizer configuration. They may overproduce well-structured English prose, named entities, short factual questions, or documents with rich metadata. Messy tables, scanned pages, minority languages, fragmented policies, and ambiguous user phrasing may be underrepresented.

Compare candidate and target distributions across:

Source family and document length.
Language, locale, and writing quality.
Query length and linguistic complexity.
Single-hop versus multi-hop structure.
Answerable, partially answerable, contradictory, and unanswerable cases.
User role and authorization level.
Freshness, effective date, and superseded-source status.
Tables, lists, images, OCR text, and plain prose.
Common traffic versus rare high-risk workflows.

Do not hide gaps in one aggregate coverage percentage. Publish counts for every required slice and reject a candidate pool that has zero examples for a critical class. Add domain-authored cases where synthesis cannot produce realistic behavior.

Validate Every Candidate in Stages

Validation should be stricter than metric scoring because it defines the evidence metrics will later consume.

Stage 1: schema validation. Require a stable ID, query, source lineage, reference, scenario type, generator manifest, split, and review status. Reject empty or malformed fields.

Stage 2: source validation. Verify that every referenced source exists in the approved corpus revision and that its authorization labels permit the intended test identity.

Stage 3: answerability validation. A reviewer confirms that supplied source spans support every reference claim and that no outside fact is required.

Stage 4: naturalness validation. Review whether a plausible user in the declared persona would ask the query without access to hidden document wording.

Stage 5: uniqueness validation. Detect exact and semantic near duplicates across all splits, grouped by source lineage.

Stage 6: adversarial validation. Add absent-evidence, conflicting-source, stale-source, distractor, permission, and prompt-injection cases deliberately.

Stage 7: pilot execution. Run baseline retrieval and generation, inspect score distributions and grader disagreements, and correct fixtures rather than automatically deleting hard cases.

case_id: refund-window-multihop-018
split: held-out
query: Which refund window applies to an annual plan renewed after July 1?
scenario: multi-hop-specific
source_ids:
  - billing-terms-2026-section-4
  - renewal-policy-2026-section-2
required_claims:
  - id: renewal-date-rule
    reviewer_status: approved
  - id: annual-plan-window
    reviewer_status: approved
privacy_review: approved-no-personal-data
leakage_group: annual-refund-policy-2026
review:
  answerable: true
  natural: true
  source_authorized: true
  reference_supported: true

The schema is an example governance contract, not a Ragas package type. Store it beside exported Ragas fields or transform it into your repository's dataset format.

Protect Document and Testset Privacy

Generated questions can reveal source details even when they do not quote text. A question about an unreleased product, employee case, customer incident, or confidential contract can itself be sensitive. References and knowledge graph relationships can expose more.

Apply these controls:

Classify source documents before loading.
Redact or replace personal and secret values at the source.
Use approved model and embedding endpoints for the classification.
Disable or govern provider retention and tracing according to policy.
Encrypt stored corpora, graphs, datasets, and run artifacts.
Restrict access by role and preserve source authorization labels.
Set retention and deletion procedures, including derived rows.
Scan generated output for secrets and personal data before export.
Use synthetic identifiers and accounts in executable fixtures.
Prevent test reports from publishing context or references to public CI logs.

Deletion needs lineage. If a source is withdrawn, locate graph nodes, generated questions, references, embeddings, cached prompts, and evaluation traces derived from it. A source ID and corpus manifest make that possible.

Connect the Testset to Evaluation and CI

After review and splitting, execute the RAG system on each query and preserve retrieved contexts plus answers. Promptfoo's current RAG guide recommends evaluating retrieval separately from output generation. That matches the dataset design: source IDs and references support retrieval checks; response and delivered context support faithfulness and relevance checks.

Run a small development subset on pull requests, impacted source slices when documents change, and the held-out set only at controlled release points. Avoid exposing held-out expected answers in routine developer logs. Store results by immutable dataset revision.

Release reporting should include:

Dataset and corpus revision.
Count by split and required slice.
Human validation and rejection rates.
Duplicate and leakage-group results.
Retrieval, generation, and end-to-end outcomes separately.
Candidate versus baseline differences with confidence or repeated-run context.
Grader and application versions.
Known coverage gaps and privacy constraints.

Do not report a synthetic testset score as production accuracy. It measures performance on a generated and reviewed distribution. Production monitoring and sampled human audits remain necessary.

Common Mistakes

Accepting generated references as truth. The same model that writes a question can invent a reference. Require source-level support review.

Using only default synthesis. Defaults do not know product traffic, risk, permissions, languages, or document formats. Design explicit slices.

Hard-coding displayed default weights. Current docs and reference behavior can evolve. Inspect and save the actual distribution for the pinned version.

Random row splitting. Overlapping chunks and related documents can leak near-duplicate facts across partitions. Split by source lineage or semantic group.

Sending confidential documents to an unapproved endpoint. Local code does not imply local processing. Audit LLM, embedding, logging, and cache destinations.

Tuning on the held-out set. Once failures guide changes, the set is no longer unseen. Preserve cases as regression tests and refresh held-out evidence under governance.

Deleting every difficult case. Hard cases may reveal fixture defects, but they may also expose real weaknesses. Adjudicate before removal and record the reason.

Confusing diversity with representativeness. A wide variety of synthetic questions can still badly misrepresent real traffic or high-risk scenarios.

Limits

Synthetic generation is bounded by source quality. Missing, stale, contradictory, or biased documents produce weak candidates. Knowledge graph transforms and model-generated relationships can add errors. Human review reduces but does not eliminate them.

Ragas API behavior and documentation can change. Pin package and adapter versions, use current official references, and compile a small canary generation job before upgrading. Do not assume a migration in metric APIs changes testset generation identically.

No synthetic set can enumerate open-ended user behavior. Production failures, adversarial research, domain expertise, and privacy review must continue to feed the evaluation program. The best dataset is a governed portfolio, not one generated file.

Synthetic Testset Checklist

FAQ: Synthetic RAG Testset

Does Ragas-generated data count as ground truth?

No. It is candidate evaluation data derived through documents, transforms, graph relationships, and generator models. Review source support, answerability, naturalness, privacy, and lineage before treating a row as an approved fixture.

Which Ragas version does this tutorial assume?

It assumes Ragas 0.4.3, listed by PyPI as current on July 14, 2026. Pin the package and integrations, then recheck official testset and migration documentation before upgrading.

Should I use the default query distribution unchanged?

Usually not. Inspect what the pinned version returns, then design a distribution around production traffic and high-risk gaps. Save the actual synthesizer classes and weights with the dataset manifest.

How do I prevent leakage between tuning and testing?

Split by source lineage or semantic group, keep references out of prompts and indexes, reserve a held-out set before tuning, and retire held-out cases once they materially guide a change. Exact row-level random splitting is insufficient for overlapping documents.

Can confidential documents be used safely?

Only under an approved data-processing design. Review LLM and embedding endpoints, provider retention, logging, caches, storage, access, and deletion. Prefer redacted or synthetic documents when possible and never assume that a local script keeps data local.

How many synthetic cases should I generate?

There is no universal number. Choose enough reviewed cases to cover required traffic and risk slices and to support the release decisions being made. More unvalidated rows can increase false confidence rather than evidence.

Should synthetic tests replace production examples?

No. Combine them with domain-authored critical cases, real failures, adversarial cases, and governed samples of production behavior. Each source covers a different part of the risk surface.

What should happen when a source document changes?

Use lineage to identify affected graph nodes and test rows, revalidate or retire them, rerun impacted slices, and preserve the old dataset revision for audit. Do not silently mutate references under the same version.

Conclusion

Ragas can efficiently transform documents into a broad candidate RAG testset through knowledge graphs, transforms, scenarios, and synthesizers. The engineering value appears only after governance: pin versions, trace every row to approved sources, validate references, measure sampling bias, protect private data, prevent leakage, and preserve a truly held-out set. Treat generation as the beginning of test design, not the end.