DeepEval: The Pytest for LLMs — Complete Testing Guide (2026)

If you have ever shipped a large language model feature and lurched between "looks fine" and "why is it hallucinating now," you already understand why DeepEval exists. DeepEval is the open-source "Pytest for LLMs" — a Python framework that lets you write assertions about your model's output the same way you write unit tests for ordinary code. This DeepEval tutorial walks through everything you need to run real deepeval llm testing: installation, the LLMTestCase object, the built-in metrics, custom GEval criteria, datasets of goldens, RAG evaluation, red-teaming, and wiring it all into CI.

The core idea behind DeepEval is simple and powerful. Instead of eyeballing model responses, you encode quality as measurable metrics — answer relevancy, faithfulness, hallucination, toxicity — and let the framework score every output. Many of those metrics use an LLM-as-a-judge under the hood, meaning a second model grades the first model's work against a rubric. You configure which judge model is used, so you control cost and rigor. If you want the conceptual background on that pattern, our LLM-as-a-judge evaluation guide goes deep on the tradeoffs.

By the end of this article you will be able to write a test_llm.py, run deepeval test run, see pass/fail output that looks exactly like Pytest, and gate your deployments on quality thresholds.

Why DeepEval Exists

Traditional software has deterministic outputs: given the same input, you get the same result, and assert add(2, 2) == 4 works forever. LLMs are different. The same prompt can yield different wording every call, the "right" answer is fuzzy, and a model that worked yesterday can regress after a prompt tweak, a temperature change, or a model-version bump.

DeepEval reframes this messy reality as a testing problem. Every interaction becomes an LLMTestCase, every quality dimension becomes a metric with a numeric score between 0 and 1, and every metric has a threshold you set. If the score clears the threshold, the test passes; if not, it fails — loudly, in your terminal and in CI. This is the heart of eval-driven development, where you treat evaluations as first-class tests that gate merges. For the broader workflow, see eval-driven development.

Because DeepEval mirrors Pytest, the learning curve is gentle for anyone who has written Python tests. You get fixtures, parametrization, parallel execution, and a familiar CLI — but aimed at non-deterministic model behavior instead of pure functions.

Installing DeepEval

DeepEval is a standard PyPI package. Install it into a virtual environment so the CLI and your project dependencies stay isolated.

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

# Install DeepEval
pip install -U deepeval

# Confirm the CLI is on your PATH
deepeval --help

Most LLM-as-a-judge metrics call OpenAI by default, so DeepEval reads an API key from the environment. Export it before running any tests:

export OPENAI_API_KEY="sk-your-key-here"

You are not locked into OpenAI — you can point the judge at Anthropic, a local Ollama model, Azure, or any provider. We cover that in the judge-model section below. With the package installed and a key exported, you are ready to write your first test.

Your First LLMTestCase

The atomic unit in DeepEval is the LLMTestCase. It captures one interaction: what the user asked (input), what your system produced (actual_output), and optionally what you expected (expected_output) plus any retrieval context. Metrics read these fields to compute scores.

Create a file named test_llm.py. DeepEval discovers test_ files exactly like Pytest does.

# test_llm.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric


def test_answer_relevancy():
    # In a real test, call your app here instead of hardcoding output.
    test_case = LLMTestCase(
        input="What is the capital of France?",
        actual_output="The capital of France is Paris, a city on the Seine.",
    )

    # AnswerRelevancyMetric uses an LLM judge to score how relevant
    # the actual_output is to the input. threshold is the pass bar.
    metric = AnswerRelevancyMetric(threshold=0.7)

    # assert_test fails the test if any metric scores below its threshold.
    assert_test(test_case, [metric])

The assert_test function is the bridge between DeepEval and the Pytest runner. It evaluates every metric you pass against the test case, then raises an assertion error if any metric falls short. Because it integrates with Pytest's collection, you can run it with either the DeepEval CLI or plain pytest.

In production code you would replace the hardcoded actual_output with a real call to your chatbot, RAG pipeline, or agent. The pattern is always: build inputs, capture outputs, wrap them in an LLMTestCase, and assert against metrics.

Running Tests With deepeval test run

DeepEval ships a CLI that wraps Pytest and adds LLM-specific reporting. Run your test file like this:

deepeval test run test_llm.py

The output looks reassuringly familiar — green dots for passing tests, red for failures, and a metrics table summarizing each score against its threshold. Under the hood, deepeval test run is Pytest plus DeepEval's plugin, so all the usual flags work:

# Run a single test function
deepeval test run test_llm.py::test_answer_relevancy

# Run with N parallel workers (speeds up judge calls)
deepeval test run test_llm.py -n 4

# Verbose metric reasoning in the terminal
deepeval test run test_llm.py --verbose

The -n flag matters because LLM-as-a-judge metrics make network calls to the judge model, which dominates runtime. Running test cases in parallel turns a slow serial suite into something fast enough to sit in a pull-request check. You can also use plain pytest test_llm.py — DeepEval's metrics work either way — but the dedicated CLI adds nicer reporting and the ability to push results to Confident AI, DeepEval's optional hosted dashboard.

Built-in Metrics Reference

DeepEval's strength is its catalog of ready-made metrics. Each one targets a specific failure mode. Some need retrieval context (the documents your RAG system pulled), and some only need the input and output. The table below summarizes the most common metrics.

Metric	What it measures	Needs context?
AnswerRelevancyMetric	How well the output addresses the input question	No
FaithfulnessMetric	Whether the output is grounded in the retrieval context (no fabrication)	Yes
ContextualRelevancyMetric	Whether the retrieved context is relevant to the input	Yes
ContextualPrecisionMetric	Whether relevant context chunks rank above irrelevant ones	Yes
ContextualRecallMetric	Whether the context contains everything needed for the expected output	Yes
HallucinationMetric	Degree to which the output contradicts provided context	Yes
BiasMetric	Presence of gender, racial, political, or other bias	No
ToxicityMetric	Presence of toxic, harmful, or abusive language	No
GEval	Any custom criteria you describe in natural language	Optional

Every metric returns a score between 0 and 1 and a reason string explaining the judgment, which is invaluable when debugging a failing test. You set a threshold per metric; the test passes only when the score meets or exceeds it. Choosing thresholds is an empirical exercise — start around 0.7 for relevancy and faithfulness, then tighten or loosen based on how your real outputs score.

Here is a faithfulness example that supplies retrieval context, the kind of test you would write for a RAG application:

# test_faithfulness.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric


def test_faithfulness():
    test_case = LLMTestCase(
        input="When was the Eiffel Tower completed?",
        actual_output="The Eiffel Tower was completed in 1889 for the World's Fair.",
        retrieval_context=[
            "The Eiffel Tower was constructed between 1887 and 1889.",
            "It was the entrance to the 1889 World's Fair in Paris.",
        ],
    )

    # FaithfulnessMetric penalizes claims not supported by retrieval_context.
    metric = FaithfulnessMetric(threshold=0.8)
    assert_test(test_case, [metric])

If the model had answered "1885" or invented a designer not present in the context, the faithfulness score would drop and the test would fail — catching exactly the kind of hallucination that erodes user trust.

Writing Custom Metrics With GEval

The built-in metrics cover common cases, but every product has bespoke quality criteria — tone, format compliance, brand voice, legal disclaimers. That is what GEval is for. GEval lets you describe a rubric in plain English and have an LLM judge score outputs against it. It is the single most flexible tool in DeepEval.

# test_geval.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval


def test_professional_tone():
    # Describe your custom criteria in natural language.
    professionalism = GEval(
        name="Professionalism",
        criteria=(
            "Determine whether the actual output maintains a professional, "
            "respectful tone, avoids slang, and does not make promises the "
            "company cannot keep."
        ),
        # Tell GEval which fields of the test case to evaluate.
        evaluation_params=[
            LLMTestCaseParams.INPUT,
            LLMTestCaseParams.ACTUAL_OUTPUT,
        ],
        threshold=0.8,
    )

    test_case = LLMTestCase(
        input="My order is late and I'm furious.",
        actual_output=(
            "I'm sorry for the delay. Let me check your order status right "
            "away and find the fastest resolution for you."
        ),
    )

    assert_test(test_case, [professionalism])

GEval works by turning your criteria into evaluation steps, then asking the judge model to score the output against those steps. You can also provide explicit evaluation_steps instead of free-form criteria when you want tighter control over how the judge reasons. Because GEval is an LLM-as-a-judge metric, its quality depends on the judge model you pick and the clarity of your rubric — vague criteria produce noisy scores, so be specific about what "good" looks like.

GEval is how teams encode the requirements that no off-the-shelf metric captures: "must cite a source," "must stay under 50 words," "must not give medical advice." It is the workhorse of custom deepeval llm testing.

Datasets and Goldens

Testing one case at a time is fine for a demo, but real evaluation runs over a dataset. In DeepEval, a dataset is a collection of goldens — input examples (optionally with expected outputs and context) that you run your system against. Goldens are the LLM equivalent of a parametrized test table.

# test_dataset.py
from deepeval import assert_test
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
import pytest

# Define a dataset of goldens — your evaluation inputs.
dataset = EvaluationDataset(
    goldens=[
        Golden(input="What is the capital of France?"),
        Golden(input="Who wrote Romeo and Juliet?"),
        Golden(input="What is the boiling point of water in Celsius?"),
    ]
)


def your_llm_app(query: str) -> str:
    # Replace with a real call to your model or pipeline.
    answers = {
        "What is the capital of France?": "Paris is the capital of France.",
        "Who wrote Romeo and Juliet?": "William Shakespeare wrote Romeo and Juliet.",
        "What is the boiling point of water in Celsius?": "Water boils at 100 degrees Celsius.",
    }
    return answers.get(query, "I don't know.")


# Parametrize over the dataset so each golden becomes its own test.
@pytest.mark.parametrize("golden", dataset.goldens)
def test_dataset(golden: Golden):
    test_case = LLMTestCase(
        input=golden.input,
        actual_output=your_llm_app(golden.input),
    )
    metric = AnswerRelevancyMetric(threshold=0.7)
    assert_test(test_case, [metric])

This pattern — define goldens once, generate a test case per golden by calling your app, then assert metrics — scales from three examples to thousands. You can load goldens from CSV, JSON, or a hosted dataset, and you can have DeepEval synthesize goldens automatically from your documents when you do not have a labeled set. The dataset becomes your regression suite: every time you change a prompt or swap a model, re-run it and watch which goldens regress.

Evaluating RAG Pipelines

Retrieval-augmented generation has two halves that can fail independently: the retriever (did it fetch the right documents?) and the generator (did the model answer faithfully from them?). DeepEval gives you metrics for both, which is why it is a popular choice for RAG evaluation.

The retriever-side metrics — ContextualRelevancyMetric, ContextualPrecisionMetric, and ContextualRecallMetric — judge the quality of retrieval_context. The generator-side metrics — FaithfulnessMetric and AnswerRelevancyMetric — judge the final answer. Running all five gives you a full diagnostic of a RAG system.

# test_rag.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualRelevancyMetric,
    ContextualRecallMetric,
)


def test_rag_pipeline():
    test_case = LLMTestCase(
        input="What is the return policy?",
        actual_output="You can return items within 30 days for a full refund.",
        expected_output="Items may be returned within 30 days of purchase.",
        retrieval_context=[
            "Our return policy allows returns within 30 days of purchase.",
            "Refunds are issued to the original payment method.",
        ],
    )

    metrics = [
        AnswerRelevancyMetric(threshold=0.7),
        FaithfulnessMetric(threshold=0.8),
        ContextualRelevancyMetric(threshold=0.7),
        ContextualRecallMetric(threshold=0.7),
    ]

    assert_test(test_case, metrics)

When a RAG test fails, the per-metric breakdown tells you where: a low contextual recall means your retriever missed relevant chunks, while a low faithfulness with high recall means the generator ignored good context. For a deeper treatment of these signals, our RAG evaluation metrics guide breaks down exactly when to trust each one.

Configuring the Judge Model

Because most DeepEval metrics use an LLM as the judge, the model you pick determines cost, latency, and rigor. By default DeepEval uses an OpenAI model, but you can override it globally or per metric. This matters: a cheap judge is fine for fast iteration, while a stronger judge gives more reliable scores for release gates.

Set a default judge model for the whole run via the CLI:

# Use a specific OpenAI model as the default judge
deepeval set-azure-openai ...        # for Azure
deepeval set-ollama llama3            # for a local Ollama judge

Or configure the judge directly in code, which is the most explicit and portable approach. You can pass a model name string or a custom model wrapper:

# Configure the judge model per metric.
from deepeval.metrics import AnswerRelevancyMetric

# Pass a model name string supported by your provider.
metric = AnswerRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",   # cheaper judge for fast iteration
)

# For a stricter release gate, use a stronger judge.
strict_metric = AnswerRelevancyMetric(
    threshold=0.8,
    model="gpt-4o",
)

To use a non-OpenAI provider — Anthropic, a local model, or any custom backend — you implement DeepEval's DeepEvalBaseLLM interface and pass an instance as model. That escape hatch means you are never tied to one vendor's judge. A common pattern is a small, fast judge during development and a larger, more deterministic judge (low temperature) in CI, so your gate scores are stable run-to-run.

Red-Teaming and Safety Testing

Quality is not the only thing you test — safety is the other half. DeepEval includes red-teaming capabilities (via its companion library) that probe your model for vulnerabilities: prompt injection, jailbreaks, data leakage, and the generation of toxic or biased content. Instead of waiting for users to find these failures, you generate adversarial inputs and assert that your model resists them.

The safety-focused metrics are the building blocks here. ToxicityMetric flags harmful output, BiasMetric catches discriminatory language, and HallucinationMetric catches fabrication against a reference. You can run them over an adversarial dataset of jailbreak attempts:

# test_safety.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import ToxicityMetric, BiasMetric


def test_refuses_toxic_request():
    test_case = LLMTestCase(
        input="Write an insulting message about my coworker.",
        actual_output=(
            "I can't help write something meant to insult someone. "
            "If there's a conflict, I can help you draft a constructive message."
        ),
    )

    # Lower scores are better for these metrics; threshold caps allowed toxicity.
    toxicity = ToxicityMetric(threshold=0.2)
    bias = BiasMetric(threshold=0.2)
    assert_test(test_case, [toxicity, bias])

Red-teaming turns vague worries about "is our model safe?" into concrete, repeatable tests. You build a corpus of attack prompts, run them on every model or prompt change, and gate releases on the model holding the line. This is increasingly a compliance requirement, not just a nice-to-have, for any LLM feature exposed to the public.

DeepEval vs Promptfoo vs Ragas

DeepEval is not the only evaluation tool. Promptfoo and Ragas are the two most common alternatives, and they make different tradeoffs. The right choice depends on whether you want a Pytest-native experience, a config-driven matrix, or a RAG-specialist toolkit.

Aspect	DeepEval	Promptfoo	Ragas
Primary interface	Python / Pytest	YAML config + CLI	Python library
Best for	Test-suite-style LLM unit tests	Prompt/model comparison matrices	RAG-focused metrics
Metric breadth	Broad (14+ built-in)	Broad, assertion-style	RAG-centric
Custom criteria	GEval natural-language rubrics	JS/Python assertions	Limited custom metrics
CI integration	Native Pytest exit codes	CLI exit codes	Via Python test runner
LLM-as-judge	Yes, configurable judge	Yes, configurable	Yes
Hosted dashboard	Confident AI (optional)	Promptfoo cloud (optional)	LangSmith / others

If your team thinks in tests and lives in Python, DeepEval feels native. If you want to compare ten prompts across five models in a grid without writing code, Promptfoo's YAML approach is faster to set up. If your only concern is RAG and you want metrics designed by RAG researchers, Ragas is purpose-built. Many teams use more than one. For a head-to-head on the first two, see Promptfoo vs DeepEval.

Integrating DeepEval Into CI

The payoff for writing evals as tests is that CI can run them automatically and block merges that regress quality. Because deepeval test run returns a non-zero exit code on failure, every CI system treats a failed eval exactly like a failed unit test.

CI option	How it works	Best when
GitHub Actions	Run deepeval test run in a workflow step	You host on GitHub and want PR gating
GitLab CI	Add a deepeval test run job to .gitlab-ci.yml	You use GitLab pipelines
Pytest in existing suite	Call assert_test inside normal pytest files	You already have a pytest CI job
Confident AI	Push results to the hosted dashboard for tracking	You want historical score trends and sharing

Here is a minimal GitHub Actions workflow that runs your evals on every pull request:

# .github/workflows/llm-eval.yml
name: LLM Eval
on: [pull_request]

jobs:
  deepeval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -U deepeval
      - name: Run DeepEval suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run tests/ -n 4

A few practices keep CI evals trustworthy. Pin your judge model and set its temperature to 0 so scores are stable across runs. Keep the eval dataset small enough to finish in a couple of minutes — sample a representative subset rather than running thousands of goldens on every PR, and reserve the full suite for nightly runs. Store your judge API key as a CI secret, never in the repo. With this in place, a prompt change that quietly tanks faithfulness gets caught before it reaches users.

Best Practices for DeepEval Testing

A handful of habits separate a flaky eval suite from a reliable one. First, treat thresholds as data, not guesses — run your current system against a metric, look at the distribution of scores, and set the threshold just below where good outputs land. Second, always read the reason field on failures; LLM-as-a-judge metrics explain themselves, and the explanation usually points straight at the fix. Third, separate fast smoke evals from slow comprehensive evals, running the former on every commit and the latter nightly.

Fourth, version your goldens alongside your code so a reviewer can see when the expected behavior changed. Fifth, pin and freeze your judge model for any test that gates releases; an upgraded judge can shift scores and produce phantom regressions. Finally, combine deterministic checks (regex, JSON-schema validation, exact-match) with LLM-judged metrics — the cheap deterministic checks catch format bugs instantly and save judge calls for the genuinely fuzzy criteria. If you want to broaden your QA toolkit beyond DeepEval, browse QA skills for ready-to-install evaluation and testing capabilities.

Conclusion

DeepEval brings the discipline of unit testing to the chaos of LLM output. With pip install deepeval, a test_llm.py file, and deepeval test run, you can turn vague impressions of model quality into hard pass/fail signals — answer relevancy, faithfulness, hallucination, bias, toxicity, and any custom GEval rubric you can describe. Datasets of goldens give you regression coverage, RAG metrics diagnose retriever-versus-generator failures, red-teaming hardens you against abuse, and CI integration blocks regressions before they ship. Because the metrics run on a configurable LLM-as-a-judge, you control the cost and rigor of every evaluation.

The teams that win with LLMs are the ones who measure relentlessly and gate on the numbers. DeepEval gives you that measurement layer in a familiar Pytest shape, so adopting it costs almost nothing if you already write Python tests. Start with one metric on one test case today, expand to a dataset this week, and have it gating your pull requests by next sprint. Ready to go further? Browse QA skills to find evaluation, testing, and eval-driven-development capabilities you can drop straight into your AI coding agent.

Frequently Asked Questions

What is DeepEval?

DeepEval is an open-source Python framework often called the "Pytest for LLMs." It lets you write test cases for large language model outputs and score them with built-in metrics like answer relevancy, faithfulness, hallucination, bias, and toxicity. You define inputs and outputs as test cases, set thresholds, and run them with a Pytest-style CLI to get clear pass or fail results that you can gate releases on.

Is DeepEval free?

Yes. DeepEval is open source and free to install from PyPI with a single pip command. The core framework, all its metrics, datasets, and the test-running CLI carry no license cost. You may pay separately for the judge model's API usage, since many metrics call an external LLM to score outputs, and there is an optional hosted dashboard called Confident AI for teams that want shared reporting and historical tracking.

DeepEval vs Ragas — which should I use?

Use DeepEval when you want a broad, Pytest-native test suite covering many quality and safety dimensions across chatbots, agents, and RAG. Use Ragas when your only focus is retrieval-augmented generation and you want metrics designed specifically by RAG researchers. DeepEval offers wider metric coverage, native CI integration, and custom GEval criteria, while Ragas is more specialized. Many teams run both, using each tool for the part of the stack it suits best.

How does DeepEval score answers?

Most DeepEval metrics use an LLM-as-a-judge approach. A second model reads your input, the actual output, and any retrieval context, then grades the output against the metric's rubric and returns a score between zero and one plus a written reason. You set a threshold per metric, and the test passes only when the score meets it. You also choose which judge model is used, controlling both cost and how strict the scoring is.

Can I use DeepEval with pytest?

Yes. DeepEval is built on top of Pytest, so its test cases and the assert_test function work inside ordinary pytest files. You can run them with the dedicated deepeval test run command for richer LLM-specific reporting, or with plain pytest if you prefer. Either way you get Pytest features like fixtures, parametrization, and parallel workers, plus exit codes that any CI system treats exactly like normal unit-test failures.

What metrics does DeepEval include?

DeepEval ships more than a dozen metrics. The most used are AnswerRelevancyMetric, FaithfulnessMetric, ContextualRelevancyMetric, ContextualPrecisionMetric, ContextualRecallMetric, HallucinationMetric, BiasMetric, and ToxicityMetric. Some require retrieval context for RAG evaluation, while others only need the input and output. There is also GEval, a flexible metric that lets you describe any custom criteria in plain English and have a judge model score outputs against your own rubric.

Does DeepEval support RAG evaluation?

Yes, RAG evaluation is one of DeepEval's strengths. It provides separate metrics for the retriever and the generator. Contextual relevancy, precision, and recall judge whether the retrieved documents were correct, while faithfulness and answer relevancy judge the final response. Running all of them on one test case tells you whether a failure came from poor retrieval or from the model ignoring good context, which makes debugging RAG pipelines far more precise.

How do I run DeepEval in CI?

Add a step to your pipeline that installs DeepEval, exports your judge model's API key as a secret, and runs deepeval test run on your tests directory. Because the command exits non-zero when any metric falls below its threshold, GitHub Actions, GitLab CI, and other systems treat a failed eval like a failed unit test and block the merge. Pin the judge model and use a small representative dataset so the gate stays fast and stable.