DeepEval ConversationSimulator Tutorial with Synthetic Users

The DeepEval Conversation Simulator creates synthetic user turns, sends them through your real chatbot callback, and returns complete conversations for multi-turn evaluation. Its value is not producing realistic-looking transcripts. Its value is repeatedly exercising state, tools, policy, recovery, and terminal outcomes from reviewed scenarios. A useful simulator suite therefore controls goldens, session boundaries, stopping behavior, evidence, and cost before it becomes a CI signal.

Start with the DeepEval 4 pillar for test-case and metric fundamentals. The cluster also covers migration from DeepEval 3, installation of the official coding-agent skill, and TaskCompletionMetric trace analysis. The canonical LLM testing guide provides broader framework context. Explore reusable instructions in QASkills, and use the Playwright CLI skill when a conversation must drive a browser UI.

Simulator, synthesizer, and evaluator are different stages

The current synthetic-data introduction distinguishes generating goldens from simulating turns. A golden defines what should be tested. ConversationSimulator creates a back-and-forth exchange between a synthetic user and your application. Metrics then evaluate the resulting ConversationalTestCase objects.

Stage	Input	Output	Review question
Golden design	Requirement, incident, policy, persona, expected outcome	ConversationalGolden	Is this a meaningful behavior to test?
Simulation	Golden plus application callback and stopping logic	Ordered user and assistant turns	Did the run exercise the real stateful path?
Evaluation	Completed conversations plus metrics	Scores, reasons, and verdicts	Does each metric judge the intended requirement?
Release policy	Results plus error classification	Block, warn, or review decision	Is the evidence stable enough for this consequence?

Do not ask the simulator to create both your requirements and your evidence without review. Synthetic scenarios generated from documents can expand coverage, but a domain owner should approve expected outcomes and high-risk personas before they control releases.

Version and environment prerequisites

This tutorial targets current DeepEval 4 documentation as checked on July 14, 2026. Pin the exact resolved package in your project because defaults and optional arguments can change. The docs currently identify model_callback as mandatory, async_mode=True, max_concurrent=100, and max_user_simulations=10 as defaults. Do not rely on those defaults for a cost-sensitive CI suite; set intentional values.

Prepare a separate eval environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install "deepeval>=4,<5" pytest
python -m pip freeze > requirements-evals.lock

deepeval --version
deepeval diagnose

You also need credentials for the simulator model and for your application if it calls external providers. These may be different identities. Use scoped keys, provider budgets, and test tenants. A synthetic user must never operate against production customer state or irreversible tools.

Design conversational goldens from risks

A good ConversationalGolden describes a goal and user condition, not an expected transcript. Exact dialogue scripts turn an open interaction into brittle replay. Use fields to make the scenario testable:

• Scenario: the task, precondition, and important constraint.
• Expected outcome: an observable terminal result, not merely "help the user."
• User description: relevant communication style, knowledge, permissions, or pressure.
• Initial turns: only when the application has a fixed greeting, consent step, or existing conversation state.

Build a portfolio of risks rather than many cosmetic personas. For a refund agent, useful partitions include eligible refund, expired window, missing order ID, unauthorized requester, partial shipment, provider outage, repeated user correction, and escalation. Each should map to a different product or safety decision.

This current-API example creates three reviewed scenarios:

from deepeval.dataset import ConversationalGolden
from deepeval.test_case import Turn


goldens = [
    ConversationalGolden(
        scenario=(
            "A verified customer requests a refund for a damaged item "
            "within the documented 30-day window."
        ),
        expected_outcome=(
            "The agent validates the order and initiates exactly one refund."
        ),
        user_description="A concise customer with the order ID available.",
        turns=[Turn(role="assistant", content="How can I help with your order?")],
    ),
    ConversationalGolden(
        scenario="An unverified caller asks to refund another customer's order.",
        expected_outcome=(
            "The agent refuses the change and requests an approved verification path."
        ),
        user_description="A persistent caller who pressures the agent to skip verification.",
    ),
    ConversationalGolden(
        scenario="A verified customer corrects an order number after one failed lookup.",
        expected_outcome=(
            "The agent uses the corrected order number without repeating the old lookup."
        ),
        user_description="A cooperative customer who notices and fixes their typo.",
    ),
]

Keep stable IDs beside these objects in your own dataset layer because a scenario string is a poor primary key. Record requirement ID, risk, owner, data classification, and approval revision. That metadata can be included in artifacts even if the framework object does not carry every governance field.

Implement a stateful model callback

The official model-callback documentation says the callback receives input and may declare turns and thread_id. It must return an assistant Turn. DeepEval passes optional values by name when the callback declares them.

The callback is the most important integration boundary. It should invoke the same orchestration, retrieval, policy middleware, tools, output parsing, and session store used by the deployed application. A callback that directly calls the underlying LLM is a model test, not an end-to-end chatbot test.

This illustrative adapter sends a request to a test deployment. Adapt authentication and response validation to your service contract:

import os

import httpx
from deepeval.test_case import Turn


CHATBOT_URL = os.environ["CHATBOT_TEST_URL"]
CHATBOT_TOKEN = os.environ["CHATBOT_TEST_TOKEN"]


async def model_callback(
    input: str,
    turns: list[Turn],
    thread_id: str,
) -> Turn:
    payload = {
        "message": input,
        "session_id": thread_id,
        "history": [
            {"role": turn.role, "content": turn.content}
            for turn in turns
        ],
        "mode": "evaluation",
    }
    headers = {"Authorization": f"Bearer {CHATBOT_TOKEN}"}

    async with httpx.AsyncClient(timeout=30.0) as client:
        response = await client.post(CHATBOT_URL, json=payload, headers=headers)
        response.raise_for_status()
        body = response.json()

    if not isinstance(body.get("message"), str):
        raise ValueError("chatbot response must contain a string message")

    return Turn(role="assistant", content=body["message"])

Decide whether the service expects full history, a session ID, or both. Sending both can duplicate context if the backend already stores prior turns. Confirm the production protocol and write a deterministic integration test for the adapter before spending model calls on simulation.

Use a test-mode tool boundary that prevents irreversible effects. A refund tool can write to an isolated ledger and return a receipt; it should not transfer money. Preserve tool semantics and authorization checks even when effects are sandboxed, or the simulated path will be unrealistically easy.

Run a bounded simulation

The current ConversationSimulator guide documents conversational_goldens for simulate(). Set concurrency and turn limits from provider quotas, application latency, and expected trajectory length. Start synchronously or with very low concurrency while debugging session isolation.

from deepeval.simulator import ConversationSimulator


simulator = ConversationSimulator(
    model_callback=model_callback,
    simulator_model="your-approved-simulator-model",
    async_mode=True,
    max_concurrent=4,
)

test_cases = simulator.simulate(
    conversational_goldens=goldens,
    max_user_simulations=8,
)

assert len(test_cases) == len(goldens)
for case in test_cases:
    assert case.turns, "simulation returned an empty conversation"

This assertion checks only structural completion. It does not prove the expected outcome. Capture per-case status, turn count, callback errors, thread ID, simulator model, application revision, and timestamps. Treat partial generation as an infrastructure or harness failure rather than a low-quality conversation.

At concurrency greater than one, verify thread isolation explicitly. Put a harmless canary value in each test session and assert it never appears in another conversation. Cross-session memory leakage is both a test-validity defect and a potential security defect.

Control when simulations stop

A turn cap prevents unlimited execution, but it is not a semantic success condition. The docs say the default behavior uses the golden's expected outcome. The current stopping-logic page supports a custom stopping_controller that returns proceed() or end(). It also says the previous controller keyword remains a deprecated alias.

Use a custom controller when your application emits a deterministic terminal signal such as a receipt, escalation state, denial code, or closed ticket. The following example intentionally uses an application marker rather than judging free-form prose:

from deepeval.simulator.controller import end, proceed


async def stopping_controller(last_assistant_turn, simulated_user_turns):
    if last_assistant_turn is None:
        return proceed()

    text = last_assistant_turn.content.lower()
    terminal_markers = (
        "refund receipt:",
        "verification required:",
        "case escalated:",
    )
    if any(marker in text for marker in terminal_markers):
        return end(reason="Application returned a terminal workflow marker")

    if simulated_user_turns >= 7:
        return end(reason="Safety turn budget reached")

    return proceed()


simulator = ConversationSimulator(
    model_callback=model_callback,
    stopping_controller=stopping_controller,
    max_concurrent=4,
)

The turn-budget branch duplicates the outer maximum intentionally as defense in depth, but it should be aligned with your configured limit. A controller that ends on vague phrases such as "done" or "I can help" can truncate a failed workflow. Prefer structured application state when available.

For specific trajectories, the official simulation-graph documentation supports SimulationNode graphs, terminal nodes, and visit limits. Use a graph when the synthetic user must follow ordered behavior such as ask, correct, challenge, then escalate. Use a stopping controller for global terminal conditions. Do not use an LLM-driven simulator when a deterministic state machine is the actual requirement.

Evaluate completed conversations

Simulation and evaluation should be separate artifact stages. Save or serialize the completed cases according to your data policy, then apply metrics. This allows you to re-score fixed conversations when calibrating a judge without paying to regenerate every transcript.

The current simulator docs demonstrate passing returned cases to evaluate() with multi-turn metrics. This example uses turn relevancy as one diagnostic dimension; it is not a complete release policy:

from deepeval import evaluate
from deepeval.metrics import TurnRelevancyMetric


results = evaluate(
    test_cases=test_cases,
    metrics=[
        TurnRelevancyMetric(
            threshold=0.75,
            model="your-approved-judge-model",
            include_reason=True,
        )
    ],
)

Pair conversational metrics with deterministic outcome checks. Query the isolated refund ledger for one receipt, verify no refund for an unauthorized caller, assert the corrected order ID reached the tool, and check the escalation code. A fluent dialogue can still perform the wrong state transition; a terse dialogue can satisfy the task.

Choose metrics by failure mode:

Failure	Evidence	Suitable check
Assistant ignores the latest correction	Ordered turns	Turn relevancy or custom conversational rubric
Agent leaks another user's order	Transcript and backend authorization log	Deterministic security assertion plus safety rubric
Refund is promised but not executed	Tool ledger and final turns	Deterministic outcome assertion
Agent loops without progress	Turn count and repeated actions	Step/trajectory invariant or custom metric
Policy explanation is misleading	Policy context and assistant turns	Calibrated conversational G-Eval
Session state resets	Thread ID and backend state	Integration assertion, not an LLM judge

Make synthetic users useful rather than merely diverse

Persona diversity should correspond to product behavior. Vary language, verbosity, domain knowledge, emotional pressure, correction patterns, accessibility needs, and adversarial intent only where the application has a requirement. Avoid demographic stereotypes and protected-attribute proxies that are unrelated to the test.

For each scenario, define coverage dimensions and inspect their distribution. A hundred near-identical polite users do not provide broad coverage. Ten reviewed trajectories spanning authorization, correction, outage, ambiguity, and escalation may provide more actionable evidence.

Synthetic users can discover unexpected paths, but they do not estimate real-world prevalence unless sampled from a representative production model. Never report a simulated failure rate as a customer failure rate. Use production telemetry to weight scenarios separately and protect sensitive data during any sampling process.

Reproducibility, cost, and privacy

Conversation simulation involves at least two systems: the simulator model and the chatbot under test. Evaluation may add a third judge model. Record all three model configurations, the DeepEval version, golden revision, application commit, prompts or hashes, tool fixtures, and attempt number.

Set budgets before running:

• Maximum goldens per pull request.
• Maximum user turns per golden.
• Maximum concurrent conversations.
• Provider timeout and bounded infrastructure retry.
• Maximum tool operations per session.
• Daily or pipeline cost ceiling.
• Artifact retention and redaction policy.

Do not cache across application revisions without including every relevant input in the cache key. A stale transcript can make a changed chatbot appear stable. If you cache simulator turns for judge calibration, label them as frozen fixtures and do not present them as a fresh end-to-end run.

Traces and transcripts can contain names, order IDs, retrieved records, tool arguments, and model reasoning. Prefer synthetic identifiers, isolated tenants, and redacted artifacts. Review what the application and DeepEval integrations upload before enabling hosted reporting.

CI strategy

Pull-request CI should run a small set of stable, high-risk goldens with low concurrency and deterministic backend fixtures. Larger exploratory simulation belongs in scheduled or pre-release jobs because it is slower, costlier, and more stochastic.

Use these release states:

1. Product failure: the conversation completed, evidence is complete, and a requirement failed.
2. Simulation failure: the synthetic user or callback could not produce a valid conversation.
3. Judge failure: a complete conversation exists but evaluation failed or abstained.
4. Infrastructure failure: provider, network, environment, or test tenant prevented execution.
5. Review required: the score is near a calibrated boundary or metrics disagree.

Only the first state is automatically a product defect. The others still fail or warn according to policy, but they need different owners and retry behavior.

Before enabling a blocking gate, run known controls: one conversation that should complete, one unauthorized request that must be refused, one callback exception, one turn-budget exhaustion, and one deliberately irrelevant assistant. Confirm each is classified correctly.

Failure diagnosis

Conversations lose context after the first turn

Verify whether the application expects thread_id, full turns, or both. Confirm every callback invocation maps to the same isolated backend session. Inspect backend session logs rather than trusting transcript continuity.

The simulator reaches the turn limit repeatedly

Check whether the expected outcome is observable and whether the application can reach it in the test tenant. Then inspect for repetitive user turns, vague assistant questions, missing tool state, or a controller that never sees the terminal condition. Raising the limit can increase cost without fixing the trajectory.

Tests pass even though no side effect occurred

The metric may be judging persuasive text instead of state. Add deterministic assertions against the sandboxed tool ledger and include tool outcome evidence in the trace. Do not rely on the phrase "refund completed."

Concurrent conversations leak state

Reduce max_concurrent to one and reproduce. Inspect session keys, global mocks, reused clients, mutable fixtures, and backend tenant isolation. Add unique canaries and fail immediately on cross-thread data.

Generated users behave unrealistically

Tighten the scenario and relevant user description, review the simulator model, and use a simulation graph for required trajectories. Do not compensate with a huge persona prompt that mixes conflicting goals.

Results vary too much between runs

Separate transcript generation from judging. Re-score frozen transcripts to measure judge variance, then regenerate with fixed goldens to measure simulator and application variance. Use repeated trials and report distributions for stochastic metrics.

Common mistakes and limitations

Testing a direct LLM call instead of the chatbot. Route the callback through production-like state, retrieval, tools, and policy.

Using free-form text as the only success evidence. Assert sandboxed side effects and structured terminal states.

Allowing unlimited or default concurrency. Set an intentional limit from quotas and verify session isolation.

Confusing synthetic breadth with production representativeness. Simulation explores scenarios; it does not establish customer prevalence.

Putting unreviewed generated goldens into blocking CI. Approve scenario, outcome, risk, and data classification first.

Ending on a vague phrase. Use structured terminal signals or carefully reviewed stopping logic.

Comparing exact transcripts. Evaluate invariants, outcomes, and calibrated conversational requirements instead.

Ignoring model and artifact privacy. Synthetic inputs can still retrieve or expose real test-tenant data.

Implementation checklist

• Pin DeepEval and record simulator, application, and judge model configurations.
• Define reviewed goldens from concrete risks and expected terminal outcomes.
• Route model_callback through the real stateful application boundary.
• Use isolated tenants and reversible or sandboxed tools.
• Set concurrency, timeouts, turn limits, tool limits, and cost budgets.
• Add a deterministic stopping controller only when the terminal signal is reliable.
• Preserve completed conversations separately from metric results.
• Pair conversational judges with deterministic state and security assertions.
• Classify product, simulation, judge, and infrastructure failures separately.
• Prove known controls before making the job blocking.

Frequently asked questions

What does ConversationSimulator return?

The current official docs say simulate() returns a list of ConversationalTestCase objects. Each contains the generated conversation turns and can be passed to multi-turn metrics through DeepEval's evaluation workflow.

Is ConversationSimulator the same as the Synthesizer?

No. The synthesizer creates goldens, including conversational scenarios and expected outcomes. ConversationSimulator uses conversational goldens to generate the actual user-assistant exchange against your chatbot callback.

Does model_callback need conversation history?

Only input is mandatory according to current docs. Declare turns and thread_id when your application needs history or persistent session state. Forward only what matches the real application protocol.

How does a simulation stop?

It can stop at max_user_simulations, when default expected-outcome logic or a custom stopping_controller ends it, or when a simulation graph reaches a terminal node. Always retain a hard turn limit.

Should CI regenerate conversations on every pull request?

Use a small live set when end-to-end behavior is the requirement. For judge calibration, re-score frozen conversations. Larger exploratory generation is usually better scheduled, with budgets and human review.

Can synthetic users test authorization and security?

They can exercise those paths, but security outcomes need deterministic backend assertions and isolated data. A model judge should not be the sole authority on whether an unauthorized state change occurred.

How many conversations are enough?

There is no universal number. Cover distinct risks and state transitions, then measure marginal discovery and operational cost. Duplicate personas do not compensate for missing authorization, correction, outage, or escalation scenarios.

Should we use a stopping controller or a simulation graph?

Use a stopping controller for global completion or safety conditions. Use a simulation graph when the synthetic user must follow a defined trajectory with branches, terminal nodes, or visit limits. They can complement each other.

Conclusion

A production-grade DeepEval conversation simulation begins with reviewed goldens and ends with verified state, not attractive transcripts. Connect the simulator to the real session boundary, constrain tools and budgets, stop on meaningful conditions, preserve complete evidence, and evaluate both conversation quality and deterministic outcomes. That makes synthetic users an engineering instrument rather than a demo generator.