DeepEval 3 to 4 Migration Guide for Traces and Multi-Turn Goldens

A DeepEval 4 migration is safest when it is treated as a test-harness change, not a routine dependency bump. DeepEval 4 introduced an agent-oriented workflow, richer local trace inspection, and current integration patterns that evaluate traces and spans. At the same time, current conversation testing is built around conversational goldens and generated test cases. A team moving from DeepEval 3 should preserve its old decisions, migrate one evaluation shape at a time, and compare both runners before changing a release gate.

This guide uses the DeepEval 4 testing pillar as the architectural baseline. The companion articles explain how to install the DeepEval coding-agent skill, generate stateful conversations, and diagnose TaskCompletionMetric traces. For cross-framework evaluation architecture, the canonical LLM testing guide covers test cases, graders, RAG, and agents. Reusable QA instructions are available in the QASkills directory, including the Playwright CLI skill for browser-backed end-to-end evidence.

What changed, and what should not change

The first-party DeepEval 4 release notes describe an agent-native evaluation loop, a terminal trace inspector, and native framework integrations. The current component-level evaluation documentation places metrics on trace spans while datasets provide goldens to the evaluation loop. These are documented capabilities. They do not prove that every v3 suite needs a rewrite or that every historical score should move.

Your migration contract should preserve business meaning:

Asset	Preserve during migration	Allowed implementation change
Golden dataset	Stable case IDs, inputs, references, labels, and data lineage	Serialization or loading API
Model invocation	Production prompt, model configuration, tools, retrieval, and normalization	Instrumentation wrapper
Metric	Requirement, judge configuration, threshold, and failure reason	Attachment point on trace or span
Release decision	Which failures block, warn, or require review	Report format and CI job layout
Evidence	Candidate output, trace, score, reason, errors, and version identifiers	Local TUI or hosted visualization

Do not accept a green migration because the new command exits zero. Require the expected number of cases, prove that a negative control fails, and compare decisions on identical outputs. That catches empty discovery, missing metrics, unintended caching, and instrumentation gaps.

Name one migration owner for the harness and separate owners for datasets, application adapters, judge calibration, and CI. A single upgrade ticket otherwise hides cross-team decisions. Record who can approve a changed verdict, who can classify an infrastructure failure, and who can authorize rollback before shadow runs begin.

Prerequisites and version baseline

Create an upgrade branch and record the exact versions used by the existing job. Capture Python, DeepEval, pytest, provider SDK, tracing integration, and model identifiers. If a hosted judge alias can move independently, record the resolved model in each result. The framework version alone is not sufficient provenance.

The following commands are a discovery workflow, not a universal dependency policy:

python --version
python -m pip show deepeval pytest
python -m pip freeze > artifacts/v3-freeze.txt
deepeval --version
deepeval --help > artifacts/v3-cli-help.txt

rg -n "(deepeval|LLMTestCase|ConversationalTestCase|Golden|@observe|assert_test|evaluate()"   tests src .github

Archive one representative successful run and one intentional failure before changing dependencies. Retain case count, metric names, scores, reasons, latency, model configuration, errors, and the commit SHA. If the v3 job does not persist these fields, add a temporary neutral export first. A result you cannot compare is not a baseline.

Use separate environments rather than upgrading in place. A conservative v4 environment pins the major line while your lockfile records the exact resolved package:

python -m venv .venv-deepeval4
source .venv-deepeval4/bin/activate
python -m pip install --upgrade pip
python -m pip install "deepeval>=4,<5" pytest
python -m pip freeze > artifacts/v4-freeze.txt
deepeval --version
deepeval diagnose

The DeepEval CLI reference documents deepeval diagnose for inspecting effective settings and deepeval inspect for opening a saved run in the terminal trace viewer. Confirm the commands against deepeval --help for the exact pinned release because CLI flags can evolve inside a major version.

Build a migration inventory by evaluation shape

Classify tests before editing them. A file name rarely tells you whether it is a single-output assertion, a stateful conversation, a component metric, or a trace-level agent decision.

1. Single-turn end-to-end tests build an LLMTestCase from input and final output. Preserve them first because they provide a simple compatibility signal.
2. RAG tests additionally depend on retrieval context and sometimes expected context. Verify both the retriever evidence and generated answer are still populated.
3. Component tests score a retriever, tool, LLM call, or sub-agent span. Their attachment point matters as much as their metric.
4. Agent tests need a complete top-level trace and meaningful final outcome. A final string without tool and state evidence can conceal an instrumentation regression.
5. Multi-turn tests use ordered turns or conversational goldens. Preserve role order, initial turns, expected outcome, and stopping semantics.
6. Synthetic-data jobs create test assets rather than evaluate releases. Keep them outside blocking CI until generated cases are reviewed and versioned.

Assign each group an owner, representative case, expected case count, current command, target v4 command, and rollback path. Migrate a vertical slice of every shape before bulk conversion. Ten easy single-turn tests do not validate an agent trace architecture.

Stage 1: make single-turn tests boring

Start with tests whose inputs and candidate outputs can be frozen. Run v3 and v4 metrics against the same candidate text so generation nondeterminism does not contaminate scorer comparison. For deterministic assertions, require exact decision parity. For model-judged metrics, inspect disagreements against adjudicated human labels rather than forcing agreement with the older judge.

A neutral comparison record can be JSON Lines with one row per case:

{"case_id":"refund-001","input":"Refund order 42","actual_output":"I can start the refund.","expected_output":"Refund initiated","old_pass":true,"old_score":0.84,"old_reason":"Outcome is aligned"}
{"case_id":"refund-002","input":"Delete another user's order","actual_output":"Order deleted.","expected_output":"Refuse unauthorized request","old_pass":false,"old_score":0.12,"old_reason":"Authorization requirement violated"}

Do not round scores before comparison, but do not define parity as identical floating-point values either. The stable artifact is the release decision and its rationale. Record raw values for analysis, then use an approved tolerance or confusion matrix appropriate to the metric.

Stage 2: migrate to trace-aware evaluation

Current DeepEval documentation distinguishes trace-level and component-level evaluation. A trace represents a complete application run; spans represent internal operations. The component-level guide documents @observe, update_current_trace, span metrics, EvaluationDataset, and evals_iterator(). It also states that component-level evaluation is currently single-turn.

This illustrative v4 migration shape instruments a synchronous agent and keeps the release metric at trace level. Replace the body and judge model with your production adapter and approved configuration:

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.evaluate import AsyncConfig
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe, update_current_trace


dataset = EvaluationDataset(
    goldens=[
        Golden(input="Create a support ticket for order 42"),
        Golden(input="Explain why order 99 cannot be refunded"),
    ]
)


@observe()
def support_agent(query: str) -> str:
    # Invoke the same orchestrator, tools, and normalization used in production.
    answer = run_production_agent(query)
    update_current_trace(input=query, output=answer)
    return answer


metric = TaskCompletionMetric(
    threshold=0.75,
    model="your-approved-judge-model",
    include_reason=True,
)

for golden in dataset.evals_iterator(
    metrics=[metric],
    async_config=AsyncConfig(run_async=False),
):
    support_agent(golden.input)

The synchronous mode is useful during migration because failures are easier to associate with a case and provider rate limits are predictable. It is not automatically the best production configuration. After parity is established, evaluate the documented asynchronous pattern and choose concurrency from service quotas and trace isolation behavior.

Trace migration fails silently when the wrapper observes a helper instead of the top-level task. Verify that one golden produces one intended trace, that the trace has an input and final output, and that tool/retriever spans are children of the correct run. Use an intentionally failing query to prove the metric is attached to the trace that contains the bad outcome.

Stage 3: migrate component metrics deliberately

A component metric should observe the component's contract, not whatever text happens to be easy to capture. For a retriever, retain query, retrieved chunks, source identifiers, and filters. For a tool, retain tool identity, validated arguments, output, and error state. For an LLM span, retain its actual prompt context and output after any application-level normalization.

The current v4 docs attach a metric to @observe(metrics=[...]) and populate the span test case through update_current_span(test_case=...). Do not attach the same metric to every nested span. That can multiply costs and produce misleading failures against operations the rubric was not written to judge.

During review, answer four questions for each metric:

• What requirement does this metric represent?
• Which trace or span contains all evidence needed to judge it?
• Which missing fields make the score invalid rather than low?
• Is the metric blocking, advisory, or diagnostic?

Treat missing evidence as an instrumentation failure. A zero score and a missing score have different owners and remediation paths.

Stage 4: remodel multi-turn goldens

Current ConversationSimulator documentation defines a ConversationalGolden with scenario, expected outcome, and user description. The simulator calls the application through model_callback, returns ConversationalTestCase objects, and stops on a turn limit, expected-outcome logic, a stopping controller, or a terminal simulation-graph node.

Do not mechanically flatten a historical conversation into a single prompt. Preserve the stateful requirement as a golden and preserve any fixed opening turns. This current API example is intentionally small:

from deepeval.dataset import ConversationalGolden
from deepeval.simulator import ConversationSimulator
from deepeval.test_case import Turn


goldens = [
    ConversationalGolden(
        scenario="A verified customer changes the delivery address before dispatch",
        expected_outcome="The new address is validated and the order is updated",
        user_description="A concise customer who can provide the order number",
        turns=[Turn(role="assistant", content="How can I help with your order?")],
    )
]


async def model_callback(input: str, turns: list[Turn], thread_id: str) -> Turn:
    response = await call_production_chatbot(
        message=input,
        history=turns,
        session_id=thread_id,
    )
    return Turn(role="assistant", content=response)


simulator = ConversationSimulator(model_callback=model_callback, max_concurrent=4)
cases = simulator.simulate(conversational_goldens=goldens, max_user_simulations=8)

The type annotation uses modern Python syntax and is illustrative; adapt it to your supported Python version. More importantly, route thread_id to the real session boundary. If every turn creates a new application session, a realistic transcript can still be testing the wrong system.

Compare migrated conversations on invariants rather than exact wording: role order, scenario coverage, terminal state, required tool effects, authorization behavior, maximum turns, and expected outcome. Synthetic users are stochastic. Exact transcript equality encourages brittle tests and does not prove task success.

Stage 5: run shadow CI before changing the gate

The migration job should run beside the v3 gate for a fixed observation window. Keep the old gate authoritative while the v4 job uploads artifacts and reports differences. Do not let both jobs independently block the same pull request until their failure ownership is clear.

This GitHub Actions fragment is an illustrative shadow layout. Lockfiles, secrets, provider access, and artifact names must match your repository:

jobs:
  deepeval-v4-shadow:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
      - run: python -m pip install -r requirements-evals-v4.lock
      - run: deepeval diagnose
      - run: deepeval test run tests/evals
        env:
          DEEPEVAL_DISABLE_DOTENV: "1"
          JUDGE_API_KEY: ${{ secrets.JUDGE_API_KEY }}
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: deepeval-v4-results
          path: artifacts/evals/**
          if-no-files-found: error

Use least-privilege secrets and sanitize traces before uploading. LLM inputs, retrieved documents, and tool arguments may contain customer or security-sensitive data. Artifact retention and access should follow the same classification as production telemetry.

Promote the v4 job only when it discovers the expected suite, its negative controls fail, its infrastructure-error rate is understood, and human review supports its changed verdicts. Keep the v3 lockfile and command available for a bounded rollback period.

Failure diagnosis matrix

Symptom	Probable layer	Investigation	Correct response
Zero tests or implausibly fast success	Discovery or command	Check paths, markers, collected cases, and CLI output	Fail the job on an unexpected case count
Scores exist but traces have no final output	Instrumentation	Inspect top-level observed function and update call	Fix trace boundaries before tuning metrics
One input creates multiple unrelated traces	Async or session isolation	Correlate golden ID, task, and trace ID	Repair context propagation and reduce concurrency
v4 scores differ only on model-judged metrics	Judge or rubric	Re-score frozen outputs and compare with human labels	Calibrate; do not force old-score equality
Conversation repeats or resets each turn	Application callback	Verify thread ID and history forwarding	Route simulation through the real session path
CI is flaky but local runs pass	Quota, secrets, network, or dotenv	Classify provider errors separately from metric failures	Add bounded infrastructure retry and explicit settings
New suite passes a known-bad candidate	Missing metric or cache	Disable suspect cache and inspect metric attachment	Block migration until the negative control fails

Common migration mistakes

Changing framework, judge, prompt, and threshold together. This destroys causal attribution. Hold model behavior and rubric stable while moving the harness, then make quality changes in separately reviewed commits.

Calling every score delta a regression. Model-judged metrics can vary. Compare release decisions, reason quality, and human agreement. Preserve distributions and disagreement examples rather than reporting only averages.

Treating traces as optional decoration. Trace-based agent metrics need a complete run. If tool effects or final outcome are absent, the judge may infer from incomplete evidence and produce a plausible but invalid reason.

Generating a new synthetic dataset during cutover. That changes both the harness and sample population. Migrate against frozen, reviewed goldens first. Generate and approve new cases afterward.

Leaving deprecated API usage because it still works. A compatibility alias is a warning window, not a target architecture. For example, current simulator docs call the stop callback stopping_controller and identify controller as deprecated. Update deliberately and make warnings visible in CI.

Using hidden local settings in release tests. The DeepEval CLI can load dotenv configuration. Current docs describe DEEPEVAL_DISABLE_DOTENV=1 for CI isolation. Ensure the CI job receives every required setting explicitly and prints a sanitized diagnostic summary.

Release checklist

• Pin exact v3 and v4 environments and archive dependency inventories.
• Freeze representative goldens and candidate outputs for scorer parity.
• Record expected test counts and add at least one known-bad control.
• Validate top-level traces before attaching component metrics.
• Preserve conversation state, role order, expected outcomes, and turn limits.
• Classify product, metric, instrumentation, and infrastructure failures separately.
• Run v4 in shadow mode and review every changed blocking verdict.
• Document artifact privacy, access, and retention.
• Keep an executable rollback job for the agreed stabilization window.
• Remove deprecated APIs only after equivalent evidence is proven.

Frequently asked questions

Is DeepEval 4 a drop-in upgrade from DeepEval 3?

Do not assume so for a production suite. Simple imports may continue to work, but trace attachment, integrations, conversation simulation, CLI behavior, and result artifacts deserve explicit validation. Build two pinned environments and compare representative tests before changing the gate.

Should historical scores be recalculated after migration?

Preserve historical v3 results as immutable evidence. Re-run a selected frozen corpus in v4 and label it with the new framework and judge configuration. Do not overwrite old records because that erases the distinction between original and recalculated results.

How long should shadow CI run?

There is no universal duration. Choose enough runs to cover routine changes, known edge cases, provider failures, and at least one release decision. A low-frequency suite may need a calendar window; a high-volume suite may use an agreed number of representative runs.

Can we change thresholds while migrating?

Technically yes, but it is poor experimental design. First prove the new harness represents the old requirement. Then recalibrate thresholds against human-labeled examples in a separate change with an explicit risk owner.

Why use frozen outputs before live agent runs?

Frozen outputs isolate metric behavior from model sampling, retrieval drift, tool state, and network changes. Once scorer parity is understood, live parallel runs can reveal generator and instrumentation differences without conflating every layer.

What should block promotion of the v4 job?

Unexpected case counts, missing traces, inactive negative controls, unexplained deterministic disagreements, uncontrolled secrets, unclassified infrastructure errors, or changed blocking verdicts without human review should all stop promotion.

Do multi-turn goldens replace conversational test cases?

They serve different stages in the current simulator workflow. A ConversationalGolden defines the scenario and expected outcome; ConversationSimulator.simulate() produces ConversationalTestCase instances containing turns that can be evaluated with conversational metrics.

Is the terminal inspector required in CI?

No. The local deepeval inspect workflow is diagnostic, not the release contract. CI still needs durable machine-readable evidence and a clear exit decision. Use the inspector to explore saved runs without making manual terminal interaction a build dependency.

Conclusion

A reliable DeepEval 3-to-4 migration preserves decisions before it adopts new capabilities. Freeze the evidence, pin both environments, move single-turn tests first, validate trace boundaries, remodel conversations as goldens, and shadow the new runner. Only then should DeepEval 4 own the release gate. That sequence lets teams benefit from agent-native traces and current simulation APIs without confusing a tooling upgrade with an improvement in model quality.