DeepEval TaskCompletionMetric: Trace Setup and Failure Analysis

The DeepEval Task Completion Metric asks whether an agent accomplished its task by analyzing the run's trace and comparing the inferred or supplied task with the observed outcome. It is not a string-similarity assertion over the final answer. A valid implementation must expose a coherent top-level trace, meaningful outcome evidence, stable judge configuration, and separate checks for tools, policy, and infrastructure. Otherwise a plausible score can conceal an incomplete run.

Use the DeepEval 4 testing guide for the surrounding framework. Related workflows cover the DeepEval 3-to-4 migration, coding-agent skill installation, and ConversationSimulator testing. The existing AI agent evaluation guide adds cross-framework strategy. Browse reusable assets in QASkills, including the Playwright CLI skill when a task completes through a real browser.

What TaskCompletionMetric measures

The official Task Completion documentation classifies the metric as LLM-as-a-judge, referenceless, agentic, and trace-based. It states that task and outcome are extracted from the trace unless a task is supplied, and that the score represents alignment between them. Current documented options include threshold, task, judge model, reason output, strict mode, async mode, and verbose mode.

This supports a precise interpretation:

• The metric judges outcome alignment, not exact wording.
• It requires enough trace evidence to identify the requested task and resulting outcome.
• It can infer the task, but an explicit task may reduce ambiguity when a golden represents a fixed requirement.
• It is model-judged, so calibration and repeated-run analysis matter.
• It is referenceless, so it does not prove factual correctness against an expected answer by itself.
• It should be complemented by process, tool, security, and deterministic assertions.

Question	TaskCompletionMetric answers it?	Better companion evidence
Did the agent achieve the requested outcome?	Yes, from trace evidence	Deterministic state assertion when available
Did it call the correct tool with exact arguments?	Not specifically	Tool or argument correctness metric plus schema checks
Did it follow an approved plan?	Not directly	Plan adherence or deterministic policy assertions
Is the final factual answer correct?	Only insofar as outcome evidence shows it	Reference-based or domain-specific metric
Did the run violate authorization?	Not safely as the sole check	Backend authorization assertion and security tests
Was the provider unavailable?	No; that is not product quality	Infrastructure error classification

An agent can complete a task through an inefficient path, or follow a plan while failing the outcome. Keep task completion distinct from process quality.

Prerequisites and version baseline

This tutorial follows current DeepEval 4 docs reviewed on July 14, 2026. Pin the exact package and judge model in your environment. Capture the agent application revision, prompts, tool schemas, retrieval configuration, golden revision, metric options, and attempt number in result artifacts.

Prepare the environment and verify effective configuration:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install "deepeval>=4,<5" pytest
python -m pip freeze > requirements-evals.lock

deepeval --version
deepeval diagnose

Use a test tenant and reversible tools. A task-completion run may invoke search, databases, email, tickets, browser actions, or payments. Replace destructive effects with contract-faithful sandbox implementations and retain receipts so success can be checked independently.

Before adding a model judge, write deterministic smoke tests for tracing boundaries and tool adapters. If the top-level trace cannot reliably capture input and output, no threshold will repair the evidence.

Design the trace before choosing the threshold

A useful trace is a causal record of one task. It should have:

1. A top-level agent span or trace representing exactly one golden invocation.
2. The user input or explicit task at the root.
3. Child spans for significant LLM, retriever, tool, and sub-agent operations.
4. Validated tool inputs, outputs, and error states where policy permits.
5. A final application outcome at the root, not merely the last model token.
6. Correlation identifiers for the golden, test tenant, and application revision.
7. Clear status for completed, refused, escalated, timed out, or failed execution.

Do not pack many unrelated batch tasks into one observed function. The judge then has to infer which outcome belongs to which request. Conversely, do not attach the top-level metric only to a tiny helper span that cannot see the result.

The current component-level guide explains that trace-level metrics evaluate the run while component metrics attach to individual spans. Use that distinction when building a diagnostic stack.

Implement a minimal trace-level evaluation

The official metric docs show EvaluationDataset, Golden, evals_iterator(metrics=[...]), and an observed agent. This illustrative implementation makes the root input and output explicit and runs synchronously for easier diagnosis:

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.evaluate import AsyncConfig
from deepeval.metrics import TaskCompletionMetric
from deepeval.tracing import observe, update_current_trace

from app.support_agent import run_support_agent


dataset = EvaluationDataset(
    goldens=[
        Golden(input="Create one support ticket for damaged order 42"),
        Golden(input="Refuse an address change for an unverified caller"),
    ]
)


@observe()
def evaluated_agent(query: str) -> str:
    result = run_support_agent(query=query, tenant="eval-sandbox")
    outcome = result.summary
    update_current_trace(
        input=query,
        output=outcome,
    )
    return outcome


completion = TaskCompletionMetric(
    threshold=0.75,
    model="your-approved-judge-model",
    include_reason=True,
    strict_mode=False,
    verbose_mode=False,
)

for golden in dataset.evals_iterator(
    metrics=[completion],
    async_config=AsyncConfig(run_async=False),
):
    evaluated_agent(golden.input)

The tenant argument and result shape are application-specific. Replace them with your own production adapter. Do not return a hand-written success string before checking tool results. The root output should summarize observed state such as ticket ID, refusal status, or escalation result.

Run one case at a time until trace boundaries are correct. Then compare the documented asynchronous loop against your quotas and context propagation. Concurrency is a performance choice, not a quality feature.

Supply an explicit task when inference is ambiguous

The metric can infer a task from the trace. Explicit task is useful when the user input contains context but the requirement is narrower, when several phrasings map to one acceptance criterion, or when the test is an authorization refusal whose successful outcome is intentionally non-completion of the user's requested action.

For example:

authorization_metric = TaskCompletionMetric(
    task=(
        "Protect the order from unauthorized changes and direct the caller "
        "to the approved identity-verification process."
    ),
    threshold=0.8,
    model="your-approved-judge-model",
    include_reason=True,
)

This reframes a refusal correctly: the agent completes the policy task by not performing the caller's prohibited action. Still add a deterministic assertion that the order record did not change. An LLM judge reason is not an authorization audit.

Do not write explicit tasks that encode the desired score, such as "judge this successful." State the observable objective and constraints. Keep the task in version control and review changes like test requirements.

Make outcome evidence concrete

Task completion is strongest when the trace includes verifiable outcomes. Build a normalized outcome object in the application layer, then present a concise representation to the trace while retaining restricted raw receipts separately.

This illustrative adapter pattern validates tool state before reporting completion:

from dataclasses import dataclass


@dataclass(frozen=True)
class TicketOutcome:
    status: str
    ticket_id: str | None
    created_count: int
    authorization: str


def normalize_ticket_outcome(tool_result: dict) -> TicketOutcome:
    status = str(tool_result.get("status", "unknown"))
    ticket_id = tool_result.get("ticket_id")
    created_count = int(tool_result.get("created_count", 0))
    authorization = str(tool_result.get("authorization", "unknown"))

    if status == "created" and (not ticket_id or created_count != 1):
        raise ValueError("created ticket must have one stable receipt")

    return TicketOutcome(
        status=status,
        ticket_id=ticket_id,
        created_count=created_count,
        authorization=authorization,
    )

The deterministic validation runs before the judge. A malformed tool response becomes a product or integration error, not a vague low completion score. Avoid putting secrets or full customer records in the trace; use synthetic IDs and redacted receipts.

For browser agents, capture the final URL, visible confirmation reference, network response, and backend state when possible. A screenshot alone may show a stale or spoofed message. The Playwright CLI skill can help collect browser evidence, but backend verification remains application-specific.

Build a diagnostic metric stack

TaskCompletionMetric tells you whether the outcome aligns with the task. Add orthogonal checks so a failure can be localized:

Layer	Example check	Failure owner
Input contract	Schema, required identity, allowed tenant	API or fixture owner
Retrieval	Context relevance, source constraints, empty result	Retrieval team
Tool selection	Correct tool and allowed order	Agent orchestration team
Tool arguments	Exact IDs, types, authorization scope	Tool contract owner
Process	Plan adherence or step efficiency	Agent policy owner
Outcome	TaskCompletionMetric plus state receipt	Product workflow owner
Communication	Relevance, clarity, policy explanation	UX or content owner
Infrastructure	Provider timeout, trace export, rate limit	Platform team

Avoid attaching all model-judged metrics to every pull request. Choose a small blocking set from high-risk requirements and run broader diagnostics on schedule. More metrics can increase cost and disagreement without increasing confidence.

When TaskCompletionMetric fails but deterministic outcome assertions pass, inspect whether the root output omits the evidence, the explicit task is poorly framed, or the judge rubric interprets domain language differently. When the metric passes but state assertions fail, block on the deterministic defect and treat the judge pass as a calibration counterexample.

Calibrate the judge and threshold

The documented default threshold is not automatically your acceptance threshold. Assemble adjudicated traces representing clear completion, clear failure, partial completion, safe refusal, escalation, and infrastructure interruption. Have domain reviewers label the intended decision before looking at metric scores.

For each candidate threshold, calculate false accepts and false rejects. Weight false accepts more heavily for irreversible or security-sensitive tasks. Keep near-boundary cases for manual review rather than pretending the score is precise.

Use repeated scoring on the same frozen traces to estimate judge variance. If verdicts cross the threshold frequently, change the rubric, model, evidence, or review band. Do not hide instability with unbounded retries or pick the best attempt.

The following application-owned helper illustrates decision classification; it is not a DeepEval API:

from dataclasses import dataclass


@dataclass(frozen=True)
class CompletionDecision:
    score: float
    lower_review_bound: float
    pass_threshold: float

    def label(self) -> str:
        if self.score >= self.pass_threshold:
            return "pass"
        if self.score >= self.lower_review_bound:
            return "review"
        return "fail"


assert CompletionDecision(0.84, 0.70, 0.80).label() == "pass"
assert CompletionDecision(0.74, 0.70, 0.80).label() == "review"
assert CompletionDecision(0.42, 0.70, 0.80).label() == "fail"

Store the raw metric result and reason even if your release policy adds a review band. Never rewrite a framework score to look more certain.

Diagnose failures from the trace outward

Start with execution validity before reading the judge reason:

1. Did exactly one intended trace exist for the golden?
2. Did the trace contain the original task and a final outcome?
3. Did all required tools or sub-agents finish?
4. Were errors, refusals, retries, and timeouts recorded honestly?
5. Did deterministic state match the trace summary?
6. Did the metric run with the expected judge and options?
7. Does the reason cite evidence actually present in the trace?

Only after these checks should you decide whether the application, metric, or test data is wrong.

Symptom	Likely cause	Evidence to inspect	Action
Low score with an empty or generic reason	Missing trace outcome or judge failure	Root input/output and metric error	Repair evidence; do not tune threshold
Pass despite a failed tool	Root output claims success without receipt	Tool span and normalized outcome	Make state validation authoritative
Safe refusal scores as failure	User request was inferred as the task	Golden requirement and explicit task	Define the policy task and keep no-change assertion
Score changes across identical traces	Judge variance or moving alias	Model resolution, config, repeated trials	Pin model and add review band
One golden contaminates another	Async context or shared fixture	Trace IDs, tenant IDs, mutable globals	Fix isolation before parallel runs
Metric never appears	Wrong attachment point or inactive loop	Observed root and evals iterator	Add negative control and trace smoke test
Every case fails after deployment	Provider, secret, or schema change	Infrastructure logs and trace completeness	Classify outage separately from product failure

Use the local trace inspector described in current DeepEval release and CLI documentation to navigate spans, but retain machine-readable artifacts for CI. Manual TUI diagnosis should not be required to understand whether a job passed.

CI and release gates

Create tiers:

• Pull request: a small, reviewed set of deterministic state checks and high-signal completion cases.
• Nightly: broader goldens, repeated judge trials, multiple models or prompts, and non-blocking diagnostics.
• Pre-release: production-like tools in isolated tenants, browser workflows, rollback cases, and human review of disagreements.
• Production monitoring: sampled traces under privacy controls, compared with incident and user-feedback labels.

Each result needs case ID, application commit, dataset revision, DeepEval version, judge model/config, metric options, trace ID, score, reason, attempt, latency, cost evidence, and failure class. Upload artifacts even when the test fails, with appropriate redaction and access.

Do not retry product failures automatically. Retry only classified transient infrastructure failures with a small bounded policy. A rerun that produces a passing judge score does not erase the initial disagreement; retain both attempts.

Before enabling a gate, include controls:

1. A clearly completed task with a valid state receipt.
2. A clearly failed task with no side effect.
3. A safe refusal where non-action is correct.
4. A partial completion missing one required step.
5. A tool exception and provider timeout classified as execution failures.
6. A trace with missing root output that must invalidate evaluation.

If these do not land in their expected classes, the gate is not ready.

Common mistakes and limitations

Scoring only the final message. TaskCompletionMetric is trace-based. Preserve tool and outcome evidence, not just prose.

Using completion as a proxy for safety. An agent can complete a prohibited action. Authorization and policy need separate deterministic checks.

Leaving the task ambiguous. Inference is convenient, but safe refusal and multi-objective cases may need an explicit reviewed task.

Accepting the default threshold without calibration. Threshold consequences belong to your product risk model and labeled traces.

Attaching the metric to a helper span. The judged span must represent the complete task and outcome.

Treating missing evidence as a low score. Incomplete traces are invalid evaluations and should have a distinct failure class.

Rerunning until green. Retain every attempt and distinguish judge variance from product changes.

Ignoring cost and privacy. Full traces can be sensitive, and model judging adds calls. Redact, budget, and restrict access.

Implementation checklist

• Pin DeepEval, judge model, prompts, tools, and dataset revisions.
• Define one top-level trace per golden and populate root input and outcome.
• Validate sandboxed tool state before reporting success.
• Use explicit tasks for policy refusals or ambiguous requests.
• Add tool, argument, process, security, and deterministic outcome checks as needed.
• Calibrate threshold and review band on adjudicated frozen traces.
• Estimate judge variance through repeated scoring of identical evidence.
• Separate product, judge, instrumentation, and infrastructure failures.
• Store redacted trace and metric artifacts for every attempt.
• Prove positive, negative, refusal, partial, exception, and missing-trace controls.

Frequently asked questions

Does TaskCompletionMetric require an expected output?

No. The official docs classify it as referenceless. It judges alignment between task and outcome extracted from the trace or from an explicitly supplied task. Add reference-based and deterministic checks when correctness requires them.

Why does the metric require tracing?

Agent completion depends on the full run, including tools, state, and final outcome. A plain final string can claim success without demonstrating it. The trace provides the evidence the judge uses to infer task and outcome.

Should I set the task explicitly?

Set it when the user input is ambiguous, several inputs share one requirement, or safe refusal is the correct completion. Keep the task objective and observable; do not encode a desired score.

What threshold should we use?

Choose it from human-labeled traces and the cost of false accepts and false rejects. The documented default is a library behavior, not an endorsement for your release policy.

Is strict mode better for CI?

Not automatically. Current docs say strict mode makes the score binary and sets the threshold to one. Use it only when your calibrated requirement truly demands perfection and the judge is stable enough for that consequence.

Can TaskCompletionMetric replace tool correctness metrics?

No. Completion judges outcome alignment. Tool and argument checks diagnose whether the right operations ran correctly, and deterministic assertions should validate exact schemas, authorization, and state changes.

How should safe refusal be scored?

Define the protected objective explicitly, such as preventing an unauthorized change and directing the caller to verification. Then assert deterministically that no prohibited side effect occurred.

What if the metric passes but the backend state is wrong?

Block on the backend assertion. Preserve the trace as a judge-calibration counterexample, improve outcome evidence or rubric, and do not weaken the deterministic source of truth.

Conclusion

TaskCompletionMetric is useful when its trace represents a real task and its output represents verified state. Instrument one coherent run, make outcomes concrete, calibrate the judge, and surround the score with orthogonal tool, security, and deterministic checks. The result is a diagnostic release signal that explains agent outcomes rather than a number attached to persuasive text.