Deterministic Graders vs LLM Judges vs Human Review

For deterministic graders vs LLM judge decisions, use deterministic code whenever correctness can be computed from observable evidence; use a calibrated LLM judge for bounded semantic criteria that code cannot express; and use human review to define the standard, resolve consequential ambiguity, and audit automated graders. Most production evals need all three in that order. Never ask a judge whether JSON parses, never ask regex to decide whether advice is genuinely helpful, and never spend expert time rechecking every obvious schema violation. Escalate only after cheaper evidence has been retained.

Use the complete LLM testing guide for the full quality system. The sibling guides cover the OpenAI Evals shutdown migration, a production-matched eval harness, and tool-using agent evaluation. Go deeper on semantic scoring in the canonical LLM-as-a-judge guide and on retrieval evidence in the Ragas RAG evaluation guide. Reusable QA instructions are available in /skills, including the Playwright CLI skill for collecting browser-visible evidence before grading.

The decision in one table

Question	Deterministic grader	LLM judge	Human review
Best evidence	Parsed output, executable behavior, state, known reference, policy rule	Input, candidate, reference/context, narrow rubric	Same evidence plus domain context and adjudication guidance
Best tasks	Schema, exact label, numeric tolerance, tests, tool arguments, forbidden action	Relevance, faithfulness, completeness, tone, comparative quality	Ambiguous, novel, subjective, high-risk, or disputed cases
Reproducibility	Reproducible when inputs and code are pinned	Variable; model and rubric must be versioned	Reviewer and process dependent
Unit cost and speed	Usually lowest and fastest	Provider call plus parsing and retries	Usually highest and slowest
Main weakness	Rejects valid variation or misses semantic harm	Can misread rubric, evidence, or boundary; needs calibration	Limited scale, disagreement, fatigue, specialist availability
Release role	Hard invariant or precise metric	Calibrated semantic signal or comparison	Standard-setting, audit, escalation, and high-risk approval

Anthropic's agent-evaluation guidance uses the same three-way classification. It describes code graders as objective, reproducible, inexpensive, and debuggable but potentially brittle; model graders as flexible and scalable for open-ended work but nondeterministic and calibration-dependent; and human graders as high-quality expert judgment that is slow and costly. The useful conclusion is not that one wins. Each method should own the requirements its evidence can actually support.

Convert requirements into atomic claims

Do not start by selecting a grader. Start by decomposing "good answer" into claims that can independently pass, fail, abstain, or error.

Consider a support response that must:

1. Be valid JSON with answer and citations fields.
2. Cite only documents retrieved for this request.
3. State the cancellation deadline exactly as the approved policy provides it.
4. Answer the customer's question without irrelevant procedural detail.
5. Use a calm, non-dismissive tone.
6. Avoid initiating cancellation until identity is verified.

Claims 1, 2, 3, and 6 have objective evidence and belong in code. Claims 4 and 5 are semantic and can use a calibrated judge. A policy owner or support quality reviewer defines examples for relevance and tone, adjudicates uncertain cases, and audits the judge. One "overall quality: 1-5" grader would hide which requirement failed and allow strong tone to compensate for an unauthorized action.

Atomic grading also makes partial credit honest. A response can be relevant but ungrounded, safe but incomplete, or correct but malformed. Store each result separately, then define a transparent release rule.

Deterministic graders: make objective rules executable

A deterministic grader is any pinned computation whose output is fixed for the same evidence. It can be a string comparison, parser, schema validator, SQL query, state check, unit test, static analyzer, citation-set comparison, numeric formula, or tool-call invariant.

Prefer it when the requirement can be stated without asking for opinion:

• Output parses and satisfies a schema.
• Classification belongs to an allowed label set.
• A price or date equals an authoritative reference after normalization.
• Generated code passes required tests without breaking existing tests.
• Retrieved IDs include a required source.
• Every cited ID came from the retrieved set.
• A refund row exists exactly once, with the approved amount.
• A forbidden tool was never called.
• Latency and cost evidence stay within a product-defined budget.

The critical phrase is observable evidence. Do not grade a response string for "refund completed" when the real outcome is a database state change. Conversely, do not require an exact sentence when several truthful phrasings satisfy the user.

Example: a deterministic structured-response grader

This application-owned function returns a detailed result rather than one unexplained Boolean. It uses only Python's standard library and checks format, citation provenance, deadline value, and action safety:

import json
from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class Check:
    name: str
    passed: bool
    evidence: Any


def grade_support_response(
    raw_output: str,
    retrieved_ids: set[str],
    expected_deadline_days: int,
    final_state: dict[str, Any],
) -> list[Check]:
    try:
        body = json.loads(raw_output)
    except json.JSONDecodeError as error:
        return [Check("valid_json", False, {"error": str(error)})]

    checks = [
        Check("answer_is_text", isinstance(body.get("answer"), str), type(body.get("answer")).__name__),
        Check("citations_is_list", isinstance(body.get("citations"), list), body.get("citations")),
        Check("deadline_exact", body.get("deadline_days") == expected_deadline_days, body.get("deadline_days")),
        Check("no_unverified_cancellation", final_state.get("cancelled") is False, final_state),
    ]

    citations = body.get("citations", [])
    if isinstance(citations, list):
        checks.append(
            Check(
                "citations_were_retrieved",
                all(isinstance(item, str) and item in retrieved_ids for item in citations),
                {"cited": citations, "retrieved": sorted(retrieved_ids)},
            )
        )
    return checks

Unit-test malformed JSON, missing fields, wrong types, duplicate or invented citations, values immediately around the deadline, and prohibited state changes. Promptfoo's official Python assertion documentation likewise supports Boolean or numeric results for custom checks, but a repository-owned function should still have direct unit tests independent of the eval runner.

Where deterministic grading goes wrong

Determinism does not guarantee validity. A wrong rule fails consistently. Common mistakes include exact matching unnormalized whitespace, demanding one tool sequence when several are valid, comparing floating values without a business tolerance, reading a self-reported outcome instead of environment state, and writing tests for an unstated file path or hidden precondition.

Anthropic recommends reference solutions that pass all graders and warns against rigid path grading when agents can find valid alternatives. Add positive controls, negative controls, and boundary fixtures. A grader that only sees passing examples has not demonstrated selectivity.

LLM judges: constrain semantics rather than outsourcing judgment

Use a judge when the requirement is semantic, bounded, and supported by evidence the judge can read. Good candidates include whether an answer addresses every part of a question, whether claims are supported by supplied context, whether a summary preserves material qualifications, and which of two responses better satisfies a narrow rubric.

OpenAI's evaluation best-practices guide says models are stronger at discrimination and recommends pairwise comparison, classification, or scoring against explicit criteria rather than open-ended evaluation. Give the judge a small decision space and anchored examples. Do not ask it to invent the standard while applying it.

A judge contract should name:

• One criterion or a small set of independently scored dimensions.
• Exact evidence: user request, candidate, approved context/reference, and observable outcome.
• Labels with operational meanings, including an insufficient-evidence option.
• Positive, negative, and boundary examples approved by domain reviewers.
• A structured response schema and parser behavior.
• Judge model, prompt, settings, and version.
• Escalation rule and calibration dataset.

Example: a bounded rubric in Promptfoo

Promptfoo documents model-graded assertions and a provider override for assertions in its assertion reference. This illustrative test keeps factual invariants separate and asks the judge only about relevance:

providers:
  - openai:gpt-5-mini

tests:
  - description: cancellation answer stays relevant
    vars:
      question: Can I cancel after shipment?
      policy: Cancellation is unavailable after shipment. A return may be requested after delivery.
    assert:
      - type: is-json
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: |
          Judge only relevance to the QUESTION.
          PASS: directly answers cancellation after shipment and may briefly name the return alternative.
          FAIL: omits the cancellation answer or adds unrelated procedures.
          UNKNOWN: the candidate output is missing or unreadable.
          Do not judge policy correctness here; a separate deterministic check owns that rule.

The provider shown is an example configuration, not a recommendation that the tested and judging model should always match. In a real suite, pass the question, policy, and candidate through the runner's documented template context and verify the rendered judge prompt. Keep a frozen set of candidate outputs so judge changes can be tested without regenerating answers.

Pointwise, pairwise, and reference-based choices

Use pointwise labels for an absolute release criterion such as PASS, FAIL, or UNKNOWN. Use pairwise comparison to choose between prompt or model candidates on one criterion. Run both A/B and B/A orderings and inspect inconsistent decisions rather than assuming presentation order is harmless. Use reference-based grading when an approved answer or checklist anchors the judgment, but tell the judge to preserve meaning rather than reward copied wording.

Pairwise preference does not establish that either candidate meets an absolute safety requirement. A less harmful answer can still be unacceptable. Apply hard invariants before preference grading and retain an absolute acceptance check where the release decision requires one.

Human review: make it a process, not a fallback label

Human review serves four distinct jobs:

1. Standard-setting: define rubrics, examples, and what evidence is material.
2. Calibration: label a representative set for automated-grader comparison.
3. Adjudication: resolve reviewer or grader disagreements using documented policy.
4. Audit and discovery: sample normal cases and inspect new failure clusters for blind spots.

Those jobs need different sampling. Calibration requires cases across labels and decision boundaries. Adjudication selects disagreements. Audit should include random ordinary traffic, not only dramatic failures. Discovery over-samples new products, languages, tools, user populations, and changed distributions.

Blind reviewers to model/provider identity when comparing candidates, randomize presentation, and hide the current automated verdict until the independent label is recorded. Train with examples, allow "insufficient evidence," and capture a reason code. For specialized legal, medical, financial, safety, or policy decisions, use appropriately qualified reviewers and governance; a general annotation pool cannot manufacture domain authority.

Human review is not automatically correct. Reviewers disagree, tire, infer unstated rules, and can be influenced by formatting. Use at least two independent labels for the calibration subset when risk justifies it, then adjudicate disagreements. Measure agreement before treating the labels as ground truth.

Measure judge agreement with labels

Raw agreement is easy to understand but does not account for agreement expected from label prevalence. Cohen's kappa is one option for two raters on categorical labels:

observed agreement Po = matching labels / total labels
chance agreement Pe   = sum over labels of P(human=label) * P(judge=label)
kappa                 = (Po - Pe) / (1 - Pe)

This standard-library implementation makes the assumptions visible and rejects mismatched inputs:

from collections import Counter


def categorical_agreement(human: list[str], judge: list[str]) -> dict[str, float]:
    if len(human) != len(judge) or not human:
        raise ValueError("human and judge labels must have the same non-zero length")

    labels = set(human) | set(judge)
    total = len(human)
    observed = sum(left == right for left, right in zip(human, judge, strict=True)) / total
    human_counts = Counter(human)
    judge_counts = Counter(judge)
    expected = sum(
        (human_counts[label] / total) * (judge_counts[label] / total)
        for label in labels
    )
    kappa = (observed - expected) / (1 - expected) if expected < 1 else 1.0
    return {"observed_agreement": observed, "expected_agreement": expected, "kappa": kappa}


human_labels = ["PASS", "FAIL", "FAIL", "PASS", "UNKNOWN", "PASS"]
judge_labels = ["PASS", "FAIL", "PASS", "PASS", "UNKNOWN", "FAIL"]
print(categorical_agreement(human_labels, judge_labels))

Do not copy a generic kappa threshold into a release gate. Review the confusion matrix by label and business consequence. A judge that agrees on abundant PASS examples but misses rare unsafe FAIL cases can have an attractive aggregate while being unfit for the job. Measure false acceptance and false rejection on the decision boundary, with uncertainty, and inspect every high-consequence error.

If humans disagree heavily, fix the rubric or evidence before optimizing the judge. Automating an unstable standard only produces disagreement faster.

A risk-based grader allocation

Allocate each requirement with two questions: Can objective evidence decide it? and What happens if the grader is wrong?

Requirement	Primary grader	Secondary control	Human role
Valid JSON and required fields	Parser/schema code	Negative fixtures	Review schema changes only
Tool called with authorized account ID	Trace/state assertion	Contract test against tool	Investigate disputed authorization policy
Answer supported by supplied sources	Claim/evidence checks plus narrow judge where needed	Citation provenance code	Label subtle entailment cases
Helpful and concise support tone	Calibrated model rubric	Length and prohibited-language code	Define examples and audit drift
Generated patch solves issue	Executable tests	Static analysis and regression suite	Review maintainability or domain nuance
High-impact medical recommendation	Deterministic policy constraints plus specialist review	Calibrated assistive judge, not sole authority	Qualified expert owns acceptance
Prompt A vs B preference	Pairwise judge on one criterion	Swap order and retain ties	Calibrate and adjudicate close cases

"Human in the loop" should not be used to excuse an unsafe automatic path. For a consequential action, deterministic authorization and approval controls belong in the application, not only in post-run evaluation. Human grading measures behavior; it does not replace runtime safeguards.

Compose graders without hiding failures

A robust gate can use hard invariants, semantic requirements, and review escalation:

def release_decision(grades: dict[str, str]) -> str:
    hard_invariants = ["valid_schema", "authorized_action", "citation_provenance"]
    if any(grades.get(name) != "PASS" for name in hard_invariants):
        return "REJECT"

    semantic = [grades.get("relevance"), grades.get("tone")]
    if "FAIL" in semantic:
        return "REJECT"
    if "UNKNOWN" in semantic or None in semantic:
        return "HUMAN_REVIEW"
    return "ACCEPT"

This is an illustrative policy shape, not a universal gate. Its important property is non-compensation: excellent tone cannot offset an unauthorized action. Unknown evidence does not become failure or pass; it routes to review. Store each grade and evidence even when the final decision is obvious.

Weighted averages are appropriate only when dimensions genuinely trade off. Publish weights, direction, normalization, and threshold. Keep critical minimums beside the total. If one judge scores several dimensions, preserve each dimension separately and test whether a compound rubric creates correlated errors.

Control cost without degrading validity

Use a cascade:

1. Run parsers, state checks, executable tests, and policy invariants.
2. Skip semantic grading when prerequisites make the candidate unusable.
3. Use a calibrated judge on the remaining semantic criteria.
4. Escalate judge UNKNOWN, boundary scores, disagreements, novel slices, and high-risk cases.
5. Sample accepted ordinary cases for ongoing human audit.

Cache judge decisions only when the case, candidate output, evidence, rubric, judge model, and settings are all identical. A cache keyed only by candidate text is invalid when the policy context changes. Record cache hits in the grade provenance.

Shorter rubrics are not necessarily cheaper if ambiguity causes retries and review. Optimize after validity: remove irrelevant context, split dimensions, require structured labels, and cap explanations while retaining enough evidence to diagnose decisions.

Failure analysis

Exact-match failures dominate otherwise good outputs

Decide whether exact wording is a real contract. Normalize case, whitespace, Unicode, ordering, or numeric representation only where the business meaning permits it. If multiple answers are valid, switch to set membership, executable behavior, reference facts, or a bounded semantic grader.

The LLM judge passes fluent but incorrect answers

Check whether the judge received authoritative context and whether correctness was combined with style in one rubric. Move known facts and state into deterministic checks. Ask the judge one grounded criterion, include FAIL boundary examples, and measure false acceptance against human labels.

Human reviewers disagree on many cases

Inspect whether the task, evidence, and rubric are ambiguous. Add anchored examples and an insufficient-evidence option, then relabel independently. Do not average conflicting categorical judgments without adjudication when the result controls a consequential gate.

Judge results change after a model update

Treat the judge as a versioned dependency. Replay frozen candidates through old and new judge configurations, compare label confusion by slice, and review changed high-risk decisions. Do not silently resolve a floating alias and continue the old baseline.

The hybrid score passes despite a safety failure

The aggregation permits compensation. Make authorization, privacy, safety, schema, and other non-negotiable requirements hard invariants. Calculate an average only after those pass, and retain the failed dimension in the report.

Review queues grow without bound

Separate grader errors from genuine uncertainty, narrow compound rubrics, improve evidence, and cluster repeated disputes into new deterministic rules or rubric examples. Set service ownership and prioritization by risk. Never auto-accept merely because the queue is old.

Version scope and limitations

This comparison is current to July 14, 2026. It does not rank specific judge models or claim a universal cost, agreement rate, or threshold. Model behavior, provider pricing, and framework syntax change; pin configurations and use the linked primary docs for the version in your repository.

The Promptfoo YAML demonstrates documented assertion concepts but is not a complete application integration. Verify rendered variables and provider configuration locally. The Python graders and release function are application-owned examples, not framework APIs. They need unit tests, error classification, privacy controls, and domain-specific evidence before production use.

Cohen's kappa is one categorical agreement measure, not proof of grader validity. Label prevalence, sample selection, reviewer quality, task risk, and asymmetric error costs matter. For ordinal scores, multiple reviewers, or continuous values, choose an analysis appropriate to that design with statistical review.

Automated evaluation cannot eliminate human accountability for high-impact decisions. It can focus expert attention and make evidence repeatable. Runtime permissions, approval gates, monitoring, incident response, and user recourse remain separate controls.

Frequently Asked Questions

Are deterministic graders always better than an LLM judge?

They are better for requirements computable from objective evidence because they are reproducible and easy to debug. They are not better at nuanced semantic judgments that cannot be reduced to a correct parser, state check, reference, or executable test. Use each method for its evidence.

When should I use an LLM-as-a-judge?

Use one for bounded semantic criteria such as relevance, faithfulness to supplied context, completeness, or comparative preference. Give it explicit labels, narrow evidence, anchored examples, structured output, an insufficient-evidence path, and calibration against independent human labels.

Does human review count as the ground truth?

It can define the reference standard when qualified reviewers follow a clear process, but individual humans are not infallible. Measure independent reviewer agreement, adjudicate consequential disputes, document qualifications, and revise ambiguous tasks before treating labels as calibration truth.

Can one overall quality score replace several graders?

Usually not. It hides failure causes and allows strengths to compensate for critical weaknesses. Keep correctness, grounding, safety, relevance, tone, latency, and cost separate. Create a roll-up only for a specific decision and preserve non-compensable invariants.

Should the judge model be different from the model under test?

Model-family diversity can be a useful experiment, but it does not remove the need for calibration. Select a judge by measured agreement on your rubric and slices, test configuration changes on frozen outputs, and retain human audit. Do not infer validity from provider identity alone.

How large should the human-labeled calibration set be?

There is no universal size. It must cover labels, important slices, edge cases, and the decision boundary with enough evidence to estimate consequential errors. Start from independently reviewed real cases, report uncertainty, and expand where disagreement or risk is concentrated.

How do I handle a judge that returns UNKNOWN?

Retain UNKNOWN as insufficient evidence and route it according to risk: obtain missing context, retry only a genuine transient error, or request human review. Do not coerce it to PASS, and do not count it as product FAIL when the candidate was never validly judged.

Can LLM judges be used in CI?

Yes, after calibration and with pinned rubrics, structured output, error classification, budget controls, retained evidence, and a policy for disagreement or abstention. Keep deterministic invariants first and avoid making a variable semantic score the only release signal.