Tool Schema Contract Testing Guide

An agent calls create_ticket with priority: "urgent", the backend expects "high", and the workflow fails after the model did exactly what the schema seemed to allow. Tool schemas are API contracts for AI systems. They deserve the same compatibility discipline as REST payloads, protobuf messages, and event schemas.

Function and tool calling moves part of application control into structured model output. The schema tells the model what arguments are legal, your runtime validates or trusts those arguments, and downstream code performs the action. A small schema change can break prompts, evals, tool routers, audit logs, and saved conversation replays. Contract testing is how you catch that before a tool change reaches production agents.

This guide targets function and tool schema compatibility: JSON Schema validation, breaking-change review, strictness, versioning, and regression fixtures. For measuring whether the model selects the right tool, read the tool calling accuracy testing guide. For broader agent workflow regressions, the agent tool use regression guide covers multi-step behavior.

Treat tool schemas as public interfaces

A tool schema has consumers even when no human imports it as a package. The model consumes names and descriptions. The runtime consumes JSON Schema. The tool implementation consumes validated arguments. Observability pipelines consume call records. Evaluation suites consume expected tool traces. That is a real interface.

The contract includes more than field types. Names, descriptions, required fields, enum values, default semantics, nullability, and additional-property behavior all influence model behavior and runtime safety. Changing a field description can alter model argument selection. Changing a required field can break saved eval prompts. Widening an enum can be safe for the schema but unsafe for a downstream switch statement.

Schema element	Consumer impact	Breaking-change risk
Tool name	Model routing and tool dispatcher	High if renamed without alias
Parameter name	Model argument generation and implementation code	High if renamed
Required list	Whether old calls remain valid	High when adding required fields
Enum values	Model choice set and backend branching	Medium to high
Description text	Model interpretation	Medium, depends on prompt sensitivity
Additional properties	Unknown argument tolerance	Medium, affects validation strategy

The mistake is thinking "the model will adapt." It may adapt in a demo. In production, prompts, examples, fine-tuning data, traces, and evals encode the old contract. The safer assumption is that schemas need versioning and tests.

Validate examples against the same schema the model sees

Every tool should have positive and negative examples. Positive examples are valid calls the runtime must accept. Negative examples are invalid calls the validator must reject. These examples catch accidental schema changes and prevent documentation drift.

Use the same JSON Schema object passed to the model. Do not maintain a separate TypeScript type and hope it stays aligned unless you generate one from the other. In Node, Ajv is a practical validator for contract tests.

import Ajv from 'ajv';
import { describe, expect, test } from 'vitest';

const createTicketTool = {
  type: 'function',
  name: 'create_ticket',
  description: 'Create a support ticket for a customer issue.',
  parameters: {
    type: 'object',
    additionalProperties: false,
    required: ['customerId', 'summary', 'priority'],
    properties: {
      customerId: { type: 'string', minLength: 1 },
      summary: { type: 'string', minLength: 5, maxLength: 160 },
      priority: { type: 'string', enum: ['low', 'normal', 'high'] },
      orderId: { type: 'string', minLength: 1 },
    },
  },
} as const;

const ajv = new Ajv({ allErrors: true, strict: true });
const validate = ajv.compile(createTicketTool.parameters);

describe('create_ticket tool schema contract', () => {
  test('accepts the documented high-priority order issue call', () => {
    const args = {
      customerId: 'cust_123',
      summary: 'Refund failed after payment capture',
      priority: 'high',
      orderId: 'ord_456',
    };

    expect(validate(args)).toBe(true);
  });

  test('rejects unsupported priority labels generated by older prompts', () => {
    const args = {
      customerId: 'cust_123',
      summary: 'Refund failed after payment capture',
      priority: 'urgent',
    };

    expect(validate(args)).toBe(false);
    expect(validate.errors?.[0]?.instancePath).toContain('/priority');
  });
});

This test does two useful things. It protects a valid documented call, and it defines that urgent is not accepted. If the product later wants urgent, the enum change should be reviewed with the tool implementation and routing prompts.

Define what counts as a breaking schema change

Tool schema compatibility is not identical to generic JSON Schema compatibility because model behavior matters. Adding an optional property is usually backward compatible for validation, but it can change model argument choices. Tightening a description can be good, but it can invalidate eval traces if the model no longer selects the same tool. Adding a required field is nearly always breaking for existing callers.

Create a local breaking-change rubric. Review every schema pull request against it. Automated checks can catch obvious structural breaks, while reviewers handle semantic changes in descriptions and examples.

Change	Compatibility assessment	Required review
Rename tool	Breaking	Add alias or versioned tool name
Add required parameter	Breaking	New tool version or runtime default
Remove optional parameter	Breaking if traces or clients send it	Migration and replay check
Add optional parameter	Usually compatible	Eval impact review
Narrow enum	Breaking	Consumer and implementation migration
Widen enum	Validation-compatible, behavior-risky	Backend switch and eval update
Clarify description	Potentially behavior-changing	Prompt and selection eval check

For agents, "contract compatible" means old valid calls still validate, new calls are handled safely, and model selection behavior remains acceptable. That is a higher bar than "Ajv says the schema compiles."

Run replay fixtures through schema validation

Saved tool-call traces are valuable regression fixtures. They show what the model actually emitted in prior releases. Before changing a schema, replay argument objects from successful production or staging traces through the new validator. Any rejected old call is a compatibility break unless you intentionally migrated the tool.

Store a curated fixture set rather than every trace. Include edge cases: missing optional fields, each enum value, maximum-length text, minimal valid request, and calls produced by important prompts. Redact sensitive values before committing fixtures.

You can keep fixture validation small and deterministic. Load JSON fixtures, compile the current schema, and assert every historical valid call still passes. If a fixture should no longer pass, move it into a named breaking-change test with a migration note.

Strict schemas and runtime validation

Strict schemas are safer for tools that perform real actions. additionalProperties: false prevents the model from smuggling ignored fields that humans might think were honored. Required fields make missing critical data fail early. Enum values keep model output inside implementation-supported branches.

Strictness has a cost. If the schema is too narrow, the model may fail to call the tool or produce invalid arguments for reasonable user requests. The answer is not to make everything free-form. The answer is to design smaller tools with clearer contracts and to add validation repair only where it is safe.

Runtime validation should happen even if the model provider supports schema adherence. Treat provider-side schema support as the first gate, not the only gate. Validate before executing side effects. Return a safe tool error when arguments fail. Record validation failures for eval triage.

Testing an OpenAI function tool request

When using OpenAI function tools, pass the function schema as part of the request and keep it sourced from the same module your contract tests validate. The example below uses the Responses API shape for a function tool and then handles the returned function call arguments through the same Ajv validator before execution.

import OpenAI from 'openai';
import Ajv from 'ajv';

const client = new OpenAI();
const ajv = new Ajv({ allErrors: true, strict: true });
const validateCreateTicket = ajv.compile(createTicketTool.parameters);

async function routeSupportMessage(message: string) {
  const response = await client.responses.create({
    model: process.env.OPENAI_MODEL ?? 'gpt-4.1-mini',
    input: message,
    tools: [createTicketTool],
  });

  for (const item of response.output) {
    if (item.type !== 'function_call' || item.name !== 'create_ticket') {
      continue;
    }

    const args = JSON.parse(item.arguments);
    if (!validateCreateTicket(args)) {
      return {
        ok: false,
        reason: 'tool_arguments_failed_schema',
        errors: validateCreateTicket.errors,
      };
    }

    return {
      ok: true,
      ticketDraft: args,
    };
  }

  return { ok: false, reason: 'no_ticket_tool_call' };
}

The schema contract test and runtime validator share the same createTicketTool.parameters. That avoids a common drift: the model sees one schema, the runtime validates a second schema, and the implementation assumes a third shape.

Versioning tools without confusing the model

Tool versioning is a design choice. You can version names, such as create_ticket_v2. You can keep the name and add backward-compatible optional fields. You can use a dispatcher that accepts both old and new shapes. The right choice depends on whether old calls must keep working and whether the model can reliably choose between versions.

For breaking changes, a new tool name is often clearer than a silent schema replacement. Keep the old tool available during migration, mark it as deprecated in the description, and update prompts and evals to prefer the new tool. Once traces show the old tool is no longer called, remove it with a release note.

Avoid presenting the model with too many near-identical tool versions indefinitely. That creates selection ambiguity. During migration, descriptions should be explicit: one tool for legacy ticket creation, one for the new categorized workflow. After migration, retire the old one.

Evaluation coverage for schema behavior

Schema tests prove validation behavior. They do not prove the model will choose the right tool or fill arguments correctly. Add evals that prompt realistic user requests and assert tool call outcomes: selected tool name, required argument presence, enum values, and absence of unsafe optional fields.

Keep schema contract failures separate from model behavior failures. If Ajv rejects a fixture, the schema or fixture changed. If the model chooses the wrong tool, the prompt, description, examples, or model behavior changed. Mixing those failures makes triage slower.

For high-risk tools, include negative eval prompts: user asks for a refund without customer identity, user requests an unsupported priority, user tries to inject extra instructions into a free-text field. The expected behavior might be asking a clarification question rather than calling the tool.

Descriptions are part of the contract

Tool descriptions are executable guidance for the model. They are not decorative documentation. A description that says "Create a support ticket" may invite calls for every complaint. A description that says "Create a support ticket after the user has provided a customer identifier and issue summary" sets a clearer precondition. Contract review should include description changes because they affect tool selection.

Descriptions should define boundaries. Say what the tool does, when to use it, and when not to use it. Avoid overlapping descriptions across tools. If refund_order and create_ticket both claim to handle refund problems, the model has a routing ambiguity. Make one tool handle the transactional refund and the other handle support escalation after refund failure.

Keep descriptions stable enough for evals. Rewriting every tool description for tone can shift model behavior across the system. When descriptions change, run tool-selection evals even if the JSON Schema did not change. A schema diff checker will not catch this class of regression.

Schema compatibility for audit and replay

Agent systems often need to replay old conversations for debugging, evaluation, or compliance. If old tool calls no longer validate, replay becomes lossy. You may be forced to patch historical traces or skip side effects, both of which reduce confidence in regression analysis.

Store schema versions with tool calls. A trace should say which schema version the model saw when it produced the arguments. During replay, validate against the historical schema first, then optionally migrate to the current shape. This is especially important for tools that perform side effects such as ticket creation, refunds, account changes, or deployment actions.

Migration functions should be tested like any other compatibility layer. If priority: "urgent" becomes priority: "high" plus escalated: true, encode that mapping explicitly. Do not hope the model will regenerate old traces into the new shape. Replays should be deterministic.

Security checks around tool arguments

Schema validation is necessary but not sufficient for secure tools. A string that passes minLength can still contain prompt-injected instructions, SQL fragments, or unexpected URLs. The schema should constrain shape, while the implementation enforces authorization and domain rules.

Contract tests should include hostile but schema-valid values. For a send_email tool, test recipients outside the allowed domain. For a read_file tool, test path traversal strings if paths are accepted at all. For a deploy_service tool, test unauthorized environments. The expected result should be a safe rejection from the tool layer, not silent execution.

Do not put secrets in tool schemas or examples. Descriptions and examples may be copied into prompts, traces, logs, and evaluation datasets. Use placeholders that reflect shape without exposing real credentials or customer data.

Runtime error contracts for invalid calls

When validation fails, the agent needs a predictable error contract. A vague exception such as "bad request" is hard for an agent loop to recover from. A structured tool error can say which field failed, whether the error is retryable, and whether the model should ask the user for missing information.

Keep this error contract separate from the tool input schema but test them together. For example, a missing customerId should produce MISSING_REQUIRED_ARGUMENT with field: "customerId" and recoverable: true. An unsupported enum might be recoverable if the model can choose a supported value. An authorization failure is usually not recoverable by argument repair.

Tool registry checks in CI

Most agent platforms eventually collect tools in a registry. That registry needs its own checks. Tool names should be unique, descriptions should not conflict, schemas should compile, examples should validate, and deprecated tools should have removal owners. A registry test can load every tool definition and run the same contract rules before any agent test starts.

This catches boring but expensive mistakes: two tools with the same name, a missing required array, an enum that no example covers, or a schema that allows arguments the implementation drops. The registry check should fail fast. Model-based evals are slower and should not be the first place you discover malformed schemas.

Frequently Asked Questions

Is a tool schema just JSON Schema?

The parameters are usually expressed as JSON Schema, but the contract also includes tool name, description, versioning, runtime validation, and model selection behavior.

Is adding an optional tool parameter breaking?

It is usually validation-compatible, but it can change model behavior. Review prompts and evals when adding optional parameters to important tools.

Should tool schemas allow additional properties?

For side-effecting tools, usually no. additionalProperties: false prevents ignored or unexpected arguments from looking accepted. For exploratory read-only tools, looser schemas may be acceptable.

Do provider-side structured outputs remove the need for validation?

No. Validate again before executing the tool. Runtime validation protects you from integration bugs, schema drift, manual calls, and unexpected provider behavior.

When should I create a new tool version?

Create a new version when old valid calls would fail, when required arguments change meaning, or when the model needs a clearly different action contract during migration.