Testing Prompt-Version Rollback Safety

At 14:06, the routing prompt is rolled back from support-router@43 to support-router@42. The deployment control reports success, yet half the workers continue serving version 43 from memory and version 42 emits a tool argument that the newly deployed executor no longer accepts. The prompt text was reversible. The running system was not.

A safe prompt rollback restores a coherent behavioral release, not merely an earlier string. That release includes model settings, tool definitions, output schema, examples, safety rules, parser expectations, and the identity used by caches and telemetry. Testing rollback means proving that an older immutable artifact can become active under today's surrounding code and that the fleet converges to it predictably.

Treat a prompt version as a release manifest

If “version 42” identifies only system.md, rollback reconstructs the rest from current defaults. A temperature change, renamed tool, or stricter JSON schema can silently combine old instructions with new runtime assumptions. Store or resolve a manifest that makes those dependencies explicit.

Manifest field	Why rollback needs it	Example verification
Prompt template digest	Proves exact immutable content	Digest matches artifact before activation
Model identifier and parameters	Behavior can change without text changes	Allowed model and bounded settings validate
Tool schema versions	Old prompt may request retired names or arguments	Every referenced tool exists in executor catalog
Output contract version	Parser compatibility is release-critical	Historical fixture validates against current reader
Template variables	Missing values can corrupt old instructions	Required keys are supplied before rendering
Evaluation suite revision	Explains the evidence used for promotion	Rollback report points to reproducible cases
Safety-policy dependency	Old behavior may violate current controls	Non-rollbackable policy layer remains enforced

Use an immutable version ID or content digest. Do not overwrite v42 in place. If a correction is necessary, publish a new version with lineage to the old one. Otherwise the label in a trace cannot tell investigators which bytes actually ran.

Separate release identity from an environment pointer. For example, production/support-router -> support-router@42 can change atomically while both artifacts remain immutable. The rollback operation updates the pointer under authorization and records previous version, target version, actor, reason, and timestamp.

Draw the compatibility window around the old artifact

The question is not whether version 42 passed when created. Ask whether it can run with the currently deployed executor, parser, data sources, and policy layer. Build a compatibility matrix for every rollback target retained by operations.

Older prompt behavior	Current runtime change	Required compatibility decision
Calls lookup_order with order_id	Tool renamed argument to id	Adapter, dual-schema support, or mark rollback unsafe
Returns output schema v2	Consumer now reads v3	Reader accepts v2 or migration converts it
Uses variable customer_tier	Renderer renamed it plan	Supply alias or reject artifact before activation
Emits legacy refusal code	Analytics expects new taxonomy	Map code without weakening safety behavior
Assumes six categories	Database now contains eight	Define fallback and evaluate unseen categories

Maintain a declared rollback horizon, perhaps the previous several promoted versions, based on release frequency and operational need. Every runtime change should test against that horizon. Supporting every prompt forever creates adapter debt; supporting only the current prompt makes rollback fictional.

Validate the artifact before changing traffic

Preflight checks should run on the target version while the current version remains active. Validate manifest schema, digest, template rendering, required secrets or data-source references, tool availability, model allow-list, and output parser compatibility. A failure must leave the active pointer unchanged.

The following TypeScript example models a registry and checks a target against a runtime catalog. It uses plain data so the policy can run in CI and in the deployment controller.

// prompt-rollback.ts
type ToolContract = { name: string; schemaVersion: number };

export type PromptRelease = {
  id: string;
  template: string;
  requiredVariables: string[];
  outputSchemaVersion: number;
  tools: ToolContract[];
};

type RuntimeCapabilities = {
  suppliedVariables: Set<string>;
  readableOutputSchemas: Set<number>;
  tools: Map<string, Set<number>>;
};

export function assertRollbackCompatible(
  release: PromptRelease,
  runtime: RuntimeCapabilities,
): void {
  const missingVariables = release.requiredVariables.filter(
    (name) => !runtime.suppliedVariables.has(name),
  );
  if (missingVariables.length > 0) {
    throw new Error(`missing template variables: ${missingVariables.join(', ')}`);
  }

  if (!runtime.readableOutputSchemas.has(release.outputSchemaVersion)) {
    throw new Error(`output schema v${release.outputSchemaVersion} is no longer readable`);
  }

  for (const tool of release.tools) {
    const versions = runtime.tools.get(tool.name);
    if (!versions?.has(tool.schemaVersion)) {
      throw new Error(`unsupported tool contract: ${tool.name}@${tool.schemaVersion}`);
    }
  }
}

Passing this function does not establish model quality. It establishes that declared interfaces are available. A prompt can still produce invalid content, choose the wrong tool, or degrade routing accuracy. Structural preflight is the first gate, followed by behavioral replay.

Replay production-shaped cases against both versions

Before rollback, run a compact shadow evaluation of target and active releases on the same sanitized inputs. Compare hard invariants first: schema validity, allowed tool names, argument validation, safety outcomes, and required citations or identifiers. Then compare graded qualities such as routing correctness and response helpfulness.

The target does not need to beat the version being rolled back from on every quality metric. Operations is usually rolling back because the current version has a serious defect. It does need to remain above minimum acceptance and avoid a worse known failure. Record that trade explicitly.

Use a corpus containing:

• Canonical cases that each supported intent must handle.
• Recent failures that triggered the rollback decision.
• Tool errors, empty retrieval results, and timeouts.
• Inputs introduced since the old version last ran.
• Adversarial and policy-sensitive requests.
• Long conversations whose earlier turns reference current schema terms.

The prompt regression testing guide covers broader corpus design. Rollback testing adds asymmetric questions: can the older artifact consume today's inputs, and will today's consumers accept its outputs?

Make schema backward readability executable

Suppose the active prompt emits output schema v3, while the rollback target emits v2. The current consumer can remain safe if it validates both and normalizes v2 into the internal representation. Test the historical wire payload, not an object already converted by test helpers.

// rollback-output.test.ts
import { describe, expect, it } from 'vitest';
import { z } from 'zod';

const V2 = z.object({
  schemaVersion: z.literal(2),
  queue: z.enum(['billing', 'technical', 'account']),
  confidence: z.number().min(0).max(1),
});

const V3 = z.object({
  schemaVersion: z.literal(3),
  route: z.object({ queue: z.string(), priority: z.enum(['normal', 'urgent']) }),
  confidence: z.number().min(0).max(1),
});

type NormalizedRoute = { queue: string; priority: 'normal' | 'urgent'; confidence: number };

function readRouterOutput(raw: unknown): NormalizedRoute {
  const v3 = V3.safeParse(raw);
  if (v3.success) {
    return { ...v3.data.route, confidence: v3.data.confidence };
  }
  const v2 = V2.parse(raw);
  return { queue: v2.queue, priority: 'normal', confidence: v2.confidence };
}

describe('rollback output compatibility', () => {
  it('normalizes the exact v2 payload emitted by the retained prompt', () => {
    const historicalWirePayload = {
      schemaVersion: 2,
      queue: 'billing',
      confidence: 0.91,
    };

    expect(readRouterOutput(historicalWirePayload)).toEqual({
      queue: 'billing',
      priority: 'normal',
      confidence: 0.91,
    });
  });

  it('rejects a v2 queue that never existed in its contract', () => {
    expect(() =>
      readRouterOutput({ schemaVersion: 2, queue: 'vip-escalation', confidence: 0.8 }),
    ).toThrow();
  });
});

Defaulting v2 priority to normal is a product decision. If old outputs cannot safely express urgent handling, the adapter should derive urgency from another trusted signal or declare v2 unsafe for rollback. Compatibility must not invent semantics for convenience.

Retire a reader only after every prompt version in the rollback horizon and every queued historical message using that schema has expired or migrated. Queue retention often extends the compatibility period beyond live request traffic.

Keep tool execution compatible in both directions

Tool calls cross two contracts: the model-facing definition placed in the prompt request and the executor implementation that receives arguments. Restoring an old definition can make the model produce an older shape. If the executor accepts only the newest shape, calls fail after rollback even when the model follows instructions perfectly.

An adapter can translate old arguments into a current internal command, but it must preserve validation. For example, map order_id to id only for contract v1, reject a payload containing both with different values, and attach the source version to audit logs. Avoid a permissive union that silently ignores unknown fields.

Create contract fixtures for every retained tool version and run them against the deployed executor build. Include success, validation rejection, authorization rejection, timeout, and idempotent replay. The function-calling regression suite guide expands this into model selection and argument-quality coverage.

If an old prompt references a tool removed for security reasons, do not restore it automatically. Safety controls and revoked capabilities should be forward-only. Mark that prompt release non-rollbackable and select an older compatible artifact or publish a repaired version based on it.

Test activation as a distributed state transition

Changing a database row is not enough when workers cache prompt releases. Define how convergence occurs: polling interval, pub-sub invalidation, request-time lookup, or version pinned at process start. The rollback procedure must either wait for convergence or route traffic only to confirmed workers.

A useful activation test starts several worker instances with version 43 cached, updates the environment pointer to 42, and observes which version ID each subsequent response reports. Assert convergence within the documented bound. Kill one worker's invalidation subscription to prove monitoring detects a mixed fleet rather than averaging it away.

Every model request should carry the resolved prompt release ID, not merely the desired environment pointer. Propagate it into traces, output records, tool calls, and metrics. During a partial rollout, a chart grouped only by “production” conceals the exact condition operators need to see.

Activation strategy	Rollback characteristic	Test emphasis
Resolve pointer per request	Fast convergence, registry dependency on hot path	Registry outage and consistency behavior
Local cache with short TTL	Bounded mixed-version window	TTL expiry and clock behavior
Push invalidation	Fast when subscribers are healthy	Dropped message and reconnect recovery
Version pinned at worker startup	Simple runtime	Coordinated restart and old worker draining

Atomic pointer update prevents an individual lookup from seeing half-written metadata. It does not make all workers change simultaneously. State the expected window so alerting can distinguish normal convergence from a stuck instance.

Include caches, conversations, and queued work

Prompt caches must include version identity in their key. A cache keyed only by rendered user input can return version 43 output after version 42 becomes active, creating the appearance of rollback failure. Test warm-cache rollback: populate under the current release, flip the pointer, issue the same input, and verify either a miss or an explicitly compatible cached artifact.

Long-lived conversations require a policy. Switching the prompt in the middle can combine old assistant messages and tool results with restored instructions. Pinning a conversation to its starting prompt improves consistency but delays rollback for existing sessions. Switching immediately reduces exposure to the bad version but needs evaluation on mixed history. Test the policy you actually operate.

Queued jobs should carry a resolved version or resolve at execution, never ambiguously do both. If they carry version 43 and execute after rollback, operations may consider that unacceptable. A cancellation or requeue procedure can rewrite only jobs whose semantics allow it. Record the decision in the rollback runbook.

Run a canary rollback before full reversal

Where incident severity permits, route a small controlled slice to the target release, including internal probes and representative production traffic. Watch hard-failure rates, schema rejection, tool validation, latency, token usage, refusal behavior, and task-specific outcomes. Percentage-based canaries need sticky assignment for conversations so turns do not alternate versions.

The canary is not a request for a vanity aggregate score. Define stop conditions in advance. One unauthorized tool invocation may be more important than a modest improvement in helpfulness. Conversely, a known formatting regression may be acceptable for ten minutes if the current release is exposing unsafe data.

Practice the mechanism outside incidents. A quarterly rollback drill can activate an older compatible prompt in staging, verify telemetry and fleet convergence, then restore the latest version. Drills find expired artifacts, missing permissions, and undocumented manual steps while there is time to repair them.

Preserve current safety rules during an old-prompt restore

Some layers must not roll back with the prompt: authorization, data-loss prevention, tool allow-lists, content policy enforcement, and emergency block rules. Keep those controls outside the versioned behavioral prompt or declare them as minimum overlays applied to every release.

Test that the old artifact cannot override or contradict the overlay. Include prompts that were benign when version 42 shipped but are now recognized as unsafe. Historical approval is evidence from an earlier threat model, not permanent permission.

At the same time, avoid appending so much current instruction text that “version 42” behaves nothing like its evaluated artifact. Version and hash the composed prompt layers, and record both base release and overlay release. Re-run the rollback corpus against the exact composition.

Prove rollback is reversible in the other direction

An operational rollback is usually temporary. After repairing version 43 or publishing 44, traffic moves forward again. Test that rollback does not mutate conversations, cache namespaces, or registry metadata in a way that prevents reactivation. The environment pointer should retain a history entry, not rewrite the target artifact's promotion record.

Run a three-step drill: serve the current release, activate the retained release, then restore the current release. At every step, assert the resolved version in model traces, tool invocations, and output records. Warm each version's cache before switching so the drill detects keys that omit version identity in both directions.

Database records created while the old schema is active may remain after moving forward. The v3 consumer must read v2 outputs written during the rollback window until retention expires or migration completes. This is why compatibility testing cannot stop when live traffic leaves version 42. Include queued work and delayed callbacks produced during the interval.

Measure time to detect, decide, activate, converge, and verify separately. A fast pointer update paired with fifteen-minute worker convergence is not a fast rollback. The drill should produce evidence for each phase and identify manual approvals that are necessary versus accidental. Automate repeatable validation, while keeping an authorized human decision for high-impact production traffic changes.

Frequently Asked Questions

How many prompt versions should remain rollbackable?

Set a declared horizon based on release cadence and adapter cost, then continuously test every version inside it. Keeping the previous two or three may suit a fast-moving router, while regulated workflows may require longer retention with stronger compatibility management.

Can we roll back prompt text without rolling back its model settings?

Only if that combination has been evaluated and represented as its own release manifest. Otherwise operators are activating an untested hybrid that happens to reuse old text.

What should happen to conversations already started on the bad version?

Choose and test either version pinning or immediate switching. Pinning preserves conversational consistency; switching reduces continued exposure. High-severity safety incidents usually favor switching plus explicit mixed-history cases.

Is schema validation enough to approve a rollback?

No. It proves structural readability, not routing quality, correct tool choice, safety, or distributed activation. Pair contract checks with production-shaped behavioral replay and a convergence test.

When is an older prompt intentionally non-rollbackable?

Mark it unavailable when it depends on a revoked tool, removed data contract, unsupported model, or superseded safety behavior that cannot be adapted honestly. Rollback capability must never resurrect a known-dangerous dependency.