X2 · Evals & Testing

Use repeatable cases, graders, regression, and external acceptance to govern capability evidence.

X2 governs evidence that a capability works. It joins deterministic tests, behavioral evals, external acceptance, and production feedback into release and revalidation decisions.

Scope

The system under test may be a model, prompt, tool, skill, router, workflow, governance policy, or complete agent system. Each object requires its own cases, environment, graders, and release gate.

Evidence	Decision supported
External facts and business receipts	Whether the real target state was reached
Schema, rule, state-machine, and idempotency assertions	Whether deterministic contracts hold
Trajectories, sandboxes, and fault injection	Whether execution, recovery, and authority boundaries hold
Calibrated model graders	Semantic quality that resists hard rules
Expert or user review	Ambiguous criteria and high-risk release decisions

Eval Contract

eval_id: payroll-change-regression-v3
system_under_test: payroll-agent@v8
task: change_one_allowance
environment: payroll-sandbox@2026.08
allowed_authority: no_production_write
required_outcomes: [receipt_matches_after_read]
forbidden_outcomes: [modify_unrelated_employee]
graders: [schema_contract, ledger_probe]
trials: 5
release_gate: all_required_cases_pass
evidence: artifacts/evals/payroll-v3/

Lifecycle

Design defines capability cases and negative cases. Pre-release runs capability and regression sets. Shadow operation compares candidates with the current version. Production failures become replayable cases. Model, tool, policy, or data changes trigger re-certification. G3 authority changes consume X2 evidence rather than a single demonstration.

Boundary with X1 and X3

X1 records what happened. X2 judges the result against a contract. X3 limits what the grader and candidate system may read or change. Editing the agent and grader together, or letting a candidate rewrite its own release gate, destroys independence.

Failure modes

Testing the final text while ignoring external state;
treating one stochastic success as stable capability;
positive cases without refusal, overreach, recovery, or unknown input;
one golden trajectory that rejects alternative correct paths;
a composite score without sample-level failures and trajectories;
uncalibrated graders whose score changes cannot be explained.

Suggested citation: ADPS, X2 · Evals & Testing, ADPS Cross-cutting Engineering Plane Specification v0.5, 20 August 2026.

Evals and testing topic · Reflection workshop · CC BY 4.0

Scope: This page defines engineering scope and interfaces; it does not certify products. Attributed practices remain governed by their case pages and public code.