Concepts/Definition

Application context

A support agent faces different failure costs in refunds, delivery enquiries, and enterprise contracts. “85% accuracy” does not identify the sample source or show whether a high-risk scenario was hidden by an average. A model, prompt, harness, or tool change can also invalidate an earlier result.

Definition

A Scenario–Benchmark Contract binds a set of production scenarios to corresponding evaluation cases. Together they define the capability boundary, release threshold, and regression conditions. Each scenario records input provenance, expected outcomes, error costs, acceptance thresholds, and applicable versions. Production failures return to their scenario rather than disappearing into an unbounded aggregate score.

scenario_benchmark_contract:
  scenario_scope: []
  sample_sources: []
  error_costs: {}
  acceptance_thresholds: {}
  bound_versions:
    model: ...
    prompt: ...
    harness: ...
    tools: ...
  production_feedback: []
  owner: ...

Engineering use

The team selects scenarios from real task flows and assigns separate thresholds to high-risk or frequent work. Every candidate version runs against the same frozen set, while newly observed production cases are reported separately. The release receipt records the contract version, component versions, per-scenario results, and approved exceptions. A change to a bound component or error cost triggers re-evaluation.

Boundary

The contract states the capability range that has evidence. It does not promise automatic generalization to uncovered work, and it does not replace security, performance, or recovery tests. Broader scope, changed business definitions, or a new critical component requires a new version with an explicit diff.