Concepts/Definition

Application context

Higher field-extraction accuracy in a service-ticket agent does not necessarily shorten ticket resolution. An added clarification step may improve local correctness while increasing human hand-offs and customer waiting time. Technical regression detects capability loss; the business needs a separate measurement layer.

Definition

A Business Evaluation Set combines a critical business process, representative scenarios, scenario weights, metric definitions, and acceptance conditions. It runs an agent version through end-to-end work and observes completion time, first-contact resolution, human takeover, incorrect actions, business loss, or other domain outcomes. Weights come from business volume and risk; the model does not choose them.

business_evaluation_set:
  business_process: ...
  scenarios:
    - id: ...
      weight: ...
      metric: ...
      data_definition: ...
      acceptance: ...
  owner: ...
  version: ...
  change_log: []

Engineering use

Product, business, and engineering owners agree on sampling, metric denominators, time windows, and exception handling. Results are reported per scenario before aggregation under registered weights. After a model or workflow change passes technical regression, the business set checks whether it improved the process or merely shifted cost to human teams and downstream systems.

Boundary with technical evaluation

A Business Evaluation Set does not replace structural, factual, security, performance, or reliability tests. Technical evaluation asks whether a component meets its specification. Business evaluation asks whether the end-to-end process improved. Different owners may maintain the two assets, but a release receipt references both versions and results.