Pattern Matrix/White Paper/G5

G5 · Observability Harness

What it solves

An LLM agent has no readable source code, the model weights are a black box, and the only window into what it actually did is the trace. An agent system with weak observability is itself a black box: regulators won't sign off, operations won't dare run it, and when something goes wrong nobody can diagnose it. On top of that, agents fail differently from traditional software. Traditional software throws an exception at a particular timestamp with a clear stack trace; an agent drifts slowly: fully correct, still correct, looks correct, confident but wrong, silent catastrophe, with no single crash point.

Observability fits the agent with a flight data recorder. It collects telemetry on every LLM call, every tool call, and every sub-agent step, and stitches the scattered spans into a complete call chain under one trace id. Regulators can be handed a complete trace at audit time, incidents can be root-caused down to the exact step, product and customer support teams can pull evidence within minutes during everyday decisions, and over the long run the same data drives optimization.

Why the coordinate is "Governance × Orchestrate"

• Vertical axis · Governance: Observability governs whether all agent operations are observable and replayable; it is the traceability foundation of governance. The records for approval gates, blast radius, and progressive commitment all depend on it, otherwise the approval log goes unread, a blast-radius incident cannot be reconstructed, and changes in trust tier cannot be explained.
• Horizontal axis · Orchestrate: Monitoring is fundamentally a centralized coordination problem—no single component can see the full trajectory. The tool cannot see the LLM's reasoning, the LLM cannot see the tool's side effects, the sub-agent cannot see the main agent's plan; only the orchestrator can assemble the complete picture. It cuts across every agent / tool / step, does not belong to any particular chain / route / loop, and is cross-cutting orchestration.

Core mechanism

Observability shapes agent telemetry into five dimensions, and missing any one of them leaves a blind spot:

The five dimensions are not flat. Log / metric / trace are the lower-level layer that records facts; cost / quality are the upper-level layer that analyzes slope. Because agent failure is drift rather than a step change, the key judgment is to monitor trajectory slope (the rate of change) rather than threshold crossing—quality degrading around a turn, the tokens consumed per result swelling between cycles. These slope signals are the real symptoms of failure. Across processes, the W3C Trace Context traceparent stitches all spans into one trace id, paired with the standard attributes from the OpenTelemetry GenAI Semantic Conventions, where gen_ai.request.model and gen_ai.response.model are recorded separately—the cure for the "requested Sonnet but actually ran Haiku" class of fallback that traditional APM cannot see.

Where it fits in production

• Any enterprise production agent: Compliance and customer-audit requirements are non-negotiable, and five-dimensional telemetry is the floor.
• Multi-tenant SaaS: Per-tenant cost attribution is the basis for billing, and also the entry point for cross-tenant isolation audits.
• Multi-model systems: An invisible model swap turns into a long blind hunt, while request.model != response.model exposes it on a single query.
• Multi-sub-agent or long-conversation systems: Without stitched traces across agents the system is a black box, and the evolution of a 200K context must be replayable.

Where it tends to go wrong

• Volume Overwhelms Signal: A busy agent produces thousands of spans an hour, and three red ones hide among two thousand greens on the dashboard with nobody watching. The fix is structured alerting—run slope alerts on only a handful of production metrics (success rate, cost per task, tool accuracy, human override rate, latency p99), use the dashboard to read trends rather than individual events, and route the rest through sampling plus query-on-demand.
• Trace Sampling Bias: Quality evaluation has a cost, so 10% random sampling is common, but if certain tenant cases skew toward the easy end the sample is all easy cases, the dashboard shows a perfect 0.95, and the hard cases have already fallen to 0.6. The fix is stratified sampling plus full sampling for SLA tiers, forcing sampling stratified by difficulty and sampling Enterprise tier at 100%.
• Schema Drift: The OpenTelemetry GenAI conventions are still evolving, a field hard-coded today gets renamed next year, and the dashboard, alerts, and cost engine all break. The fix is schema versioning plus dual-write plus a migration window, and hard-coding no field until a stable convention lands.
• Building your own observability backend: The trace store plus query engine plus dashboard infrastructure takes several engineers a year to build from scratch. The 2026 engineering discipline is to connect directly to mature tools such as Langfuse / Phoenix and add a thin layer of business extension.

Key metrics

• Five-dimension collection completeness (healthy range: neither cost nor quality is missing): whether the AI-native cost and quality dimensions are being collected. Without these two dimensions the slope signal of failure disappears, and after-the-fact replay shows only "everything looks normal."
• Ratio of trajectory-slope alerts to threshold alerts (healthy range: slope alerts dominate): whether the rate of change is being monitored. Pure threshold alerting is largely ineffective in the agent era, because failure is drift rather than line-crossing.
• Cross-process trace stitching rate (healthy range close to 100%): whether a sub-agent's spans can be attached back to the parent trace id. If they cannot, you are left with a pile of isolated spans and cannot reconstruct the complete call chain.
• Difficulty-distribution bias of the sample (healthy range: the sample distribution tracks the actual distribution): whether the sampled cases represent the whole. A skew toward the easy end makes the quality dashboard systematically overstate true quality.

Minimal implementation skeleton

each LLM call / tool call / sub-agent step produces a span:
                fill OTel GenAI attributes (with request.model and response.model recorded separately)
                cost = compute_cost(span)                 # computed from tokens and the actual model
                if sampled → quality = llm_judge(span)     # hallucination / refusal / scope_creep
                emit to the OTel pipeline (all five dimensions at once)

            across sub-agents: the main agent injects the W3C traceparent into the carrier,
                               the sub-agent extracts it and continues under the same trace_id

            trajectory_slope(tenant, metric, window): returns the rate of change, not a threshold
            query_for_audit(trace_id): one trace_id stitches all spans together for regulator replay
            

Three places must be changed for engineering rollout: replace PRICING with actual model prices; replace llm_judge with a real cheap judge call (such as Haiku) rather than a mock; add tenant_id / user_id / workflow_id into the business slots, and wire emit to a real collector or Langfuse endpoint.

Enterprise rollout example

In February 2026, at an edtech company with 120,000 monthly active middle-school students, homework-feedback quality suddenly collapsed at nine on a Tuesday morning, and complaints poured in—"the AI wrote the wrong solution steps," "the AI refused to answer a basic physics question." The team checked the application log (normal), checked the metrics (request count / latency / error rate, normal), checked Datadog APM (everything OK), and stayed stuck for six hours. Finally an engineer happened to open the Anthropic Console and found that a cost-saving deployment in the early hours of Monday had switched the main agent's model from Sonnet 4.6 to Haiku 4.5, and a misconfigured routing sent every problem to Haiku. The conclusion at the post-mortem was: "We had log plus metric plus trace, but none of them told me which model the task used. Observability for an LLM agent needs the traditional three pillars plus the two AI-native dimensions of cost and quality, and missing even one means you cannot diagnose it." Traditional APM could not see that cost per call dropped from 0.03 to 0.001 at the moment of the model swap (which looks like a good thing), nor that the quality score dropped from 0.92 to 0.71 at the same instant (which is the actual problem); set the two curves side by side and the model swap was immediately exposed. This is why five-dimensional telemetry is the floor.

In the engineering implementation of an execution-oriented agent, observability can be realized through a unified activity-event abstraction: express every activity event of the agent uniformly with Activity plus Frame, and the developer can see the complete timeline of the execution process (for example via real-time SSE push). One distinction is worth noting—"observable" here means making the execution process visible to the developer, which is not the same as "the agent's perception of the world." The former is telemetry on the governance side; the latter belongs to the perception module.

Relationship to other patterns

• Approval gate (G1) / Blast radius (G2) / Progressive commitment (G3): the infrastructure they depend on. The records for these three—the approval log, blast-radius triggers, changes in trust tier—all converge in G5 via the trace_id. Without G5, the first three are promises that cannot be honored.
• Hook pipeline (G4): complementary. The hook itself must leave a trace, and a Stop hook is commonly used to attach an audit commit, making it one of G5's data-collection entry points.
• Traces from the reasoning / reflection modules: the CoT trace is observability for the reasoning dimension, and the silent correctness discussed in reflection is exactly what G5 addresses with trajectory slope.

What this pattern teaches

The biggest user of observability is not the agent but people—it compresses the communication and impact analysis that consumes most of an engineer's time, the kind of "which customers does this change affect" question, from two weeks of meetings down to an hour of querying traces. The governed agent ends up with more autonomy, because the organization knows the worst case is bounded.

This page is part of the ADPS pattern white papers. Back to the pattern matrix, or see the runnable code catalog.