Topics/Observability-Driven Agent Evolution: From Runtime Timelines to Verifiable Changes

Topics synthesize engineering questions that cross several modules. Pattern definitions, attributed cases, and workshop records remain authoritative on their own pages.

When an agent run fails, a final answer and a few application logs rarely explain the failure. Engineers need to know what the agent saw, which tool it selected, which state it changed, and what evidence supported its completion claim. Once a team starts editing prompts, tools, skills, memory, or execution structure, it faces a harder question: did the change improve later tasks without introducing unacceptable regressions?

These questions cross Perception, Action, Reflection, Memory, and Governance. G4 Observability Harness gives a local implementation pattern. This topic examines observability as the engineering base for continuous agent evolution.

Two meanings of visibility

Perception determines which external signals an agent can read. Observability determines whether an engineering team can reconstruct a run and measure the effect of a change. The former supplies inference inputs. The latter supplies evidence for debugging, review, release, and accountability.

An observability system should support at least three queries:

1. What happened during this run?
2. Which component, version, or decision produced the result?
3. After a change, did later outcomes improve and did regressions remain acceptable?

Four observable surfaces

A runtime trace supports single-run debugging. Evolution also requires component versions, a prediction attached to each change, and evidence from subsequent evaluations.

How evidence enters the change process

Instrument and version
        ↓
Collect runtime events and external outcomes
        ↓
Aggregate by task, version, and causal chain
        ↓
Locate a candidate component and propose a testable change
        ↓
Run capability and regression evaluations
        ↓
Approve release or roll back

The change record turns an intuition into a checkable contract. It states which component will change, which task class should improve, which regressions are protected, and which deterministic evidence and evaluation results will decide the outcome.

Two reusable engineering practices

Activity and Frame runtime timeline

The Dongfang Yiteng execution-agent case established a unified timeline early in the project. An Activity is a business-semantic step. A Frame records the model call, tool event, or state change inside it. Intent classification, routing, ReAct iterations, approval waits, and business receipts can be queried under one run_id.

The same design serves three roles. Developers locate the failing Frame. Business users inspect progress and receipts. Operators inspect latency, cost, and failure distributions. Production views apply role-based redaction instead of exposing debugging prompts and business data to end users.

Component, experience, and decision observability

Fudan University's 2026 Agentic Harness Engineering research separates automatic evolution into three observability concerns:

• Component observability represents system prompts, tool descriptions and implementations, middleware, skills, sub-agent configuration, and long-term memory as file-level, comparable, revertible artefacts.
• Experience observability keeps trajectories, failures, evaluations, and environment state in a drill-down evidence corpus instead of retaining only a summary.
• Decision observability attaches a prediction to each edit and verifies it against outcomes from the next evaluation round.

The research also exposes a boundary. An agent may propose a plausible fix from its trajectory and still fail to predict damage to other tasks. Evaluators, runtime protections, and rollback channels therefore belong to a protected governance boundary and should not be freely rewritten with the agent.

A minimal event contract

{
  "run_id": "run-20260814-0042",
  "task_id": "publish-map-17",
  "goal_version": "goal-v3",
  "component_versions": {
    "prompt": "sha256:...",
    "toolset": "registry-v12",
    "skill": "gis-publish-v5"
  },
  "event_type": "tool_result",
  "input_ref": "artifact://plan/step-4",
  "tool": "verify_get_map",
  "state_delta": {"verification": "failed"},
  "evidence_ref": "artifact://receipts/getmap-17",
  "latency_ms": 842,
  "cost_usd": 0.01
}

A change record can add decision_id, predicted_effect, observed_effect, and rollback_ref. Every field should answer a real diagnostic or review query; collection volume is not a goal by itself.

Relationship to ADPS modules

Observability is therefore both the implementation responsibility of G4 and shared infrastructure for long-running work, evaluation, reflection, and memory.

Common failure modes

• Logs contain text but no semantic link among tasks, steps, state, and evidence.
• Only the final response is stored; tool parameters, intermediate state, and external outcome are missing.
• Traces are not bound to prompt, model, tool-set, and skill versions, so failures cannot be reproduced.
• Dashboards show tokens, latency, and cost without external acceptance or business outcomes.
• An auto-evolving agent can edit both itself and the evaluator that judges it.
• Event collection is extensive, but there is no fixed reviewer, remediation entry point, or regression process.

Questions for further discussion

A productive workshop can examine three artefacts: a real task timeline, a diff that links failure evidence to a component change, and a release record containing capability and regression results. Product tours and metric names alone do not expose the engineering trade-offs.

Sources

• G4 Observability Harness
• Dongfang Yiteng execution-agent case
• Fudan University: Agentic Harness Engineering
• First ADPS Action Module Workshop
• First ADPS Reflection Module Workshop