Pattern White Paper/Workshops/Reflection

ADPS held its first Reflection Module workshop on 12 August 2026. Participants brought production experience from R&D security, retail algorithms, quality evaluation, telecom knowledge engineering, software development, and travel services. The discussion began with in-application Generator-Critic loops and moved into offline evaluation, Skill evolution, attribution, self-heal authority, and delayed business feedback.

The agenda listed Han Zhao, lead for model backend and inference for Tencent Cloud Code Assistant, as a core workshop guest. The transcript records the invitation but no formal remarks, so this page does not attribute a technical position to him.

The workshop did not add a catalog number. It changed how the Reflection row should be read: F1–F4 describe different change targets and execution structures, while online/offline operation, feedback delay, evidence strength, and change authority determine how they enter production.

1. In-process reflection and out-of-process evaluation

Haili Zhang placed reflection in LangChain's wider Agent Development Lifecycle. In-process rubrics, graders, and revision loops help the current run converge. Outside the application, datasets, evaluators, and trace analysis compare versions, detect regressions, and calibrate reviewers.

The two share evaluation components but perform different jobs. Evaluation records a judgement. Reflection uses that judgement to propose or apply a change. A change without a re-run and regression evidence has not closed the loop.

The White Paper now records this boundary in the Reflection Module Overview.

2. Online reflection fixes the current run; offline reflection fixes the system

Wei Wang proposed a clear distinction. Online reflection follows the current run and reacts to tool errors, invalid arguments, or incomplete artifacts. Offline reflection examines a period of trajectories and looks for repeated, conflicting, or systemic behaviour across runs.

He described an environment-building agent that added a local remedy whenever it met a new failure. The environment became more stable, but the execution path grew substantially longer. An offline review revealed duplicate and conflicting patches.

This example gives online reflection a production constraint: keep its change surface small and rollback-safe, and subject accumulated changes to periodic global review.

3. Feedback delay determines loop length

Jiaqi Li compared support, operations, code generation, and business analysis. Code has compilation, tests, and deployment logs, so correctness evidence may arrive during the current run. A business recommendation may pass through product, operations, engineering, and user behaviour before anyone can judge it.

Real time is therefore constrained by the arrival of ground truth, not only by agent speed. While the final outcome is unavailable, an online reviewer may still check structure, references, and known contradictions. Business value must wait for delayed labels and human attribution.

The workshop recorded immediate/delayed feedback and closed/open tasks as related but non-equivalent dimensions. Open-ended work may contain hard local checks, while the final user experience of a closed task may still arrive later.

4. Reflection needs hard evidence and a stopping rule

Mo Zhou described five production properties: automation, termination, observability, reuse, and bounded cost. He then separated implementation into four layers:

1. Hard validation gives priority to machine-verifiable signals.
2. Model diagnosis inspects artifacts and trajectories.
3. Memory and asset admission determines what may affect later runs.
4. Scheduling and control owns activation, rounds, cost, degradation, and human hand-off.

The White Paper adopts this structure and narrows the reuse condition. A one-run correction may be deliberately discarded. Any result that will affect later runs needs admission, versioning, and retirement.

Zhou also discussed experiments in meta-reflection, deliberative reflection, and knowledge-completeness checks. They may help high-risk tasks without deterministic truth, but they remain exploratory and have not been added as formal patterns.

5. Three environments move bad cases out of production

Dong Zhang described development, evaluation, and production environments. Development permits close human-agent interaction. Evaluation runs a candidate rule or version against benchmarks. Production uses validated flows and assets.

When a bad case appears in production, the team keeps the production path stable, adds the failure to an evaluation set, produces candidate rule or knowledge changes, runs regressions, and promotes the new version through the environments. The practical implication is clear: a reflection that changes shared assets or production policy should usually enter an offline release process.

6. Attribution is a separate engineering step before healing

Qianchun Lu described reliability governance for organizational Agents and Skills: pre-release admission and inspection, runtime health checks and circuit breaking, and post-run trace review, remediation, and re-testing. His team is working on common primitives for instrumentation, failure classes, and evaluation.

He separated two kinds of root cause. Invalid tool arguments, missing steps, and network instability may have direct remedies. Missing domain knowledge, changed business rules, and different team expectations exceed the original agent's authority. A diagnosis may also be a false positive or identify a problem that has no authorised automatic repair.

This discussion tightened F4 Self-Heal Loop: it requires an identifiable failure, an executable repair, independent verification, and rollback. Knowledge, business decisions, and responsibility boundaries require human intervention.

7. Skill evolution must address creation, comparison, and coexistence

Wei Wang and Jiaqi Li both raised evaluation problems that appear as Skill estates grow. A Skill may work alone but mis-trigger, steal routing, or conflict with existing workflows in combination. Teams also need to separate the contribution of the agent, one Skill, and a bundle of Skills.

Pylon Peng described a daily reflection process. A main agent evaluates sub-agent work. A post-run hook collects trajectories and analyses whether memory, Skills, rules, sub-agents, or harness components need to be created or updated. When a tool argument or business rule changes, the corresponding Skill documentation and validation cases change together.

This daily batch process provides an implementation bridge between F2 Skill Package and F3 Experience Replay. Automatically generated assets still pass evaluation and a release gate before receiving durable authority.

8. Cross-model review can expose some shared blind spots

Jiaqi Li described a SQL generation and review system. When similar models generated and reviewed the SQL, some defects passed through review. Separating generation and review across models improved defect discovery in that setting.

Cross-model review adds diversity but does not guarantee independence. Models may share data, assumptions, and rubrics. The White Paper treats model diversity as a supporting measure; tests, rules, source evidence, and expert review remain stronger grounds.

9. Changes entering the White Paper

1. Add a bilingual Reflection Module Overview defining the boundary among observability, evaluation, reflection, and release governance.
2. Record online reflection and offline reflection as operating modes, without adding pattern numbers.
3. Extend F1 Generator-Critic with reflection contracts, trajectory review, critic calibration, and multi-dimensional rubric trade-offs.
4. Extend F2 Skill Package with candidate, isolated evaluation, coexistence evaluation, canary, version, and retirement stages.
5. Extend F3 Experience Replay with offline trace mining, delayed feedback, and consolidation of local patches.
6. Tighten F4 Self-Heal Loop by separating repairable runtime failures from missing knowledge and changed business logic.
7. Keep meta-reflection, deliberative reflection, and prospective reflection as research directions rather than formal patterns.

10. Open questions

• How should online change authority be tiered by task risk?
• How can a delayed business outcome be attributed to the exact agent version and trajectory that produced it?
• Which metrics should be common to isolated and combined Skill evaluation?
• Who owns admission and rollback when reflection changes memory, a Skill, or the harness?
• What reproducible cost and quality evidence would justify cataloguing meta-reflection, deliberative reflection, or knowledge-completeness checks?

Related pages

• Reflection Module Overview
• F1 Generator-Critic
• F2 Skill Package
• F3 Experience Replay
• F4 Self-Heal Loop
• White Paper contributors