Workshops/Reasoning

26 August 2026

The workshop covered online question answering, large codebase analysis, multimodal generation, latency-sensitive services, GIS toolchains, and physical simulation. Participants discussed how to choose reasoning paths, control call costs, and verify results.

1. Reasoning triggers, budgets, and acceptance

Huisheng Yin framed System 1 and System 2 as five system-design questions. Which requests enter a slow path? Which topology does that path use? How much time, model usage, and tool usage may it consume? Who verifies the result? What evidence is sufficient to stop?

A request can be easy to understand yet dangerous to execute, such as restoring sensitive data to an earlier state. Difficulty controls reasoning depth; action risk controls approval, authority, and verification. A policy-change example made the same separation visible: old policy, new policy, and effective date may be checked in parallel, but they must converge under one evidence standard. A single complexity score would hide that distinction.

An equipment alarm illustrates R4 Iterative Hypothesis Testing. The initial hypothesis is a mechanical fault. After a field engineer reports a recent configuration deployment, the next round adds a configuration check. The record should identify that new information and explain why the investigation changed direction.

2. Stronger models require controls to be measured again

Qingfeng Li and Huisheng Yin debated whether an established Skill can begin to constrain a stronger model. Earlier models benefited from manually decomposed tasks, explicit plans, and multi-sample reasoning. After a model upgrade, the same Skill may remain useful or may duplicate the model's own planning. Li kept the answer conditional on model version, scenario, and a local benchmark.

Online question answering supplied a concrete check. An answer should carry source excerpts from the knowledge base. The run log records which evidence supported the answer, and the team reviews those logs to decide whether an existing SOP still fits. The auditable record is evidence, output, and execution trace, not private chain-of-thought tokens.

Jia Huang compared software code with proprietary enterprise protocols. Code has public training material, compilers, and mature test tools. Proprietary business processes often do not. A prose-only instruction may still lose domain objects, protocol fields, state boundaries, and acceptance criteria.

After a model upgrade, teams can compare three versions: retain the current control, simplify it, and delegate most of it to the model. Quality, cost, latency, and failure modes determine whether the control remains in the model, harness, or deterministic program. Results should not be inherited across model versions without testing.

3. Large context must first become a usable structure

Dong Zhang described security analysis over a large codebase. Feeding many files directly to a model can leave it with disconnected points. Static analysis can extract syntax trees, call relationships, control flow, and data flow, then present a coherent path from entry point to sensitive operation.

This preparation removes irrelevant material while retaining causal structure. Acceptance therefore checks more than compression ratio. Entry points, propagation paths, and critical states must survive. The work sits at the Perception–Reasoning boundary: Perception turns raw material into structure; Reasoning forms a decision on that structure.

Dong Zhang described continuous improvement as “scenario plus benchmark.” A team builds a test set and acceptance threshold for one defined scenario. New failure cases are added to the dataset to test proposed changes to the prompt, retrieval, harness, or model before release.

4. Chains, trees, and reflection loops can nest

Stable steps fit a chain. Open questions expand into a tree of hypotheses. A branch may also contain a local reflection loop. Production graphs commonly combine all three: one step in a chain can open a tree, and one node in that tree can gather evidence repeatedly.

Dong Zhang separated precompiled branches from runtime branches. A known business process may be encoded as a workflow. An exploratory task may allow the model to propose hypotheses while it runs. Both still need one convergence node that compares conflicting conclusions under the same evidence rules.

Tree search also needs maximum branch count, maximum depth, and a cheap pruning rule. Huisheng Yin suggested handling retries at individual nodes: correct the format for a syntax error, reconsider the plan for a business-rule failure, and retrieve additional material when evidence is missing. The failure type determines which step to repeat.

5. Recursion, traceability, and permission failures

Dong Zhang described two kinds of loop exit. Reaching a resource or round limit forces a stop. Reaching the goal, passing a quality check, or making no material progress for two rounds can also end the current loop.

6. Online reasoning is constrained by the serving path

Han Zhao separated serving-layer latency controls from agent-layer routing. Quantization, prefix caching, and request scheduling reduce repeated work at the serving layer. The agent layer then selects a model, Skill, prompt parameters, and reasoning effort for the task.

Prefix caching depends on the order of the system prompt, tool descriptions, and user messages. Multi-instance deployments must check cache hits when requests reach different instances and adapt scheduling to the provider's caching rules.

A capable main model can choose a downstream model, yet that choice may itself consume too much latency and compute. Zhao described a lightweight classifier or small DAG that proposes a model from known scenarios. The proposal anchors the main model and does not replace its final decision. High-volume interactive services need to measure the saving; an offline task may not need the extra layer.

Models, harnesses, and artifacts have different update processes. Prompts, Skills, agent definitions, and plugins can change quickly. Model updates require data preparation, training, deployment, and regression. Harness updates affect routing, loops, and tool behavior, so they need compatibility and failure-path tests.

7. Speculative execution spends extra compute to reduce waiting

Fuhai Zhong described a latency-sensitive user service. An upstream classifier first narrows a request to a broad business category. Historical hit rates then identify a few likely agent candidates. Those candidates begin loading tools and fetching slower data while a routing agent decides which capability is actually required.

When the router selects a candidate, the system can reuse its prepared result. On a miss, the selected agent starts later while sharing any generally useful data already fetched. Evaluation needs candidate hit rate, extra-call cost, end-to-end latency, wrong selection, and stale results. Average latency alone can hide added cost and tail failures.

8. People and model judges begin with the same rubric

When the discussion reached confidence scores, Zhong described a layered evaluation path: verify that facts came from real tool data; check that the strategy followed business steps; require evidence for the conclusion; validate labels, schema, and required fields deterministically; use a model judge for open-ended quality; run historical regression; and retain product or business sampling.

Huisheng Yin asked how a model judge can be consistent when human reviewers do not share a stable standard. Zhong's answer was to start people and model judges from one checklist: required points, deductions for omissions, and format errors that block the result. Human ratings may still differ, but the team can now locate whether the problem is the standard or its application.

Fuhai Zhong also described a mirror agent. In an offline environment, it reproduces the tool interfaces, prompt, configuration, and answer policy of a deployed business agent. A developer or coding agent can modify tools, prompts, model selection, and configuration, then run the existing test set. Product or operations staff sample the results. Approved changes move to production code and still pass end-to-end and regression tests. The mirror shortens experimentation; it does not replace production acceptance.

9. Service requests and rendering checks in GIS publishing

Yuke Xiong's GIS project checks data processing, service publication, and visual rendering. A coordinate error introduced during data processing can leave the publishing call successful while the browser shows a blank map. Diagnosis requires inspecting coordinate processing, service requests, and the rendered result.

Mature command-line tools, database statements, and REST APIs allow many decisions to be compiled into fixed stages, routing tables, and error maps. A known transient failure can follow a retry rule. An unknown error should stop the current path and update the hypothesis, or hand a person or agent the symptoms, hypotheses tested and rejected, and supporting evidence. A raw log bundle is large but does not tell the receiver where to continue.

After publication, the system requests the service, opens it in a headless browser, stores a screenshot, and uses image checks or existing tools to confirm that the map appears.

This setting currently favors serial checks. Evidence is cheap and precise, and the toolchain is stable. Parallel exploration would add cost and make causal diagnosis harder. A pattern catalog should preserve why a pattern was not selected as well as why another one was.

10. Technical evaluation is followed by business evaluation

Didi Li noted that some physical-simulation settings lack even a shared semantic layer before reasoning begins. Data can be irregular and discontinuous, while physical structures, physical laws, and business processes remain in separate silos. She described maintaining physical and business ontologies that connect objects, processes, and operational data in a shared semantic space.

Passing individual technical metrics does not guarantee the assembled business result. Every component may pass inspection while the yield of the assembled module still falls. A business evaluation set follows real processes, weights critical scenarios, and connects them to business measures. Its version cadence differs from unit tests and should be maintained independently.

Business users may have difficulty describing requirements that span roles and processes. Jia Huang recommended the fieldwork methods used in forward-deployed engineering (FDE) and enterprise modeling: use interviews, questionnaires, and observation to establish business objects, processes, and acceptance requirements, then configure the agent's source material and tools accordingly.

11. Changes to the pattern specifications

1. R1 Chain of Thought retains explicit reasoning artifacts, evidence bindings, and decision summaries; private model CoT is not an audit record.
2. R2 Complexity-Based Routing separates task difficulty from action risk and adds latency, cost, model, and fallback budgets.
3. R3 Parallel Exploration adds speculative execution and distinguishes multiple solutions to one question from subtask fan-out.
4. R4 Iterative Hypothesis Testing adds known transients, unknown failures, reasoning handoff packages, and external acceptance.
5. R5 Talker-Reasoner retains tests for topic changes, background cancellation, stale results, and front-channel overreach.
6. Scenario–benchmark contracts, common convergence nodes, mirror agents, and business evaluation sets enter concept review.

12. Questions still open

• What evidence shows that a reasoning control should move from the harness into the model, or back out again?
• At what candidate hit rate, latency budget, and extra cost does speculative execution become worthwhile?
• How should a common convergence node handle conflicting evidence, judge bias, and dependent branches?
• How can a mirror-to-production gap be measured, and which changes may be written back automatically?
• Who owns the metrics, weights, and versions of a business evaluation set?
• When users cannot state intent precisely, how should interactive clarification, domain modeling, and FDE divide the work?

Related pages

• Reasoning module overview
• Agent pattern composition
• Agent evaluation and validation
• Reasoning assetization

Based on the 26 August 2026 workshop transcript. Internal project details have been anonymized.

Edit source · Published source · Review process