Pattern Matrix/White Paper/R1

Problem

The model returns a conclusion, but no one knows how it got there. When the conclusion is correct, no one asks; when it is wrong, there is no way to debug it; and when a regulator comes to inspect, there is no evidence to produce. One of the biggest differences between an agent and traditional software is this "gray-box" quality—it is neither a black box nor a true white box. If the reasoning layer in the middle is not captured, there is no way to root-cause an incident.

Here, Chain-of-Thought means engineering the reasoning trajectory: storing permitted reasoning artifacts or summaries, preserving input evidence, handling cross-model fallback, and controlling reasoning effort by task complexity. The familiar “Let's think step by step” prompt is only one prompting technique; reasoning APIs expose different structured fields that the runtime must normalize.

Classification: Reasoning × Chain

• Vertical axis · Reasoning: CoT concerns the trajectory from evidence to decision. Depending on the model interface, the system may receive a public rationale, a structured summary, or protected reasoning tokens. The other reasoning patterns build on this trajectory in different ways.
• Horizontal axis · Chain: Reasoning itself is a linear trajectory, running from input to output, where each step depends on the one before it (thinking → intermediate conclusion → final answer). This is a naturally chained structure, not a parallel fan-out, and not a looping iteration.

Solution and mechanics

CoT is a multi-form engineering category. Its runtime responsibilities include:

1. Persistence: Retain the reasoning summaries, evidence references, decisions, and model metadata that the provider and policy allow, indexed by trace_id.
2. Cross-model normalization: Providers expose public summaries, structured fields, or protected reasoning tokens differently. Normalize available artifacts into one schema and retain provider, model, interface version, and visibility metadata.
3. Cross-model fallback strip: a thinking block's signature is bound to the model that generated it. When the primary model is rate-limited and traffic switches to a fallback model, the fallback model does not accept signatures from another model. All incompatible thinking blocks must be stripped before sending, otherwise the entire request is rejected and the agent call fails.
4. Effort control: thinking consumes paid tokens, but more thinking does not necessarily mean higher quality. Provide an effort control surface (off / low / medium / high / max), so simple tasks use low and complex tasks use high.

One boundary matters: a model's written rationale may be a post-hoc account rather than a faithful record of its internal computation. Treat it as an observability signal, not proof of how the model reached the answer. Critical decisions still need external evidence and verification.

Applicability

• Multi-hop logic and judgments that need explanation: claims review, contract risk identification, credit decisions—each conclusion must be explainable to a regulator in terms of "what the basis is."
• Compliance audit-trail scenarios: Finance, healthcare, and legal workflows may require decision evidence, model metadata, approvals, and retained rationale. The exact record follows applicable policy and provider constraints.
• Teaching and debugging scenarios: Show processed reasoning summaries and intermediate evidence when this helps locate an error, without assuming access to private model reasoning.

Known failure modes

• Forcing step instructions onto a reasoning model: Adding a generic “analyze step by step” instruction can increase token use without improving the target task. Test prompting choices on the actual evaluation set.
• Not stripping thinking on fallback: failing to clean up cross-model signatures when the primary model goes down and traffic switches to fallback causes the entire request to be rejected and the agent call to fail. This kind of incident is not exposed in normal operation but breaks out all at once when the primary model is rate-limited.
• Writing traces only to log files: When a reviewer needs the evidence, applicable rules, model version, and final decision for a case, scattered log records are difficult to reconstruct. Use structured traces that are queryable by trace_id.
• Running every task at one effort level: using the same effort for a simple task ("what day is it today") and a complex task ("what risks does this contract have") both wastes tokens on the simple task and shortchanges the quality of the complex one.
• Enabling deep reasoning in latency-sensitive scenarios: Extended reasoning can violate an interactive response budget. Use a lower effort level, asynchronous execution, or the Talker-Reasoner pattern.

Verification and metrics

• Reasoning token share: Track the share of reasoning tokens by task class. A material rise from the local baseline may indicate over-thinking; a drop should be checked for under-reasoning on difficult tasks.
• Fallback strip success rate: Verify that model-specific reasoning fields are removed or transformed before a fallback call. Any incompatible payload is an operational defect.
• Trace queryability: Verify that a historical decision can be reconstructed from its trace ID using retained evidence, model metadata, outputs, and allowed reasoning summaries. Applicable policy defines what may be stored.

Reference implementation

task arrives → choose effort tier by complexity (off / low / medium / high / max)
            model returns retainable summary / structured decision / model metadata → normalize into a unified schema → into structured trace
            if primary model is rate-limited and falls back:
                strip all thinking blocks incompatible with the target model, then send the request
            dual views for audit retrieval:
                audit view    → permitted reasoning summary + evidence + final decision + fallback chain
                customer view → redacted rationale, no protected reasoning details
            return final_answer + audit trace (indexed by trace_id, retained according to policy)
            

Four points for implementation: do not add step instructions to a reasoning model; make effort control a per-request / per-task configuration; keep extending tag normalization as new models appear; and emit traces to a structured trace bus that supports long-cycle retrieval.

Illustrative scenario

Liang Bo's execution-oriented agent separates explanatory reasoning from the structured answer that downstream programs consume. The runtime stores provider-permitted summaries and evidence in the trace, while the answer is a JSON decision object that drives subsequent actions. This keeps audit material separate from the executable control contract.

When JSON parsing fails, the runtime uses a predefined safe fallback instead of crashing or retrying at random. The answer schema is strictly validated, parse failures enter monitoring, and audit retention follows the visibility allowed by the model interface and organizational policy.

Related patterns

• Complexity routing (R2): CoT's effort control adjusts the reasoning tier by task depth, while routing selects the model tier by task complexity—the same line of thought, lifted from a single call to the task level.
• Parallel exploration (R3): each branch of a parallel run is usually a chain of thought internally, and parallelism layers "multi-sampling" on top of CoT.
• Iterative hypothesis verification (R4): each round of iteration is also a chain of thought, and iteration strings multiple chains of thought together along the time dimension for repeated correction.
• Dual-mode architecture (R5): the deep reasoning on the Reasoner side is essentially a complete chain of thought, while the Talker side deliberately does no deep reasoning.

Design conclusion

The engineering task is to manage available reasoning artifacts, evidence, decisions, and fallback metadata across their lifecycle. Private model reasoning is not required for an auditable action contract.