Pattern Matrix/White Paper/R1

R1 · Chain-of-Thought

What problem it solves

The model returns a conclusion, but no one knows how it got there. When the conclusion is correct, no one asks; when it is wrong, there is no way to debug it; and when a regulator comes to inspect, there is no evidence to produce. One of the biggest differences between an agent and traditional software is this "gray-box" quality—it is neither a black box nor a true white box. If the reasoning layer in the middle is not captured, there is no way to root-cause an incident.

Chain-of-Thought pulls this reasoning process out of the model's interior and turns it into first-class data in the trajectory. In 2026 it is no longer the 2022 prompt trick of saying "Let's think step by step"—a reasoning model already emits its reasoning process, and what the engineering layer has to do is manage the lifecycle of that process: how to persist it, how to audit it, how to migrate it when falling back across models, and how to control its depth by task complexity. Writing an agent without writing a CoT trace is like writing a backend without logging.

Why the coordinate is "Reasoning × Chain"

• Vertical axis · Reasoning: CoT makes the "why" explicit by having the model write out its thinking process. It is the origin pattern of the pure reasoning domain. The other four patterns in the reasoning module (Route, Parallel, iterative, dual-mode) all extend in different directions from this CoT baseline.
• Horizontal axis · Chain: Reasoning itself is a linear trajectory, running from input to output, where each step depends on the one before it (thinking → intermediate conclusion → final answer). This is a naturally chained structure, not a parallel fan-out, and not a looping iteration.

Core mechanism

In 2026 CoT is a multi-form engineering category, and its core is to manage the reasoning trajectory as structured data rather than discarding it once read. In practice there are four hard responsibilities:

1. Persistence: thinking blocks go into the trajectory log, indexed by trace_id, stored in structured form, supporting replay, rollback, and migration across sessions.
2. Cross-model normalization: different reasoning models use different tag formats (OpenAI uses <reasoning>, DeepSeek uses <think>, Anthropic uses structured fields). Normalize them into a unified schema before storage, so the agent always sees a trajectory in a consistent format and an audit only needs to read one structure.
3. Cross-model fallback strip: a thinking block's signature is bound to the model that generated it. When the primary model is rate-limited and traffic switches to a fallback model, the fallback model does not accept signatures from another model. All incompatible thinking blocks must be stripped before sending, otherwise the entire request is rejected and the agent call fails.
4. Effort control: thinking consumes paid tokens, but more thinking does not necessarily mean higher quality. Provide an effort control surface (off / low / medium / high / max), so simple tasks use low and complex tasks use high.

One boundary worth noting: a reasoning model has weak control over its own CoT, and the reasoning it writes may be a "script" composed after the fact rather than the actual reasoning path. So CoT is a useful observability signal, but it cannot be treated 100% as the model's actual thinking process; critical decisions need an additional layer of external verification.

Suitable production scenarios

• Multi-hop logic and judgments that need explanation: claims review, contract risk identification, credit decisions—each conclusion must be explainable to a regulator in terms of "what the basis is."
• Compliance audit-trail scenarios: finance, healthcare, and legal domains require the complete reasoning of a decision to be retained long-term, and the CoT trace is the carrier of this compliance baseline.
• Teaching and debugging scenarios: a counterexample—these scenarios should instead make thinking explicitly visible, where exposing the reasoning process is a plus.

Where it goes wrong

• Forcing step instructions onto a reasoning model: the model is already thinking, and feeding it "please analyze step by step" disrupts the reasoning and burns more tokens. Wharton 2025 empirical work shows this addition is nearly ineffective or even worse on reasoning models.
• Not stripping thinking on fallback: failing to clean up cross-model signatures when the primary model goes down and traffic switches to fallback causes the entire request to be rejected and the agent call to fail. This kind of incident is not exposed in normal operation but breaks out all at once when the primary model is rate-limited.
• Writing traces only to log files: when a regulator wants to "pull the complete reasoning of a given case on a given day," a trace scattered across log files cannot be dug out. It should be structured into OpenTelemetry / Datadog, queryable by trace_id.
• Running every task at one effort level: using the same effort for a simple task ("what day is it today") and a complex task ("what risks does this contract have") both wastes tokens on the simple task and shortchanges the quality of the complex one.
• Enabling thinking in latency-sensitive scenarios: in scenarios where user input must get a response within 200 milliseconds, extended thinking introduces second-level latency and should be turned off or set to effort=low.

Key metrics

• Reasoning token share (healthy band 30–60%): the proportion of thinking tokens in total tokens. A p99 above 80% indicates over-thinking on simple cases and wasting cost; below 20% indicates under-thinking with an accuracy risk. Crossing the threshold in either direction should trigger an alert.
• Fallback strip success rate (healthy band near 100%): the proportion of correct cleanup of cross-model thinking when the primary model switches to fallback. Below 100% means there is a latent risk that can cause an agent call to fail outright.
• Trace queryability rate (healthy band 100%): the proportion of any historical decision whose complete reasoning can be pulled by trace_id. In compliance scenarios this is a hard requirement.

Minimal implementation skeleton

task arrives → choose effort tier by complexity (off / low / medium / high / max)
            model emits thinking blocks → normalize into a unified schema → into structured trace
            if primary model is rate-limited and falls back:
                strip all thinking blocks incompatible with the target model, then send the request
            dual views for audit retrieval:
                regulator view → complete thinking + final decision + fallback chain
                customer view  → redacted summary only, no exposure of reasoning details
            return final_answer + complete trace (indexed by trace_id, retained long-term)
            

Four points for implementation: do not add step instructions to a reasoning model; make effort control a per-request / per-task configuration; keep extending tag normalization as new models appear; and emit traces to a structured trace bus that supports long-cycle retrieval.

Enterprise implementation example

An enterprise execution-oriented agent team (Liang Bo's team) found a dividing line during implementation: chain-of-thought is not "writing an extra paragraph of thinking for the user to read," but producing a Decision control signal that a program can consume on every call. The concrete approach is to split the model output into two parts, thinking and answer—the thinking part is the reasoning process that goes into the trace for archiving, and the answer part is a structured JSON decision object that downstream programs parse and execute directly. Once the two are separated, CoT shifts from "content" to "control flow": fields in answer directly drive subsequent actions, while the process in thinking is available for audit retrieval.

The key engineering safeguard is a deterministic fallback when JSON parsing fails—the model occasionally writes a decision that does not conform to the schema, and in that case the agent does not crash or retry at random but falls back to a predefined safe default action. This design is the key to moving from a "content-oriented agent" (generating text for people to read) to an "execution-oriented agent" (producing signals for programs to use). The accompanying decisions also include archiving the full thinking for audit, strict validation of the answer schema, and wiring parse-failure counts into monitoring.

Relationship to other patterns

• Complexity routing (R2): CoT's effort control adjusts the reasoning tier by task depth, while routing selects the model tier by task complexity—the same line of thought, lifted from a single call to the task level.
• Parallel exploration (R3): each branch of a parallel run is usually a chain of thought internally, and parallelism layers "multi-sampling" on top of CoT.
• Iterative hypothesis verification (R4): each round of iteration is also a chain of thought, and iteration strings multiple chains of thought together along the time dimension for repeated correction.
• Dual-mode architecture (R5): the deep reasoning on the Reasoner side is essentially a complete chain of thought, while the Talker side deliberately does no deep reasoning.

What this pattern teaches

The essence of Chain-of-Thought in 2026 is not "teaching the model to think," but managing the process the model is already running through as first-class data—it is the foundation of observability in the agent era, not an option.

This page is part of the ADPS pattern white papers. Back to the pattern matrix, or see the runnable code catalog.