Pattern Matrix/White Paper/R4

Problem

Some tasks cannot be answered in a single pass of reasoning: the root cause is unknown, the evidence must be gathered step by step, and the first judgment is often wrong. Diagnostic tasks are the typical case—when a fault alarm fires, the "most likely cause" that the model ranks by prior may be badly miscalibrated, with the true root cause ranked far down or not even considered at first. If the model reasons once and then commits, it walks further and further in the wrong direction; if it simply retries, the retry itself does not solve the problem, because the issue is not that "the result is unstable" but that "the understanding of the world is still wrong."

Iterative hypothesis testing replaces single-pass reasoning with a loop of "hypothesize → validate → revise → hypothesize again." The fundamental difference from ordinary retry is this: retry assumes that the retry itself can solve the problem, while iterative hypothesis testing assumes that each loop must update the understanding of the world. Its design center is also not "retry until success," but converging through falsification to the single hypothesis that has not been refuted. It is the dual of parallel exploration (R3): parallel opens N lines at once along the spatial dimension, while iteration runs one line many times along the time dimension.

Classification: Reasoning × Loop

• Vertical axis · Reasoning: It follows empirical, scientific-style reasoning of "hypothesize → validate → revise → hypothesize," not a single deduction. Each round actively revises its own belief rather than waiting for the result to stabilize.
• Horizontal axis · Loop: Multiple iterations until the evidence converges or a limit is reached form a natural loop structure. The dual-mode architecture (R5) in the same column is also in the Loop column but has a different emphasis—iteration is a single agent looping with itself to validate hypotheses, while dual-mode is two agents collaborating to split the handling of "talking" and "thinking."

Solution and mechanics

A production implementation separates three responsibilities. They may be stages in one agent or roles assigned to different agents:

1. Hypothesis generation (Planner): List candidate hypotheses from symptoms and historical cases, ranked by prior probability. Select the model and effort setting through local evaluation of hypothesis coverage, missed causes, and cost.
2. Evidence collection (Generator): Given a hypothesis, choose tools that can test it, such as metrics, logs, or sensors. Evidence must come from traceable, reproducible data sources; the model's own impression is not external evidence.
3. Judgment (Evaluator): Decide whether the evidence confirms, falsifies, or leaves the hypothesis unresolved. The Evaluator should search explicitly for counterevidence. Compare prompts and model configurations on replay data instead of assuming that one framing always improves accuracy.

The loop rules are: falsified goes back to the generation stage, confirmed exits, and insufficient evidence goes back to gather more. Two engineering practices are easy to overlook but critical: first, when entirely new evidence appears, reset the whole hypothesis tree rather than fine-tuning the old one; second, when the limit is reached without convergence, trigger human-in-the-loop (HITL) rather than letting the agent force-pick the hypothesis with the highest prior and ram into it.

Applicability

• Diagnostic tasks: Industrial fault localization, medical diagnosis, safety incident root-cause analysis. The root cause is unknown, evidence must be gathered step by step, and a single line that goes astray can be reset and restarted.
• Complex code debugging: Change one place, validate one place, let the failure information flow back to form new hypotheses, and repeatedly converge to the real bug.
• Judgments that require a strict evidence chain: Scenarios where every step's conclusion must be backed by reproducible evidence and the reasoning path must ultimately be explainable to a regulator.

Known failure modes

• Using it under a tight response budget: When latency matters more than iterative diagnosis or the cost of error is low, a single pass may be sufficient.
• Continuing after the configured limit: Reaching the cap without convergence calls for reset, task decomposition, better evidence, or human escalation, not an unbounded extra round.
• The Evaluator only looks for supporting evidence: This framing amplifies confirmation bias. Record supporting, falsifying, and missing evidence, and prefer tests that distinguish competing hypotheses.
• Fine-tuning the old tree when new evidence arrives: When entirely new evidence appears, the hypothesis tree should be reset and regenerated; patching the old tree will be dragged off course by the wrong prior.
• No circuit breaker: Every iteration must have a hard max_iterations cap plus a cost cap, or the cost explodes and the agent hangs.

Verification and metrics

• Convergence Rate: The share of cases that reach a verifiable conclusion within budget. Separate failures caused by task size, unavailable evidence, and weak hypotheses.
• Iterations to convergence: Track the distribution by task class. Sustained growth indicates low information gain per round or weaker hypothesis generation.
• Falsification Rate: Observe whether candidate hypotheses are actually eliminated by counterevidence. A long period with no falsification may indicate confirmation bias.
• HITL Trigger Rate: Interpret escalation by task risk, evidence gaps, and outcome. The key question is whether cases that required a human were escalated.

Reference implementation

Task arrives → Planner generates a hypothesis list (ranked by prior probability)
            Loop (limit max_iterations):
                Pick the unvalidated hypothesis with the highest prior
                Generator collects evidence (deterministic data source)
                Evaluator judges (emphasize falsification over confirmation):
                    confirmed   → converge, exit
                    falsified   → prune, continue to the next hypothesis
                    if all hypotheses are falsified → take the new evidence back to the Planner to regenerate
                Entirely new evidence arrives → reset the whole hypothesis tree
            Loop ends still without convergence → trigger HITL, attach the full hypothesis tree + evidence + iterations run
            Keep a trace on file throughout (compliance scenarios require long-term retention)
            

For production, evaluate model assignments for each responsibility, enforce a strict evidence schema, attach the hypothesis tree and evidence to human escalation, and retain the trace according to the applicable audit policy.

Illustrative scenario

Consider a plant alarm whose initial hypotheses focus on common mechanical faults. A field engineer then supplies a new fact: a remote configuration change occurred before the alarm. The system resets the hypothesis tree instead of forcing the fact into the old ranking, and it traces the changed control parameter. Hypothesis generation, evidence collection, and judgment use separate schemas; the evaluator searches for counterevidence; insufficient evidence escalates to a human; and restarting critical equipment always requires approval. Convergence must be evaluated through incident replay and field review.

Related patterns

• Parallel exploration (R3): A dual relationship. Parallel opens N lines at once along the spatial dimension and selects the best in one shot; iteration runs one line many times along the time dimension and converges step by step.
• Chain of thought (R1): The interior of each iteration round is a chain of thought; iteration strings multiple chains of thought together along the time dimension and revises them repeatedly.
• Complexity routing (R2): Using different model tiers for the three roles inside iteration is exactly the application of routing thinking inside a loop.
• Dual-mode architecture (R5): Also in the Loop column. Iteration is a single agent looping to validate against itself; dual-mode is two agents collaborating and dividing the work.

Design conclusion

The exit condition of iterative hypothesis testing is not "find a correct hypothesis" but "falsify all the wrong hypotheses"—it encodes centuries of the human scientific method into a probabilistic system, because the reliability of a probabilistic system can only be built through falsification.