Pattern Matrix/White Paper/R4

R4 · Iterative Hypothesis Testing

What problem it solves

Some tasks cannot be answered in a single pass of reasoning: the root cause is unknown, the evidence must be gathered step by step, and the first judgment is often wrong. Diagnostic tasks are the typical case—when a fault alarm fires, the "most likely cause" that the model ranks by prior may be badly miscalibrated, with the true root cause ranked far down or not even considered at first. If the model reasons once and then commits, it walks further and further in the wrong direction; if it simply retries, the retry itself does not solve the problem, because the issue is not that "the result is unstable" but that "the understanding of the world is still wrong."

Iterative hypothesis testing replaces single-pass reasoning with a loop of "hypothesize → validate → revise → hypothesize again." The fundamental difference from ordinary retry is this: retry assumes that the retry itself can solve the problem, while iterative hypothesis testing assumes that each loop must update the understanding of the world. Its design center is also not "retry until success," but converging through falsification to the single hypothesis that has not been refuted. It is the dual of parallel exploration (R3): parallel opens N lines at once along the spatial dimension, while iteration runs one line many times along the time dimension.

Why the coordinate is "Reasoning × Loop"

• Vertical axis · Reasoning: It follows empirical, scientific-style reasoning of "hypothesize → validate → revise → hypothesize," not a single deduction. Each round actively revises its own belief rather than waiting for the result to stabilize.
• Horizontal axis · Loop: Multiple iterations until the evidence converges or a limit is reached form a natural loop structure. The dual-mode architecture (R5) in the same column is also in the Loop column but has a different emphasis—iteration is a single agent looping with itself to validate hypotheses, while dual-mode is two agents collaborating to split the handling of "talking" and "thinking."

Core mechanism

A mature iterative hypothesis-testing setup often uses a three-agent division of labor that maps to a strict loop:

1. Hypothesis generation (Planner): List N candidate hypotheses based on the symptoms and historical cases, ranked by prior probability. This step uses the most expensive model, because generating the "right hypothesis list" determines the direction of the entire trajectory.
2. Evidence collection (Generator): Given a hypothesis, decide which tools to run to validate it—query a metric, retrieve a log, read a sensor. This is mostly tool calls, and a cheap model is enough. The evidence must come from deterministic, reproducible data sources; "the model looked at it itself and felt it was OK" does not count as evidence.
3. Judgment (Evaluator): Combine the evidence to decide whether the hypothesis is confirmed, falsified, or insufficiently supported. This step uses a mid-tier model, and the system prompt should emphasize falsification over confirmation—"your job is to find strong counter-evidence." This framing measurably improves accuracy.

The loop rules are: falsified goes back to the generation stage, confirmed exits, and insufficient evidence goes back to gather more. Two engineering practices are easy to overlook but critical: first, when entirely new evidence appears, reset the whole hypothesis tree rather than fine-tuning the old one; second, when the limit is reached without convergence, trigger human-in-the-loop (HITL) rather than letting the agent force-pick the hypothesis with the highest prior and ram into it.

Where it fits in production

• Diagnostic tasks: Industrial fault localization, medical diagnosis, safety incident root-cause analysis. The root cause is unknown, evidence must be gathered step by step, and a single line that goes astray can be reset and restarted.
• Complex code debugging: Change one place, validate one place, let the failure information flow back to form new hypotheses, and repeatedly converge to the real bug.
• Judgments that require a strict evidence chain: Scenarios where every step's conclusion must be backed by reproducible evidence and the reasoning path must ultimately be explainable to a regulator.

Where it goes wrong

• Forcing it through under too tight a time budget: When the time budget is under 30 seconds, or the cost of error is low, do not use iteration—adding iteration to a task that one pass of reasoning could handle is over-engineering.
• Forcing it past 3 rounds without convergence: If it has not converged after 3 iterations, the task itself is usually wrong (the symptom description is inaccurate, or it is simply not a problem in this system), and it should be reset or escalated to a human, not burn a 4th and 5th round.
• The Evaluator looks for supporting evidence: "Find supporting evidence" falls into confirmation bias and is more than 30% less accurate than "find falsifying evidence."
• Fine-tuning the old tree when new evidence arrives: When entirely new evidence appears, the hypothesis tree should be reset and regenerated; patching the old tree will be dragged off course by the wrong prior.
• No circuit breaker: Every iteration must have a hard max_iterations cap plus a cost cap, or the cost explodes and the agent hangs.

Key metrics

• Convergence Rate (healthy range >80%): The share of cases that converge automatically before reaching the limit. Below 60% means the agent often reaches the limit without a conclusion—either the task is too complex and should be decomposed, or evidence collection is inadequate. This is the most comprehensive metric.
• Average iterations to convergence (healthy range 2-4): How many rounds successful cases take on average to converge. Below 2 means the task is too simple to need iteration; above 5 means each round is inefficient and hypothesis quality is poor.
• Falsification Rate (healthy range 30-60%): The share of hypotheses falsified each round. Below 20% means the agent is in confirmation bias and not actively falsifying; above 80% means most generated hypotheses are wrong.
• HITL Trigger Rate (healthy range 5-15%): The share of cases that escalate to a human. Below 2% means the agent is overconfident and commits even when it should not; above 30% means the agent is too weak and needs a human to backstop everything.

Minimal implementation skeleton

Task arrives → Planner generates a hypothesis list (ranked by prior probability)
            Loop (limit max_iterations):
                Pick the unvalidated hypothesis with the highest prior
                Generator collects evidence (deterministic data source)
                Evaluator judges (emphasize falsification over confirmation):
                    confirmed   → converge, exit
                    falsified   → prune, continue to the next hypothesis
                    if all hypotheses are falsified → take the new evidence back to the Planner to regenerate
                Entirely new evidence arrives → reset the whole hypothesis tree
            Loop ends still without convergence → trigger HITL, attach the full hypothesis tree + evidence + iterations run
            Keep a trace on file throughout (compliance scenarios require long-term retention)
            

Four points for landing it: assign the three models by routing (the expensive model generates hypotheses, the cheap model collects evidence, the mid-tier model judges); make the evidence schema strict to prevent the model from inventing evidence; let HITL escalation carry the full hypothesis tree and evidence, not a simple notification; keep the trace on file for the long term for audit.

Enterprise landing example

A mid-sized chemical plant's industrial fault-diagnosis agent assists front-line engineers in locating the root cause when an alarm fires. In the early hours of the pilot's third week, a polyethylene production line raised an alarm—the reactor temperature was rising abnormally. The agent pushed 5 candidate hypotheses ranked by likelihood, and the engineer validated them one by one: cooling pump failure (flow looked normal, eliminated), sensor drift (compared against the redundant sensor and they agreed, eliminated), process recipe anomaly (the feed log looked normal, eliminated). When the fourth hypothesis required waiting for lab analysis and the line had already been down for an hour, the engineer manually fed in a new fact—the PID control log showed a remote login that had modified settings in the early hours. The agent immediately rebuilt the hypothesis tree from scratch, drilled down to find that a temperature-control parameter had been changed, which would cause the temperature to oscillate upward and matched the current state, and finally located the cause: another shift had run a parameter-optimization test and forgotten to roll it back.

Two observations afterward were key. First, the true root cause had only a 2% prior in the first round—the model's prior was badly miscalibrated, and it was dug out only by repeated iteration plus a human supplying a fact. Second, after the new fact was fed in, the agent reset and rebuilt the entire tree rather than fine-tuning it. The team rewrote the system on this basis, adding a three-layer structure (generate hypotheses, collect evidence, judge) and making six key decisions: a three-layer hypothesis tree that supports drill-down; a low HITL threshold (escalate to a human when confidence is insufficient after 3 rounds, and require human confirmation whenever critical equipment is restarted); reset on new evidence rather than compaction; a strict evidence schema; an Evaluator that emphasizes falsification; and a long-term trace on file for every run. After the changes, the convergence rate rose from 50% to 82%, and the average fault-localization time was compressed from 3.2 hours to 1.1 hours.

Relationship to other patterns

• Parallel exploration (R3): A dual relationship. Parallel opens N lines at once along the spatial dimension and selects the best in one shot; iteration runs one line many times along the time dimension and converges step by step.
• Chain of thought (R1): The interior of each iteration round is a chain of thought; iteration strings multiple chains of thought together along the time dimension and revises them repeatedly.
• Complexity routing (R2): Using different model tiers for the three roles inside iteration is exactly the application of routing thinking inside a loop.
• Dual-mode architecture (R5): Also in the Loop column. Iteration is a single agent looping to validate against itself; dual-mode is two agents collaborating and dividing the work.

What this pattern teaches

The exit condition of iterative hypothesis testing is not "find a correct hypothesis" but "falsify all the wrong hypotheses"—it encodes centuries of the human scientific method into a probabilistic system, because the reliability of a probabilistic system can only be built through falsification.

This page is part of the ADPS pattern white papers. Back to the pattern matrix, or see the runnable code catalog.