Pattern Matrix/White Paper/P3

P3 · Progressive Discovery

What problem it solves

An agent faces a legacy codebase of 15,000 files, a contract it has never seen, a long incident log. It does not know what the relevant code is called and does not know which page holds the key evidence. Feeding the whole codebase in blows the context window, and RAG often fails to recall (the relevant code's variable name is merge_user_state, the comments contain no "order", so semantic retrieval does not match).

Progressive discovery turns this into an engineering process: use a broad scan to gather a batch of candidates, let what is seen reshape what should be looked at next, and iterate until the signal is strong enough. It governs active exploration of an unknown space within the current session, and its core constraint is cost — one round of exploration is roughly 18K tokens and about 45 seconds, far cheaper than indexing the entire codebase into the prompt.

Why the coordinate is "Perception × Loop"

• Vertical axis · Perception: Progressive discovery is staged information acquisition that "takes a quick look before deciding whether to dig deeper", a form of progressive attention on the perception side. It decides what the agent looks at next; it is an input-side matter, not reasoning or memory.
• Horizontal axis · Loop: look → decide → look again → decide, where each round carries the previous round's findings into the next decision, terminating when "there is enough information" or "the cost cap is reached". It is iterative, unlike triage's single-pass routing or compression's linear cascade.

Core mechanism

A single discovery runs the three forage-focus-deepen stages, decreasing in breadth and increasing in depth:

Three mechanisms decide whether discovery succeeds. First, give the agent three atomic tools — grep, read, glob — not a high-level wrapper like search_codebase(). Only atomic tools support the feedback loop where "what is seen reshapes the next step". Second, hard caps. Cap the number of loops (usually 3, or 2 for incident scenarios) and cap the tokens per loop (about 20K); if three rounds still find nothing, the keywords are wrong or a human is needed, and continuing the loop yields no further return. Third, keyword derivation is the lever. The first round's keywords decide the hit quality of the forage stage, and a sloppy first round cannot be salvaged by more loops afterward, so it is better to first use a lightweight model to translate the task description into 5-8 precise keywords before running grep. When one round does not produce enough signal, return to forage with sharper keywords rather than guessing blindly.

Production scenarios it fits

• Root-cause localization of a bug in an unfamiliar codebase: the original author has left, the documentation is sparse, and no one knows which files a given pipeline runs through. Anthropic disclosed that 78% of Claude Code sessions in Q1 2026 involved multi-file edits, which means agents increasingly face unfamiliar areas of code.
• Operations incident response: after receiving an alert, derive keywords from the metric, trim the log by time window, localize the fault source through the three stages, and give the on-call engineer an initial assessment report plus a "what to do first" recommendation.
• Contract clause risk scanning, research literature review: any scenario that fits the pattern of "receive a task → do not know where the relevant information is → explore to locate it".
• Privacy-sensitive and not a large codebase (< 50K files): grep + read keep the code on the file system and out of a vector store, a safer choice than RAG.

Where it tends to go wrong

• Forage keywords too broad: the task "user login has gotten slow" is written as ["login", "slow"], returns 5,000 candidates, and the forage-stage tokens blow up immediately. Make the keywords narrow and precise; ["LoginController", "auth_timeout", "session_create"] is far more exact.
• Focus stage picks the wrong files: the scorer miscounts and ranks test files ahead of production files, the agent reads a pile of spec files, and the key services/auth.rb goes unread. Give the scorer business weights: production files > test files, recently modified > old code, core directories > peripheral directories.
• Deepen dead end: following dependencies down into a third-party library's source, reading hundreds of lines without obtaining any signal. Set boundaries for Deepen: do not follow into third-party libraries (unless the bug report names one), do not follow dependencies beyond 2 hops.
• Discovery collides with RAG: the same agent runs Discovery and RAG at once, the two sets of results overlap or conflict, and the agent does not know which to trust. Pick one, do not mix — Discovery for < 50K files, persistent indexing for > 400K.

Key metrics

• Median cycles to find the answer, cycles_to_success_p50 (healthy line = 1): one three-stage round handles 80% of tasks. A p50 rising to 2-3 indicates declining keyword-derivation ability, possibly from a model upgrade, a changed business scenario, or longer user descriptions; tune the moment it rises.
• forage/focus token ratio (healthy band 0.3-0.5): forage should be lighter than focus. Above 0.7 means keywords are too broad (too many irrelevant candidates pulled in); below 0.2 means keywords are too narrow (not enough candidates gathered).
• Zero-signal rate, zero_signal_rate (healthy line < 5%): the share of sessions that finish all three stages without finding any signal. A rise almost always means an atomic tool is broken (stale grep index, misconfigured read permissions, scorer bug), and it alarms earlier than the task failure rate.

Minimal implementation skeleton

discover(task, keywords):
                loop up to max_cycles (default 3):
                    Forage: run grep for each keyword → collect candidates → score by relevance to task → keep top-30
                    if cycle_tokens exceed budget → stop
                    Focus: pick top-8 to read in full, recording dependencies
                    Deepen: extract dependencies from what was read (import / function references), follow at most 5
                    if signal is sufficient → success, break
                    else → refine keywords from what was discovered, run another round
                emit one DiscoveryEvent per stage (phase / keyword / candidate count / files_read / tokens / wall_time)
            

The three atomic tools (grep / read / scorer) are injected rather than wrapped, so the same code can run against the local file system, an MCP server, or an external indexing engine, swapping only the injected implementation.

Enterprise deployment example

A bug at an e-commerce customer: order confirmation emails occasionally mixed in order items belonging to other customers. The development team faced a 15,000-file legacy monolith, had tried feeding the whole repository to the agent (800K tokens blew the window), had tried RAG (the relevant code had no "order" so recall failed), and finally let the agent explore on its own with grep + read. Forage: grep "send.*confirm" returned 30 candidates, and mailers/order_confirmed.rb was picked out by file name. Focus: five files read in full, surfacing the call chain MailerWorker → Cache.get_user → render. Deepen: following Cache.get_user, the cache key was found to use order_id but not customer_id, so different customers' order ids colliding in the same bucket hit the wrong cache — the bug was found. One round, about 18K tokens, about 75 seconds. The retrospective conclusion was that this bug had nothing to do with semantics and everything to do with code structure, and only grep + read + follow imports, which touch the structure directly, could find it.

Relationship to other patterns

• Context triage (P1): naturally coupled. Triage's P3 handle is the entry point that progressive discovery pulls from on demand — one selects passively, the other finds actively. The most important distinction is activeness — triage and compression are both passive (they act only when information arrives or the window fills), whereas discovery is the agent deciding for itself what to grep and what to read.
• Semantic compression (P2): the three divide labor along different time dimensions of tokens, with discovery governing unknown tokens (deciding which to find). Pushing exploration to a sub-agent so the main agent only gets back a summary is exactly compression's ObservationMasking idea replicated at the discovery layer.
• Procedural memory (Memory module): it is advisable to build this as a two-layer architecture of "discovery finds + memory persists". The final_files explored this time are retained across sessions, so the next similar task first checks memory and starts discovery only on a miss, and the agent's experience begins to accumulate.

What this pattern teaches

The strongest agent is not the one that knows the most but the one that knows where to look — progressive discovery is the echo of information foraging theory in the LLM era. The agent does not try to understand the entire codebase; it stops once it has found enough.

This page is part of the ADPS pattern white papers. Back to the pattern matrix, or see the runnable code catalog.