Pattern Matrix/White Paper/P3

Problem

An agent faces a large legacy codebase, an unindexed contract collection, or a long incident log. It may not know the name of the relevant code or where the key evidence sits. Loading everything exceeds the window, and RAG may miss an implementation whose variable is merge_user_state but whose comments never use the business term “order.”

Progressive discovery starts with a broad scan, lets the evidence reshape the next query, and iterates until the signal is sufficient, no new evidence appears, or the budget is exhausted. It governs active exploration of an unknown space within the current session.

Classification: Perception × Loop

• Vertical axis · Perception: Progressive discovery is staged information acquisition that "takes a quick look before deciding whether to dig deeper", a form of progressive attention on the perception side. It decides what the agent looks at next; it is an input-side matter, not reasoning or memory.
• Horizontal axis · Loop: look → decide → look again → decide, where each round carries the previous round's findings into the next decision, terminating when "there is enough information" or "the cost cap is reached". It is iterative, unlike triage's single-pass routing or compression's linear cascade.

Solution and mechanics

A single discovery runs the three forage-focus-deepen stages, decreasing in breadth and increasing in depth:

Give the agent atomic tools such as grep, read, and glob so that each observation can reshape the next query. Configure hard limits for cycles, per-cycle budget, candidate count, and dependency depth using repository scale, task risk, and replay results. If the limit is reached without evidence, revise the query, switch retrieval methods, or hand off. A lightweight model can derive a compact set of initial keywords, which are then updated from new evidence.

Applicability

• Root-cause localization in an unfamiliar codebase: The original author has left, documentation is sparse, and no one knows which files a pipeline crosses. The agent must find an entry point and narrow the search along call relationships.
• Operations incident response: after receiving an alert, derive keywords from the metric, trim the log by time window, localize the fault source through the three stages, and give the on-call engineer an initial assessment report plus a "what to do first" recommendation.
• Contract clause risk scanning, research literature review: any scenario that fits the pattern of "receive a task → do not know where the relevant information is → explore to locate it".
• Privacy-sensitive codebase that remains directly searchable: grep + read can keep code inside the controlled file system instead of copying it into a vector store.

Known failure modes

• Forage keywords too broad: Translating “user login has gotten slow” only into ["login", "slow"] produces too many candidates. Add component names, error fields, or call entry points to narrow the query.
• Focus stage picks the wrong files: the scorer miscounts and ranks test files ahead of production files, the agent reads a pile of spec files, and the key services/auth.rb goes unread. Give the scorer business weights: production files > test files, recently modified > old code, core directories > peripheral directories.
• Deepen dead end: Following dependencies into a third-party library may produce no new signal. Set scope and depth limits unless current evidence points outside the repository.
• Discovery collides with RAG: The two paths may return overlapping or conflicting results. Rank sources by freshness, permission boundary, and index coverage, and preserve provenance for both.

Verification and metrics

• cycles_to_success: Track convergence by task class. A sustained increase against the local baseline calls for inspection of keyword derivation, tool availability, and candidate ranking.
• forage/focus budget ratio: Forage should stay broad and light while Focus carries most reading. Diagnose shifts together with candidate count and final hits.
• zero_signal_rate: The share of sessions that stop without useful evidence. When it rises, inspect index freshness, read permissions, scorer behavior, and task descriptions.

Reference implementation

discover(task, keywords):
                loop up to max_cycles:
                    Forage: run grep for each keyword → collect candidates → score by relevance → keep top_k
                    if cycle_tokens exceed budget → stop
                    Focus: pick focus_k to read in full, recording dependencies
                    Deepen: extract dependencies from what was read and follow within deepen_budget
                    if signal is sufficient → success, break
                    else → refine keywords from what was discovered, run another round
                emit one DiscoveryEvent per stage (phase / keyword / candidate count / files_read / tokens / wall_time)
            

The three atomic tools (grep / read / scorer) are injected rather than wrapped, so the same code can run against the local file system, an MCP server, or an external indexing engine, swapping only the injected implementation.

Illustrative scenario

Consider an e-commerce system whose order-confirmation emails occasionally include another customer's items. Semantic retrieval misses the implementation because the relevant code does not use the word “order.” The agent first runs grep "send.*confirm" to locate sending entry points, then reads the mailer and cache call chain, and finally follows Cache.get_user. It finds that the cache key lacks a tenant dimension. This failure depends on code structure, so traversing files and calls can be more effective than semantic recall alone.

Related patterns

• Context triage (P1): naturally coupled. Triage's P3 handle is the entry point that progressive discovery pulls from on demand — one selects passively, the other finds actively. The most important distinction is activeness — triage and compression are both passive (they act only when information arrives or the window fills), whereas discovery is the agent deciding for itself what to grep and what to read.
• Semantic compression (P2): the three divide labor along different time dimensions of tokens, with discovery governing unknown tokens (deciding which to find). Pushing exploration to a sub-agent so the main agent only gets back a summary is exactly compression's ObservationMasking idea replicated at the discovery layer.
• Procedural memory (Memory module): it is advisable to build this as a two-layer architecture of "discovery finds + memory persists". The final_files explored this time are retained across sessions, so the next similar task first checks memory and starts discovery only on a miss, and the agent's experience begins to accumulate.

Design conclusion

The strongest agent is not the one that knows the most but the one that knows where to look — progressive discovery is the echo of information foraging theory in the LLM era. The agent does not try to understand the entire codebase; it stops once it has found enough.