Pattern Matrix/White Paper/Perception
ADPS Agent Design Pattern White Paper · Module Overview
Perception Module · Control What Enters the Current Decision
Input boundaries, the four-stage perception pipeline, Context Contracts, formal patterns, and open questions.
Scope: This page describes the Perception subsystem as a whole. The P1–P4 specifications remain authoritative for pattern-level mechanics and verification.
Perception determines what an agent can see now and which signals may influence the next decision. Its inputs include user messages and images, but also code, logs, tool results, events, business state, project rules, and artifacts left by earlier work.
A larger model window does not remove the need for input governance. Too much material can bury the current goal. Sources without time and scope can carry stale facts into a new decision. Untrusted documents and tool descriptions can also introduce hostile instructions into an execution path. The perception subsystem turns those inputs into context with an explicit scope, provenance, budget, and trust level before reasoning begins.
Start from the decision
At the ADPS perception workshop on 13 August 2026, practitioners from security, developer productivity, open-source maintenance, geospatial systems, and game development reached a common engineering judgment: define the decision first, then work backward to the inputs it needs.
A Context Contract should answer at least six questions:
- What is the current goal, and who defines completion?
- Which materials must enter context, and which remain behind handles?
- Which state must be refreshed before use, and which historical material is still valid?
- Where did each input come from, and what are its scope and access rules?
- Which precedence rule applies when inputs conflict?
- What was unavailable to this run, and could that absence change the conclusion?
The perception trace should record selected, deferred, dropped, and unavailable inputs. This separates material the system never discovered from material it discovered and deliberately withheld.
A four-stage perception funnel
The Perception workshop distilled a four-stage pipeline from several production settings. It provides a useful runtime skeleton for the subsystem:
| Stage | Engineering question | Typical work | Output |
|---|---|---|---|
| Signal ingress | Can the system obtain the data? | Protocol adaptation, authorization, idempotency, timeout, retry | Provenanced raw signal |
| Preprocessing | Can downstream components use it consistently? | Cleaning, deduplication, normalization, validation, noise filtering | Normalized record |
| Semantic aggregation | How do isolated signals acquire business meaning? | Entity linking, context completion, cross-source alignment, source pointers | Structured fact packet |
| Event assessment | Should these facts trigger downstream work? | Classification, severity, confidence, rule checks | Trigger recommendation and rationale |
The final stage emits a category, severity, and trigger recommendation. Reasoning, action, and governance decide whether to block a release, freeze an account, or change production state. When perception performs the business action itself, input rules and business policy become difficult to test and evolve independently.
Keep triggering separate from execution topology
Signals can arrive through callbacks, scheduled polling, streams, or direct requests. These mechanisms answer when new information enters the system. Chain, route, parallel, orchestrate, loop, and hierarchy still describe how control unfolds after the work starts.
| Ingress mechanism | Suitable signals | Main cost |
|---|---|---|
| Callback event | Code commits, ticket changes, alerts | Stable event contracts and deduplication keys |
| Scheduled polling | Compliance scans, inventory, state reconciliation | Higher latency and potentially expensive scans |
| Stream processing | High-volume logs, telemetry, message streams | More complex operations, replay, and debugging |
| Multi-source corroboration | High-risk threats, fault isolation, end-to-end audit | Semantic alignment and conflict handling |
Event-driven processing therefore remains an ingress and triggering mechanism. It does not become a new perception pattern or automatically constitute a seventh execution topology.
Source and modality are separate dimensions
The workshop refined the boundary of P4. Image, text, audio, and table are different modalities. Host logs, network traffic, source-code scans, and historical tickets are different sources. Multiple sources may all be text, while several modalities may come from one PDF.
A production observation should record both:
observation_id: obs_01K2...
source:
system: ci
channel: webhook
scope: repo://payments
modality: text
captured_at: 2026-08-13T12:10:00Z
valid_at: 2026-08-13T12:09:58Z
content_ref: artifact://ci/run-8842/log
transform:
method: error-normalizer-v3
parent: raw://ci/run-8842
trust:
integrity: verified
confidence: 0.94
Source fields drive authorization, freshness, and corroboration. Modality fields select parsers and representations. P4 retains its catalog name, Multi-Modal Fusion, while the specification now includes multi-source alignment. ADPS will revisit a broader name after more independent cases are available.
How the four specifications divide the work
| Pattern | Responsibility | Boundary |
|---|---|---|
| P1 Context Triage | Load, compact, defer, or discard known candidates under a context budget | Goal, identity, safety constraints, and current errors cannot be displaced by ordinary material |
| P2 Semantic Compaction | Reduce history already in the window while retaining decision evidence | Errors, rejected approaches, business values, and source pointers need explicit protection |
| P3 Progressive Discovery | Move from broad search to focused reading when the location of evidence is unknown | Each cycle updates the query and stops on evidence, budget, or no new signal |
| P4 Multi-Modal Fusion | Parse, align, and corroborate different modalities and sources | Preserve original evidence, transformations, and conflicts; the fused result is not a new source of truth |
The four patterns can appear in one pipeline. P4 normalizes representations and sources. P1 chooses the current working set. P3 explores when evidence is missing. P2 reclaims context after earlier material has been consumed.
Three recurring failure modes
More input produces worse decisions. Sending every available source to the model increases noise, conflict, and attack surface. The decision goal should constrain perception scope, and replay tests should measure both missed and false signals.
Perception and policy are fused. A parser that cleans a signal and immediately takes a business action cannot be evaluated independently. The stable interface should return facts, labels, confidence, and evidence. Action requires a separate admission path.
The perception layer is never recalibrated. Businesses, system architecture, models, and sources change. Perception needs its own evaluation set for selection omissions, lossy compaction, zero-signal runs, trigger latency, and cross-source conflicts.
Security boundary
Every external source adds a data and instruction entrance. Tool descriptions, web pages, documents, images, and logs can all carry untrusted content. The perception layer needs source allowlists, scope checks, a separation between data and instructions, sensitive-field handling, and a complete trace. Tool admission, sandboxing, and approval still control execution authority.
Perception may label an input as suspicious and reduce its trust level. A model assessment alone cannot expand the agent's permissions.
Why AI-driven software engineering starts with perception
For a coding agent, the repository is the primary environment. Requirements, architecture decisions, source code, tests, build scripts, and runtime logs that remain in chat or individual memory are effectively invisible. A greenfield project can establish a full specification chain from the beginning. A brownfield project usually needs local discovery, dependency reconstruction, targeted documentation and tests, followed by independently acceptable vertical slices.
This discussion crosses memory, action, reflection, and governance, so ADPS has published it separately as AI-Driven Software Engineering. The perception overview retains only the direct context requirements.
Anthropic's September 2025 article on context engineering for agents treats context as a finite resource that must be curated on every inference turn. OpenAI's February 2026 account of harness engineering describes putting versioned repository artifacts, enforceable constraints, and feedback loops into an environment agents can inspect. Both accounts support the workshop's practical conclusion: model capability becomes useful only through an environment the agent can read and validate.
Open questions
- Should P4 eventually be renamed to cover both multi-source and multi-modal fusion?
- What is the minimum stable fact contract between perception and reasoning?
- How should prompt injection, source poisoning, and cross-tenant leakage be evaluated together?
- How can polling, streaming, and callbacks share deduplication, freshness, and replay semantics?
- How can architectural constraints in a brownfield repository become discoverable, enforceable, and continuously calibrated agent assets?
Workshop record
This overview incorporates the first ADPS Perception workshop, held on 13 August 2026. Jia Huang chaired the discussion. The public core workshop guests were Dong Zhang, Xianglong Huang, Qingfeng Li, and Cheng Huang.
Read the workshop record · All workshops · White Paper contributors