Pattern Matrix/White Paper/P4

Problem

Enterprise input often mixes PDFs, charts, tables, scanned pages, and logs. Sending an entire report to a vision model can produce a magnitude error in a chart. Converting everything to plain OCR text removes the spatial relationships in bars, axes, and legends, leaving only a low-information statement such as “the chart shows market share.”

Multi-modal fusion engineers this problem, and the key is before the data enters the context: which segment should go through vision, which should be converted to text, which should be discarded. It handles the form of the input before the agent has "thought" about anything—if the form is wrong, then no matter how downstream selects, compacts, or explores, it is just continuing to burn resources on the wrong form.

Classification: Perception × Parallel

• Vertical axis · Perception: Fusion processes multi-channel input (PDF, image, audio, structured data) and synthesizes a unified representation. This is the most primitive "fusion" problem in perception, occurring before reasoning.
• Horizontal axis · Parallel: Each channel of data is handed simultaneously to its own specialist processor (PDF parser, OCR, table extractor, log sub-agent) for parallel conversion. The channels do not depend on each other and can run at the same time, after which a fuser gathers the results from the channels into unified prompt content. This is a classic fan-out / gather: N channels processed in parallel + one merge, neither a single sequential chain nor single-point routing.

Solution and mechanics

The core judgment in fusion is to choose the right carrier for each channel of data, then merge. The decision criterion is: keep it as an image when spatial information is the signal, convert it to markdown when structural information is the signal.

Vision cost depends on model, resolution, and detail settings, so a fixed cross-provider table is misleading. Benchmark representative documents with the models and settings actually used, recording token use, latency, extraction quality, and cache behavior. Long logs can follow a three-stage pipeline: filter irrelevant content with grep, awk, or jq; have a sub-agent produce a structured summary; then place only the JSON artifact and a pointer to the raw log in the main context.

Applicability

• Financial research report analysis: PDF tables + analyst text + market-size charts. None of the channels is complete on its own; they must be assembled before a judgment can be made.
• Insurance claims: Accident photos + claim report text + structured policy data combined.
• Operations incident response: Monitoring screenshots + long log fragments + configuration files. Long logs must go through the three-layer pipeline.
• Any scenario where input mixes structured and unstructured data: The decision criterion is whether the channels of information are complementary—if it is just a repeated expression of the same information, skip fusion and go straight to reasoning, since fusion has a cost.

Known failure modes

• Misreading figures: A chart-reading error can propagate through later turns. Cross-check extracted values against text, tables, or source data; route mismatches to human review; and preserve the figure source and extraction method with downstream claims.
• Image input cost grows inside a loop: Re-sending the full image at every step multiplies cost. Set limits for spend, tokens, elapsed time, and recursion depth; use thumbnails or cropped regions by default; and cache stable input.
• Sub-agent loops and lost findings: A truncated handoff can leave a downstream agent with incomplete instructions, causing repeated requests back to the sender. Store the complete finding in a state store and pass a pointer ID; require every sub-agent to declare a budget and termination condition before it starts. This illustrates the failure mechanism and is not presented as a verified enterprise incident with a specific loss figure.
• One-size-fits-all truncation threshold: Retention should follow each tool's information distribution. Shell output often needs both the command context near the beginning and the error stack near the end; file reads may need the beginning, the end, or relevant excerpts depending on file type. A fixed ratio can discard decisive evidence.

Verification and metrics

• Token share by modality: Track image, table, text, and log usage separately. A sudden change from the local baseline should trigger inspection of conversion paths, duplicate submission, and cache behavior.
• Budget enforcement in the agent loop: Record triggers and interceptions for spend, tokens, elapsed time, and recursion depth. A budget that is missing or enforced only after a call is a runtime defect.
• Secondary extraction rate on the chart path: Of the charts that go through vision, how many do a chart→CSV extraction before computing. Taking a vision-output number directly for reasoning is a high-risk signal.

Reference implementation

fuse(inputs):                       # inputs are multiple heterogeneous channels
                for each channel, dispatch by form:
                    TEXT   → straight into content
                    IMAGE  → base64 through vision (take this path only when spatial info is the signal)
                    TABLE  → convert to markdown (structure is the primary signal)
                    PDF    → extract TOC + locate key pages + key figures through vision + tables to md + drop decorative images
                    LOG    → bash pre-filter → sub-agent summary → structured JSON back to main context
                    AUDIO  → STT to text
                merge into unified content blocks, return + a trace (per channel: modality / tokens_out / method)
            health_check: watch token share by form, alert early on anomalous forms
            

Specialist tools (ocr / stt / pdf_extract / log_subagent / bash_filter) should be injected rather than wrapped, so that migrating environments only swaps the injected implementation (e.g., OCR from Textract to a local engine) without touching the fuser's main flow. Multi-modal dependencies (cv2, pyaudio, pdfplumber) should be lazy-imported inside the function body with an ImportError fallback, to avoid crashing on startup in a headless environment.

Illustrative scenario

Consider a research-report agent evolving through three implementations. The first sends the whole PDF to vision and misreads the magnitude of a chart. The second converts everything to OCR text and loses spatial relationships. The third extracts the table of contents, locates relevant pages, keeps key figures as images, converts tables to markdown, drops decorative material, and cross-checks chart values against text and tables. The lesson is to choose a representation for each modality and retain the provenance needed to verify it.

Related patterns

• Context triage (P1), semantic compaction (P2), progressive discovery (P3): Fusion sits at the very front of the four perception blades. If the form is wrong, then no matter how downstream Triage selects, Compaction compacts, or Discovery explores, it is just continuing to burn resources on the wrong form.
• Fan-out gather (C2, collaboration module): In the three-layer pipeline for long logs, handing the filtered data to a sub-agent for processing is fan-out gather expressed at the perception layer—multiple specialist processors process in parallel and converge at a central fuser.
• Layered memory (Memory module): The countermeasure for sub-agent infinite loops, the Memory Pointer Pattern, shares the same idea as the memory module—store the complete artifact in an external store and pass only the pointer in the context.

Design conclusion

Multi-modal fusion is not prompt engineering, it is data-form engineering—good fusion lets the agent see the appropriate little rather than the complete much.