Pattern Matrix/White Paper/P4

Problem

Enterprise input often mixes PDFs, charts, tables, scanned pages, and logs. Sending an entire report to a vision model can produce a magnitude error in a chart. Converting everything to plain OCR text removes the spatial relationships in bars, axes, and legends, leaving only a low-information statement such as “the chart shows market share.”

Multi-modal fusion decides how each segment should enter the context: vision input, extracted text, structured data, or no input at all. This decision happens before the agent reasons over the material. A poor representation cannot be repaired by better selection, compaction, or exploration downstream.

Classification: Perception × Parallel

• Vertical axis · Perception: Fusion converts PDFs, images, audio, logs, and structured data into representations that downstream checks can consume while preserving source references.
• Horizontal axis · Parallel: Independent modality processors can run concurrently, after which a fuser joins their typed artifacts. The gather step records how each claim maps back to its source.

Solution and mechanics

Choose each representation from the signal the next check needs. Preserve images for spatial evidence, use Markdown or JSON for document structure, keep timestamped transcripts with audio references for speech, and filter logs into typed events. Fusion then joins those artifacts while retaining source references.

Vision cost depends on model, resolution, and detail settings, so a fixed cross-provider table is misleading. Benchmark representative documents with the models and settings actually used, recording token use, latency, extraction quality, and cache behavior. Long logs can follow a three-stage pipeline: filter irrelevant content with grep, awk, or jq; have a sub-agent produce a structured summary; then place only the JSON artifact and a pointer to the raw log in the main context.

Applicability

• Financial research report analysis: PDF tables + analyst text + market-size charts. None of the channels is complete on its own; they must be assembled before a judgment can be made.
• Insurance claims: Accident photos + claim report text + structured policy data combined.
• Operations incident response: Monitoring screenshots + long log fragments + configuration files. Long logs must go through the three-layer pipeline.
• Mixed structured and unstructured input: Use fusion when different channels contribute distinct evidence. If they repeat the same information, select the most reliable representation instead of processing every channel.

Known failure modes

• Misreading figures: A chart-reading error can propagate through later turns. Cross-check extracted values against text, tables, or source data; route mismatches to human review; and preserve the figure source and extraction method with downstream claims.
• Image input cost grows inside a loop: Re-sending the full image at every step multiplies cost. Set limits for spend, tokens, elapsed time, and recursion depth; use thumbnails or cropped regions by default; and cache stable input.
• Sub-agent loops and lost findings: A truncated handoff can leave a downstream agent with incomplete instructions, causing repeated requests back to the sender. Store the complete finding in a state store and pass a pointer ID; require every sub-agent to declare a budget and termination condition before it starts. This illustrates the failure mechanism and is not presented as a verified enterprise incident with a specific loss figure.
• One-size-fits-all truncation threshold: Retention should follow each tool's information distribution. Shell output often needs both the command context near the beginning and the error stack near the end; file reads may need the beginning, the end, or relevant excerpts depending on file type. A fixed ratio can discard decisive evidence.

Verification and metrics

• Token share by modality: Track image, table, text, and log usage separately. A sudden change from the local baseline should trigger inspection of conversion paths, duplicate submission, and cache behavior.
• Budget enforcement in the agent loop: Record triggers and interceptions for spend, tokens, elapsed time, and recursion depth. A budget that is missing or enforced only after a call is a runtime defect.
• Secondary extraction rate on the chart path: Of the charts that go through vision, how many do a chart→CSV extraction before computing. Taking a vision-output number directly for reasoning is a high-risk signal.

Reference implementation

fuse(inputs):                       # inputs are multiple heterogeneous channels
                for each channel, dispatch by form:
                    TEXT   → straight into content
                    IMAGE  → base64 through vision (take this path only when spatial info is the signal)
                    TABLE  → convert to markdown (structure is the primary signal)
                    PDF    → extract TOC + locate key pages + key figures through vision + tables to md + drop decorative images
                    LOG    → bash pre-filter → sub-agent summary → structured JSON back to main context
                    AUDIO  → STT to text
                merge into unified content blocks, return + a trace (per channel: modality / tokens_out / method)
            health_check: watch token share by form, alert early on anomalous forms
            

Define small interfaces for OCR, speech-to-text, PDF extraction, log filtering, and specialist agents. Environment-specific implementations can then change without rewriting the fusion flow. Load optional multimedia libraries only on the path that needs them, and return a typed capability error when a dependency is unavailable.

Illustrative scenario

Consider a research-report agent evolving through three implementations. The first sends the whole PDF to vision and misreads the magnitude of a chart. The second converts everything to OCR text and loses spatial relationships. The third extracts the table of contents, locates relevant pages, keeps key figures as images, converts tables to markdown, drops decorative material, and cross-checks chart values against text and tables. The lesson is to choose a representation for each modality and retain the provenance needed to verify it.

Related patterns

• Context Triage (P1), Semantic Compaction (P2), Progressive Discovery (P3): Fusion determines the representation. Triage selects among represented sources, compaction reduces admitted material, and discovery acquires missing evidence.
• Fan-Out/Gather (C2): Independent modality processors may use C2, with one gather step producing the fused artifact and provenance map.
• Layered Retention (M1): Large source artifacts can remain in external storage while the context carries a versioned pointer and a bounded extraction. Cross-session retention still requires memory admission.

Design conclusion

Measure fusion through extraction error, missing cross-modal links, context cost, and acceptance failures by modality. The fused artifact should preserve the evidence required by downstream checks without copying every source in full.

Edit source · Published source · Review process