Pattern Matrix/White Paper/M4

Problem

Many agent systems preserve progress but discard the details of failure when a session ends. A later task can then repeat the same error. For example, an agent may write test-environment settings into production configuration and later make the same class of mistake in another project because the conditions and remediation were never retained.

A failure journal makes these events retrievable across tasks. The difficult part is the write path: an error is an observed fact, while root cause and lesson are judgments. The model that just failed may also misdiagnose the failure, so diagnosis and long-term publication cannot be one automatic step.

Classification: Memory × Loop

• Vertical axis · Memory: It accumulates "failure" as persistent learning material across calls, a library of negative examples within long-term memory—one side records successes (procedural memory), the other records failures (failure journals).
• Horizontal axis · Loop: Write a failure once, read it once on the next task, then write again. This is a cross-call outer loop. Failure signals circulate between tasks, making the agent more seasoned with each pass.

Solution and mechanics

A failure journal has three record layers and one recall path:

1. Failure Event preserves immutable facts: Store input references, tool calls, errors, system version, permission scope, and the original trace. Later diagnoses link to the event rather than rewriting it.
2. Candidate Diagnosis preserves a testable explanation: Store the hypothesized cause, proposed fix, author, confidence information, and applicability. Automatically generated diagnoses default to candidate and stay out of active recall.
3. Verified Lesson publishes reusable experience: After reproduction, tests, policy checks, or human review, publish a prevention rule, recall condition, valid versions, and invalidation criteria. A later lesson may supersede or revoke it.
4. Recall at task boundaries: Before task start, capability entry, or a high-risk tool call, filter by tenant, scope, system version, and failure signature, then rerank by risk and relevance. Only accepted and currently applicable lessons are injected.
5. Write back evidence of use: Record whether the lesson was retrieved, adopted, and associated with avoiding the repeated failure. A counterexample creates a new candidate diagnosis rather than mutating the old event.

Events, diagnoses, and lessons have different retention policies. Events follow audit requirements, weak candidates may expire quickly, and verified lessons require version and applicability management.

Applicability

• Production-grade, long-running agents: Agents that run for months and repeatedly handle the same class of tasks. Which failures to record and which to pull up when a task arrives is where its core value lies.
• Multi-tenant SaaS agents: High-risk failures such as cross-tenant misrouting require special handling—never evicted, force-recalled at the start of every task, alarm triggered on several consecutive records. The task signature uses a two-factor "tenant plus intent" key, so one tenant's failure should not cause another tenant to receive irrelevant reminders.
• Agents performing high-risk, irreversible operations: Scenarios where the cost of an error is large and the same mistake cannot be made twice. DevOps, finance, contract processing.

Known failure modes

• Forcing it onto one-off tasks with no repeated execution: For pure demos, prototypes, and one-off scripts, discarding failures is fine. The value of a failure journal lies in repeated tasks; without repetition there is no recall payoff.
• Free text only, no classification: The first version of a failure log is often free text with three fields, "task / error / timestamp," from which the agent can learn nothing. The schema must be enforced (enums plus constrained fields) so failures can be clustered, counted, and queried.
• Treating the model's explanation as fact: The same agent generates a root cause and prevention rule immediately after failing, then writes it to long-term memory. A false diagnosis becomes a stable reason to avoid the correct path.
• Promoting every minor error into a lesson: Recall noise grows with the archive. Only candidates with sufficient reuse value or risk enter verification.
• Using semantic similarity without condition filters: Similar text does not imply the same tenant, tool version, permission, environment, or failure stage. Filter structured conditions before semantic retrieval.
• Applying old lessons across versions: The API, schema, or process changes while the historical fix remains active. Lessons need applicable versions and invalidation criteria.
• Lumping high-risk and ordinary failures together: A cross-tenant data leak and an API timeout require different treatment. High-risk failures need independent alerting, policy-controlled retention, and forced recall where applicable.

Verification and metrics

• Repeat failure rate: The share of similar tasks that repeat the same failure signature. Compare versions and cohorts rather than database cleanliness alone.
• Candidate promotion rate and rejection reasons: Measure which diagnoses become verified lessons and why others fail due to false causes, broad scope, or weak evidence.
• Recall precision and miss rate: Test whether a published lesson appears for applicable tasks and stays out of unrelated ones.
• Lesson adoption and avoided-failure rate: Record whether the agent used the lesson and whether the same failure signature disappeared, compared with replay without injection.
• Stale-lesson use rate: Measure lessons that still influence decisions after supersession, revocation, or version expiry.
• High-risk failure count: Monitor tenant-boundary and other safety events independently. There is no acceptable background rate for a boundary violation.

Reference implementation

FailureEvent:
                event_id / task_signature / category / stage / error
                input_refs[] / tool_calls[] / system_version / trace_ref / occurred_at

            CandidateDiagnosis:
                diagnosis_id / event_id / hypothesis / proposed_fix
                proposed_by / confidence / applicability / status(candidate|rejected|verified)

            VerifiedLesson:
                lesson_id / diagnosis_id / prevention_rule / recall_conditions[]
                valid_from / valid_to / applies_to_versions[] / supersedes / review_evidence[]

            recall(task):
                filter accepted lessons by tenant + scope + version + risk
                retrieve by failure signature and semantic relevance
                return lessons + source events + recall trace
            

A production implementation keeps FailureEvent append-only and manages candidate diagnoses and published lessons in separate stores or lifecycle states. Review evidence, publisher, and supersession all enter the audit trail.

Illustrative scenario

Consider a multi-tenant SaaS support agent that repeatedly encounters tool timeouts, permission failures, and tenant-boundary mistakes. The tool trace and business version first become a FailureEvent. The model's “expired token” explanation enters Candidate Diagnosis only. After reproduction, the system publishes “refresh credentials and revalidate tenant_id” as a Verified Lesson with an applicable version range. Retrieval filters by tenant, tool version, and intent before ranking. Unknown categories enter needs_review. Evaluation covers repeat failures, false recalls, and stale-lesson use.

Related patterns

• Procedural memory (M5): A twin relationship; together they form the agent's experience base. The failure journal records "I did this wrong before, don't do it this way again," while procedural memory records "I did this right before, follow this next time." One side records failures, the other records successes.
• Progress tracking (M3): Also in the Loop column. Progress tracking loops over "the steps done right," the failure journal loops over "the pits stepped in." The two look symmetric, but in industry their treatment is completely asymmetric—everyone does the former, few do the latter seriously.
• Layered retention (M1): The failure journal is a dedicated partition within long-term memory, with retention tiers chosen by risk, access, and policy.
• Self-Heal Loop (Action module): The self-heal loop only handles failures within the current task and does not persist across tasks. The failure journal goes one step further—it persists the current failure signal and recalls it on the next similar task.
• Versioned Memory (candidate): Failure events remain immutable while diagnoses and lessons evolve through versions, supersession, and revocation.

Design conclusion

A failure journal derives its value from publication discipline. Failure events supply evidence, candidate diagnoses remain falsifiable, and only verified lessons travel across tasks. Without that separation, a false lesson propagates more reliably than forgetting.