Pattern Matrix/White Paper/M4

M4 · Failure Journals

What problem it solves

Most agent systems take progress tracking seriously, yet few take failure records seriously. Failures are quietly discarded, wiped clean the moment a session ends, and the next time a similar situation comes around the agent falls into the same pit from scratch. A typical scenario: after correctly fixing the main task, the agent's attention drops and it writes the test-environment configuration into the production configuration—two weeks later, on another project, it does almost exactly the same thing. This is not occasional carelessness; it is a fixed pattern, and the agent has no memory of "I have stumbled on this before."

A failure journal turns failures into an archive that can be reviewed repeatedly. Its purpose is not to show errors to operations; it is to let the agent accumulate professional experience of its own—so that next time a similar situation arises, the model proactively retrieves the reminder "I have taken this kind of fall." Erasing failures is the same as erasing evidence, and without evidence the model cannot adapt.

Why the coordinate is "Memory × Loop"

• Vertical axis · Memory: It accumulates "failure" as persistent learning material across calls, a library of negative examples within long-term memory—one side records successes (procedural memory), the other records failures (failure journals).
• Horizontal axis · Loop: Write a failure once, read it once on the next task, then write again. This is a cross-call outer loop. Failure signals circulate between tasks, making the agent more seasoned with each pass.

Core mechanism

A complete failure journal system does four things, forming a closed loop:

1. Detection: Decide whether this task counts as a "failure." This includes task-level failures, as well as failures of information retrieval itself (such as RAG index lag, missed retrieval, or false retrieval—new failure types introduced in the LLM era).
2. Classification: Assign the failure to a category. Industrial implementations distinguish more than a dozen—transient vs. permanent, client-side vs. server-side, payload vs. context, generic vs. vendor-specific, recoverable vs. unrecoverable—each with its own dedicated recovery strategy.
3. Recording: Archive in structured form; the schema must be stronger than free text. A failure record contains at minimum a task fingerprint, category, one-sentence summary, root cause, remediation action, and executable lessons.
4. Recall: When a new task arrives, proactively retrieve relevant historical failures by similarity and inject them into the prompt. This is the fundamental difference between a failure journal and an ordinary try-catch—the latter only handles the failure on the spot, the former remembers across tasks. Recall works by injecting past experience into the context at inference time (the training-free path), without retraining the model.

Two supporting engineering practices come with it: discarding failure trajectories by default is an enormous waste (research shows about 85% of failure trajectories are discarded), and they can be used for training after reverse annotation; tiered retention (for example, full detail kept for 48 hours, compressed summaries for 30 days, only aggregate metrics beyond that) makes recall feasible in engineering terms without letting the database grow without bound.

Production scenarios where it fits

• Production-grade, long-running agents: Agents that run for months and repeatedly handle the same class of tasks. Which failures to record and which to pull up when a task arrives is where its core value lies.
• Multi-tenant SaaS agents: High-risk failures such as cross-tenant misrouting require special handling—never evicted, force-recalled at the start of every task, alarm triggered on several consecutive records. The task signature uses a two-factor "tenant plus intent" key, so one tenant's failure should not cause another tenant to receive irrelevant reminders.
• Agents performing high-risk, irreversible operations: Scenarios where the cost of an error is large and the same mistake cannot be made twice. DevOps, finance, contract processing.

Where it tends to go wrong

• Forcing it onto one-off tasks with no repeated execution: For pure demos, prototypes, and one-off scripts, discarding failures is fine. The value of a failure journal lies in repeated tasks; without repetition there is no recall payoff.
• Free text only, no classification: The first version of a failure log is often free text with three fields, "task / error / timestamp," from which the agent can learn nothing. The schema must be enforced (enums plus constrained fields) so failures can be clustered, counted, and queried.
• Free-form writing produces a noise repository: Let the agent write the failure journal freely and it becomes overly pessimistic, recording an entry for every minor hiccup; after six months the journal turns into a noise repository and recall precision falls off a cliff. Use two-layer review: schema enforcement (free text ≤ 200 characters) plus an access_count-triggered cleanup (entries not recalled for a long time are archived to a cold tier).
• Recorded but recall does not match: Coarse similarity measures such as Jaccard have too high a miss rate, so recording is as good as not recalling. Task signatures should be compared with embedding cosine similarity.
• Lumping high-risk and ordinary failures together: A cross-tenant data leak and an API timeout differ by several orders of magnitude in severity; mixing them lets life-or-death failures get drowned out. High-risk failures need independent alerting plus permanent retention plus forced recall.

Key metrics

• Repeat failure rate decline (healthy zone: sustained decline): The proportion of times the agent repeats the same error within the same class of task. This is the true criterion for a failure journal—99% cleanliness of failure data is useless if the agent never recalls it; a 30% drop in repeat failure rate is the real value.
• Recall rate / recall hit rate (healthy zone: reasonably high): The share of failure records that have been recalled, and the proportion of recalled failures relevant to the current task. A persistently low recall hit rate for procedural memory indicates a problem with the memory architecture.
• First-attempt success rate (healthy zone: rising over time): The proportion of times the agent gets it right on the first try, which should rise month over month once failure lessons accumulate.
• High-risk failure count (healthy zone: near 0 with strong monitoring): The number of life-or-death failures such as cross-tenant misrouting, which must be monitored and alerted independently.

Minimal implementation skeleton

FailureEntry: failure_id / task_signature / category(enum) / summary
                          / root_cause / remediation / lessons[] / access_count
            FailureJournal:
                record(entry)          → on duplicate failure_id, increment access_count; on overflow, evict by access + recency
                by_category(cat)       → query by category
                recall_for_task(sig)   → embedding-similarity top-K recall of relevant historical failures
                render_for_prompt()    → format and inject into prompt (CER paradigm: glance at past pitfalls before acting)
            Tiered retention: 48h full detail / 30d compressed summary / 90d+ aggregate metrics
            

Four points for engineering deployment: use embedding cosine for the similarity function rather than Jaccard; hook recall at task startup and before high-risk tool calls (a PreToolUse hook works); actually implement tiered retention and place each tier on a different backend; and apply two-layer review on writes (schema enforcement plus access_count cleanup).

Enterprise deployment example

A B2B SaaS company provides a customer-service agent to more than two hundred enterprise tenants, handling about fifty thousand tickets a day. The failure distribution is roughly thirty percent tool errors, twenty-five percent cross-tenant misrouting, twenty percent context overflow, and fifteen percent semantic drift. The rewrite produced six key decisions: cross-tenant misrouting (boundary_leak) was set as the highest priority—a hundred times more severe than tool errors (the former is a data-leak incident), never evicted, force-recalled at the start of every task, with three consecutive records triggering PagerDuty; the task signature uses a two-factor "tenant plus intent" key, so ACME's failure does not cause BETA to receive irrelevant reminders; following the CER paradigm, the top-5 relevant historical failures are injected as a system prompt so that the implicit belief update happens before acting; failures must be classified, free text is not allowed; the failure archive is exported weekly as markdown and committed to git for team review (a git diff is the best visualization of failure trends); UNKNOWN-class failures enter a needs_review state and only enter the archive after human review, avoiding noise that pollutes recall. In the second month after deployment, boundary_leak dropped from eight cases a day to one, the tool_error repeat rate fell by forty percent, and first-attempt accuracy rose from 78% to 85%.

Relationship to other patterns

• Procedural memory (M5): A twin relationship; together they form the agent's experience base. The failure journal records "I did this wrong before, don't do it this way again," while procedural memory records "I did this right before, follow this next time." One side records failures, the other records successes.
• Progress tracking (M3): Also in the Loop column. Progress tracking loops over "the steps done right," the failure journal loops over "the pits stepped in." The two look symmetric, but in industry their treatment is completely asymmetric—everyone does the former, few do the latter seriously.
• Layered retention (M1): The failure journal is a dedicated partition within the long-term memory layer, and tiered retention (48h / 30d / 90d+) is precisely the application of layering to the failure archive.
• Self-Heal Loop (Action module): The self-heal loop only handles failures within the current task and does not persist across tasks. The failure journal goes one step further—it persists the current failure signal and recalls it on the next similar task.

What this pattern teaches

Erasing failures is the same as erasing evidence, and the model cannot learn—a failure journal is not an error log, it is the agent's professional record. A weaker-model agent that has run for half a year with a failure archive may, on your specific business, outperform a smarter model that has just started up.

This page is part of the ADPS pattern white papers. Back to the pattern matrix, or see the runnable code catalog.