Pattern Matrix/White Paper/F3

Problem

A company accumulates engineering decisions across chat, tickets, wikis, reports, and email. Much of this material is rarely reopened, even though a past investigation may contain the exact pitfall a new team is about to repeat. When an experienced employee leaves, the organization can lose the failed paths, diagnostic clues, and judgment behind a decision. Not all of that can be compressed into a skill package, but it still has reuse value.

Experience Replay solves the same problem inside an agent system by keeping historical trajectories from becoming silent assets. When a new task arrives, it retrieves similar past trajectories (successful, failed, or fragmentary) and injects the reusable parts into the current context. Its difference from Skill Package is one of granularity: a Skill Package holds verified, callable units, while Experience Replay holds broader reference assets that may still require adaptation. Research on contextual experience replay suggests that this can work without model retraining; production value still has to be reproduced against a local baseline.

Classification: Reflection × Hierarchy

• Vertical axis · Reflection: The system reviews previous tasks and feeds applicable evidence into later work. It uses complete experience as reference material without requiring that every entry become a callable skill.
• Horizontal axis · Hierarchy: Historical evidence sits above the current task as a reference layer. The store is also layered into raw trajectories, extracted lessons, reusable artifacts, and candidates for promotion to skills. This differs from F2, where runtime routing selects a packaged capability.

Solution and mechanics

An Experience Replay loop has six segments: Task → Retrieve → Adapt → Execute → Distill → Store. Two core design judgments decide whether it can be industrialized:

1. Multi-level layered storage: Experience must be stored in layers; a single layer is either too scattered or too empty. AgentRR offers two layers—low-level (concrete actions: which tool was called, what parameters were filled in, what observation came back, used for debugging) plus high-level (generalized strategy: what approach was used, what type of problem it addresses, when it works, closer to an engineer's judgment). Manning extends this upward into a three-level abstraction ladder:

2. Training-free injection: Retraining the LLM is too expensive. CER's approach is to inject past trajectories (after trimming and summarization) as context into the current prompt, letting the LLM do in-context learning within the context window and skipping the training stage. During retrieval, experience with high effectiveness and successful outcomes is prioritized; after reuse, effectiveness is written back—useless lessons are automatically deprecated.

A production system also separates experience formation from experience use. Runtime retrieval reads only released experience. An offline process groups traces, human hand-offs, and business outcomes, consolidates duplicate local patches, adds applicability conditions, and publishes an experience only after replay and review.

Feedback latency matters. Code tasks may receive a test result within minutes; recommendations, service decisions, and operating changes may take days to show an outcome. Store trajectory_id, runtime version, contemporaneous evidence, and the later outcome label. Without that link, an entry can be marked as verified simply because it looked reasonable when the task ended.

Online use and offline curation

• Online: Retrieve by task type, environment version, and risk label; run an applicability check; record which experience influenced which step.
• Offline: Compare batches of trajectories, hand-offs, and delayed outcomes; merge repeated patches; then replay, review, publish, down-rank, or archive the result.

The online path optimizes latency and rollback. The offline path optimizes coverage and consistency. Writing every runtime patch straight into the shared store creates a growing set of exceptions, conflicts, and retrieval cost.

Applicability

• Agents with long cumulative runtime: Customer service, operations, and data analysis are suitable when tasks recur and experience can accumulate. The benefit grows with the quality and coverage of the experience store.
• Organizational knowledge accumulation: Connect experience scattered across the company's wiki, document systems, and reports, then retrieve it in a form usable by the current task. Reusing existing knowledge infrastructure is usually the practical starting point.
• Reuse of failure assets: Failed trajectories may still contain useful subgoals, diagnostic evidence, or tool-use fragments. Hindsight relabeling can preserve that value, provided the new label records provenance and does not disguise a failure as a verified success.

Known failure modes

• Cold-start deadlock: A new store has little evidence, but postponing collection means it never matures. Seed it with reviewed historical records and best practices, mark their provenance, and start capturing new trajectories from the beginning.
• Stale-lesson drift: An experience that was once correct may no longer apply after a product revision, yet the agent keeps injecting it. Bind experience to system versions, downweight stale entries, and review or deprecate them on an explicit schedule.
• Retrieval-bias amplification: Embedding similarity does not equal task-structure similarity. "User growth analysis" retrieves "user churn analysis" experience and forces the churn funnel framework onto growth—growth and churn are inverse problems. The defense is to add a task-type classifier for coarse filtering, block cross-type retrieval, and have the LLM perform a second applicability review after retrieval.
• Unstructured bulk dump: Cramming every trajectory into a vector DB and letting the agent pick on its own, with no multi-level layering.
• Misattributing a delayed outcome: If a later business result cannot be linked to the original trajectory, version, and time window, the system may reward the wrong experience.
• Accumulating local patches: Each runtime failure appends a special rule. Short-term recovery turns into long-term retrieval noise and execution latency unless offline curation consolidates or removes the patches.

Verification and metrics

• Retrieval adoption rate: The proportion of recalled experience that the agent or reviewer actually uses. Low adoption can indicate poor task signatures, weak ranking, or stale entries.
• Effectiveness evidence: Compare outcomes when an experience is used with an appropriate baseline, and keep sample size and task type visible. Archive lessons that repeatedly fail review or provide no benefit.
• Store coverage and retrieval count: During cold start, track whether representative tasks are being captured and whether retrieval reaches the right task families.
• Quality gain: Compare replay-enabled runs with a no-replay baseline on a fixed set. Treat it as a lagging indicator and report retrieval cost alongside it.
• Trajectory-to-outcome linkage: Measure how many delayed outcomes can be connected to the exact task, version, and evidence available at execution time.
• Negative transfer and patch debt: Track tasks that regress after experience injection and local patches that remain unreviewed or unconsolidated.

Reference implementation

# Retrieve + inject (CER training-free)
            past = retrieve(task)                      # top-K, prioritize high effectiveness + success
            context += render_for_context(past)        # high-level strategy + author metadata
            heuristics = get_L2_by_signature(task)     # cross-task regularities
            context += render(heuristics)

            result = agent.run(task, prior_context=context, trajectory_collector=traj)
            record_adoption(task.id, past, result.trajectory)  # which experience changed which step

            # Record facts first; do not publish a lesson directly
            record_trace(task, traj, immediate_outcome, author_id, runtime_version)
            attach_delayed_outcome(task.id, delayed_outcome)

            # Write back effectiveness (lesson accuracy feedback loop)
            for e in past:
                update_effectiveness(e, current_task_succeeded=result.ok)

            # Consolidate local patches offline and publish only after replay and review
            candidate = consolidate_patches(signature, traces, outcomes)
            publish_L2(candidate, when=replay_passed and review_approved)
            

Choose embedding and distillation models from measured retrieval quality and cost. Keep outcome evidence rather than an unexplained score. Store author_id, trajectory_id, runtime version, and provenance. Use semantic task features rather than an opaque hash for task_signature. Runtime retrieval reads released experience; candidate lessons remain in the offline curation area.

Illustrative scenario

Consider a data team investigating why retention did not improve after a product change. Different analysts may repeat the same cohort analysis and miss a shared confounder, while a useful historical report remains buried in a document system after its author leaves. Experience Replay can retrieve that report and its heuristic about checking time-window effects, then inject the high-level method together with the original SQL, author, sources, and time range. A reviewer still decides whether the old conditions apply. The reusable design lies in layered storage, provenance, applicability review, and feedback after reuse; any claim about saved effort would require project records from a named case.

Related patterns

• Skill Package (F2): A paired sibling pattern whose boundary must be clear. Skill Package holds verified callable units: a path has enough evidence to be packaged and called directly. Experience Replay holds broader reference assets: a prior method and its pitfalls remain available, but the agent must judge whether they apply. The former is a module, the latter a case library. The agent calls a skill first and falls back to experience retrieval when there is no match; experience that succeeds repeatedly may graduate into a skill after review.
• Failure Journals (memory module): A shared-source input. Failure Journals supplies the library of failed trajectories, Experience Replay supplies the retrieval-plus-injection mechanism, and the two work together. After AgentHER, failed trajectories also became high-value training material.
• Generator-Critic (F1): A progression in level. Generator-Critic changes a single output, while Experience Replay lets accumulated experience be reused across tasks.

Engineering judgment

Experience Replay organizes historical trajectories, summaries, and reusable artifacts as evidence-bearing retrieval assets. Its value should be measured through later task outcomes, negative transfer, and human hand-off, with delayed outcomes linked back to the original run.

Further reading

• Reflection module: Make feedback change the system
• First Reflection workshop, 12 August 2026
• LangSmith Evaluation