Pattern Matrix/White Paper/F3

F3 · Experience Replay

What problem it solves

A company runs for a few years, employees turn over across several generations, and everything the engineers ever did is scattered across Slack messages, Jira tickets, the internal wiki, and email archives. These are the company's "silent assets"—80% of them are never reopened, but any single entry might save half a day's work when a new task is picked up. A senior employee who leaves takes away more than skills: the pitfalls they stepped into, the failed paths they tried, the process by which they once discovered a key variable. None of this can be summarized in a skill package, but it carries reuse value all the same.

Experience Replay solves the same problem inside an agent system—keeping historical trajectories from becoming silent assets. When a new task arrives, it proactively retrieves similar past trajectories (successful, failed, fragmentary) and injects the reusable parts into the current context. Its difference from Skill Package is one of granularity: a Skill Package holds verified, callable units, while Experience Replay holds broader reference assets—experience that is useful but not necessarily verified. CER (Contextual Experience Replay) raised the GPT-4o baseline success rate on WebArena by 51%, and is entirely training-free.

Why the coordinate is "Reflection × Hierarchy"

• Vertical axis · Reflection: What was learned should not be learned in vain; it is fed back into later tasks. This is reflection grounded in looking back over history. It neither changes the present output (Generator-Critic) nor solidifies into a callable unit (Skill Package); instead it treats complete past experience as a reference asset for the agent to draw on.
• Horizontal axis · Hierarchy: Historical trajectories form an "upper experience layer" that wraps around the current decision—the upper layer of experience supplies reference and constraint for the present action below. Experience itself is also stored in layers: raw trajectories, extracted lessons, reusable artifacts, candidates for promotion to skills—from coarse to fine. A multi-level structure in which the upper layer parametrizes the lower one is exactly the essence of the Hierarchy topology, distinct from the single "route to which skill" choice in Skill Package.

Core mechanism

An Experience Replay loop has six segments: Task → Retrieve → Adapt → Execute → Distill → Store. Two core design judgments decide whether it can be industrialized:

1. Multi-level layered storage: Experience must be stored in layers; a single layer is either too scattered or too empty. AgentRR offers two layers—low-level (concrete actions: which tool was called, what parameters were filled in, what observation came back, used for debugging) plus high-level (generalized strategy: what approach was used, what type of problem it addresses, when it works, closer to an engineer's judgment). Manning extends this upward into a three-level abstraction ladder:

2. Training-free injection: Retraining the LLM is too expensive. CER's approach is to inject past trajectories (after trimming and summarization) as context into the current prompt, letting the LLM do in-context learning within the context window and skipping the training stage. During retrieval, experience with high effectiveness and successful outcomes is prioritized; after reuse, effectiveness is written back—useless lessons are automatically deprecated.

Production scenarios it fits

• Agents with long cumulative runtime: Customer service, operations, data analysis—scenarios where tasks recur and experience can accumulate. Engineering value grows over time: significant effect only after 3 months, becoming a core dividend after 6 months.
• Organizational knowledge accumulation: Connect the experience scattered across the company's wiki, Confluence, and Notion, and the agent retrieves and organizes it for use within seconds. In enterprise scenarios, hooking into the existing knowledge base is the most economical option.
• Reuse of failure assets: 80% of trajectories are failures. AgentHER uses hindsight relabeling to make failed trajectories more valuable—a trajectory that failed to pursue goal A is often a successful demonstration of some reachable goal B, scoring 7.1 to 11.7 percentage points higher than using only successful trajectories on WebArena plus ToolBench.

Where it tends to go wrong

• Cold-start deadlock: The first 3 months carry almost no value, but without accumulation there is no value after 6 months either. Teams easily decide to "pause it until the system stabilizes, then turn it on," and then it never comes back up. The defense is synthetic experience seeding during the cold-start period: write 30-50 bootstrap entries from the team's past documented best practices and load them into the store.
• Stale-lesson drift: An experience that was correct half a year ago no longer applies after a product revision, yet the agent keeps injecting it and steers people the wrong way. The defense is continuous decay of the effectiveness_score plus time-stamp downweighting of heuristics plus active quarterly review to deprecate.
• Retrieval-bias amplification: Embedding similarity does not equal task-structure similarity. "User growth analysis" retrieves "user churn analysis" experience and forces the churn funnel framework onto growth—growth and churn are inverse problems. The defense is to add a task-type classifier for coarse filtering, block cross-type retrieval, and have the LLM perform a second applicability review after retrieval.
• Unstructured bulk dump: Cramming every trajectory into a vector DB and letting the agent pick on its own, with no multi-level layering.

Key metrics

• Retrieval hit rate (healthy zone >70%): The proportion of recalled historical experience that is actually adopted. Low values indicate the task_signature used a hash instead of semantics, or retrieval bias.
• Effectiveness score (healthy zone ≥0.5): Task success rate after experience is retrieved. Lessons below 0.5 that have been reused 5 or more times go into the archive; this is key to resisting superstitious lessons (pseudo-experience from misattribution).
• Experience store growth plus retrieval count (the two to watch during cold start): For the first 3 months, do not watch success-rate improvement; watch whether the store is growing and whether experience is being retrieved, to buy time for reflection engineering.
• Quality gain (healthy zone: significant after 3 months, pulling away from baseline after 6 months): The success-rate gain relative to no experience reuse. This is a lagging indicator.

Minimal implementation skeleton

# Retrieve + inject (CER training-free)
            past = retrieve(task)                      # top-K, prioritize high effectiveness + success
            context += render_for_context(past)        # high-level strategy + author metadata
            heuristics = get_L2_by_signature(task)     # cross-task regularities
            context += render(heuristics)

            result = agent.run(task, prior_context=context, trajectory_collector=traj)

            # Record new experience (auto-distill high-level, use a cheap model)
            record(task, traj, outcome, author_id)     # high + low two layers

            # Write back effectiveness (lesson accuracy feedback loop)
            for e in past:
                update_effectiveness(e, current_task_succeeded=result.ok)

            # Periodically distill L2 (only when >= 5 entries share a signature, to prevent overfit)
            monthly_distill_L2(signature)
            

Four engineering points: use the cheapest model for both embedding and high-level distillation (high frequency; the store will grow to hundreds of thousands of entries); the effectiveness_score is the core metric, with a health line of ≥0.5; do not skip the author_id metadata—it lets the agent tell the user "who has done this, and whom to consult"; use semantics rather than a hash for the task_signature, or the hit rate will be low.

Enterprise rollout example

An 8-person data team at a SaaS company. The PM asks, "After the Q2 new feature launched, D7 retention didn't visibly rise—analyze why." Eight data scientists work on it separately, and a month later produce 5 independent reports (3 unfinished). The post-mortem finds that 3 ran almost identical cohort analyses, 2 used different attribution methods but both missed the same confounding variable (the natural traffic baseline shifted during the Black Friday window), and 0 cited the team's similar analysis from 6 months earlier—that analysis was done by Sarah, who had since left. She had built out the method, had stepped on the confounding issue herself and written a case note, but it was archived in SharePoint where no one looks. After the team adopts Experience Replay, the next time a similar task comes up, the first person to start triggers retrieval and pulls up Sarah's analysis (including the heuristic "always check time-window effects in retention analysis"), injected into the prompt as a cross-task heuristic. Six accompanying decisions: a multi-level schema (high-level approach plus low-level SQL), CER injection of the top-5 plus author metadata, embedding with a cheap model, storing successful plus failed plus neutral trajectories, a 30-day cold tier, and monthly automatic L2 distillation. The 8 person-months of waste are compressed down to 1 person-month for a well-done new analysis.

Relationship to other patterns

• Skill Package (F2): A paired sibling pattern whose boundary must be nailed down. Skill Package holds verified callable units ("this has been done 3 times, all successful—seal it into a skill and call it directly"); Experience Replay holds broader reference assets ("Sarah did this 6 months ago; the method and the pitfalls are in the trace, and the agent learns from them on its own"). The former is a module, the latter a case library. The agent calls a skill first and falls back to experience retrieval when there is no match; experience that succeeds repeatedly graduates into a skill.
• Failure Journals (memory module): A shared-source input. Failure Journals supplies the library of failed trajectories, Experience Replay supplies the retrieval-plus-injection mechanism, and the two work together. After AgentHER, failed trajectories also became high-value training material.
• Generator-Critic (F1): A progression in level. Generator-Critic changes a single output, while Experience Replay lets accumulated experience be reused across tasks.

What this pattern teaches

Experience Replay belongs to the class of reflection-engineering patterns where "time is your friend"—investment up front, dividends later. It turns the scattered, fragmentary experience inside an organization into a retrievable asset. An agent's maturity is not about how smart it is today, but about how much smarter it is in three months, in six months, than it is now.

This page is part of the ADPS pattern white papers. Back to the pattern matrix, or see the runnable code catalog.