A Survey of Experience-Driven LLM Agents

At a Glance

LLM agents record in their trajectories which strategies failed and how they recovered, yet an agent that resolves a problem in one episode may repeat the same diagnostic detour in the next. Work on reflection, agent memory, workflow learning, and skill acquisition addresses this gap under different names: a distilled lesson is a self-reflection in Reflexion, an insight in ExpeL, a causal abstraction in CLIN, and a memory item in ReasoningBank. Because names and content do not line up, it is hard to see which guidance a method retains, when that guidance applies, and how to compare reported gains.

The survey makes four contributions.

From LLMs to Experience-Driven Agents

Agents differ in whether their retained experience changes during later work. In fixed-repository reuse the repository stays fixed; in closed-loop learning later trajectories inform the formation and maintenance of experience, and the updated repository guides subsequent episodes. This cycle is the experience loop. One success supports a lesson only in comparable situations: another import failure may have a different cause.

Foundations

What counts as agent experience

Agent experience is action-guiding information that is grounded in one or more task-execution trajectories and retained to guide later episodes where it applies.

Correctness and benefit are not required: later checks test them. Outside the core scope are audit-only logs, factual or preference memory without a guidance role, instructions with no task practice behind them (generic instructions, tool documentation), and learning solely through parameter updates.

Three representational forms

Consider one code-repair episode: tests fail at the import stage, the agent traces the failure to a declared dependency missing from the active environment, restores it, and confirms that the import succeeds. The agent can retain three forms of experience from it.

Form describes only how guidance is expressed. The levels are not stages of maturity and do not rank generality: a scoped lesson can apply as narrowly as the precedent it came from, and several forms can coexist. All three have developed side by side since 2023.

The experience unit ⟨g, c, e, r⟩

Whatever its form, every piece of experience — an experience unit — can be described along four analytical dimensions. They are questions to ask of any unit, not fields a system must store.

E_i = ⟨ g_i, c_i, e_i, r_i ⟩

Reliability asks whether guidance is well supported under its stated conditions. Utility asks whether using it pays off for a particular executor and task; it depends on context and is not a fixed attribute of the unit. If the executor already checks dependencies unprompted, the lesson may be reliable yet add little utility.

Four experience types

Independently of form, experience can be classified by the execution outcome its guidance draws on. Each type needs a different check before its guidance is relied on.

The experience loop

During episode k, retained experience is turned into situated guidance for the current decision:

A_k,t = Utilize(R_k, q_k, h_k,≤t)

Closed-loop learning adds a path from execution back to the experience repository R:

(y_k, T_k) = Execute(q_k, R_k)
𝓔_k = Form(T_k, R_k, 𝓗_<k)
R_k+1 = Update(R_k, 𝓔_k, T_k)

Formation proposes a possibly empty set of candidate units 𝓔_k; maintenance decides which enter R_k+1 and how retained units change. An experienceless agent never receives situated guidance, fixed-repository reuse keeps R_k unchanged, and producing A_k,t does not by itself retain a new unit.

The Experience Lifecycle Framework

Making the experience loop work requires four kinds of decisions: what to learn from a trajectory, how to store it so that it can be found, how to use it to guide behavior or train the executor, and what to keep or change as evidence accumulates.

The division also locates failures. For the dependency lesson, a version claiming that every import failure requires installing a dependency points to unsupported generalization during formation; a retained lesson missing from the lookup index points to organization; applying it despite incompatible environment assumptions points to utilization; and leaving it unchanged after new evidence contradicts it points to maintenance.

Experience Formation

Browse 43 papers →

Formation turns trajectories into candidate experience: a precedent to consult, a lesson to apply, or a procedure to follow. The survey's central argument here: the more a mechanism adds beyond what the trajectories recorded, the more its support must come from outside them — through counterexamples, tests beyond the source tasks, or new execution.

Trajectory-level

• Select and annotate RAP, SEER, TRAD
• Segment and compress Synapse
• Relabel and reconstruct ECHO, BAGEL, BREW

Semantic-level

• Reflective abstraction Reflexion, CLIN, ReasoningBank
• Comparison and diagnosis AutoGuide, ExpeL, TF-GRPO
• Cross-trajectory generalization AutoManual, EMG, EDV

Procedural-level

• Extraction and parameterization AWM, SSO, TraceCompiler
• Synthesis and composition Voyager, SkillWeaver, Metis
• Execution-guided verification ASI, SkillCAT, SkillOpt

Key findings

• Deciding more in advance pays off only while conditions stay stable. A precedent leaves interpretation to the receiving agent, a lesson states a conclusion, and a procedure fixes the steps. In Metis (AppWorld, GPT-4o executor), when each task could use only memory from earlier tasks, text memory reached 73.3% task success against 53.3% for code.
• No form wins on every benchmark. In ExpeL, insights beat retrieved trajectories on HotpotQA (36% vs 31%) while trajectories win on ALFWorld (55% vs 50%); in Memp on ALFWorld with GPT-4o, scripts alone reach 56.4%, trajectories 74.3%, and both 77.9%. Complementary forms help.
• Each formation check supports only what it tests. Source success concerns the recorded behavior, a model's judgment remains an interpretation, a successful rerun shows the candidate works under the tested conditions, and only held-out tasks test transfer.

Experience Organization

Browse 47 papers →

A stored unit helps only if the agent can find it and reach the context needed to read it. Even a correct match leaves four questions: Q1 was the fix recorded with the failure? Q2 is the condition under which it worked kept with it? Q3 is the match one step of a longer procedure, or related to other precedents? Q4 can the agent open what lies behind or inside the match?

Pattern	Useful when	Upkeep and failure modes
Flat	A symptom, task, or usage description matches the request, and the record itself holds the repair and its conditions (Q1–Q2).	Keys must anticipate later needs; a missing key fails silently and relations stay implicit.
Tree	Many similar records call for a narrower place to search, or the missing context is a broader group, containing routine, or base procedure (Q3).	Branch choices can omit records; revising a shared base affects every variant.
Graph	The repair, a continuation, or a supporting skill sits in another record that would not match on its own (Q1, Q3).	Relations must be built, checked, and updated; unreliable links and wide expansion mislead or overload access.
Hierarchical graph	The match needs its source trajectory or implementation, or must be checked or trimmed before use (Q4).	Cross-layer correspondences must stay consistent; expansion adds inspection cost.

Key findings

• Better keys before added structure. The right record is often missed only because of how it is keyed; keying it by the observable symptom, as MemGovern's repair cards do with error signatures, is the cheaper first remedy. Added structure should beat that improved-key baseline.
• Structure keeps conditions, evidence, and reliability reachable — condition keys make applicability searchable and source links keep supporting trajectories reachable — but the agent must still judge whether the guidance fits.
• Reported ablations rarely isolate structure from content. Removing a level or an edge usually removes what it holds, so whether structure itself explains reported gains remains open.

Experience Utilization

Browse 60 papers →

An agent holding useful experience may still overlook it, follow a familiar procedure under the wrong conditions, or spend more effort consulting experience than the task requires. At runtime, utilization answers four connected questions; during training, experience can also be internalized into model parameters.

Experience triggering

When should the agent consult experience?

Fixed schedules, observed failures, internal-state signals, coach gates, learned policies.

Experience selection

Which experience, if any, is worth using?

Retrieval, reranking and filtering, and allocation across multiple agents.

Experience adaptation

What should change before reuse?

Contextual grounding, guidance specialization, procedural reconfiguration, feedback-guided revision.

Experience application

How should guidance shape behavior?

Five functions: interpretation, generation, decision-making, procedure reuse, and evaluation.

Experience internalization transfers experience-supported behavior into model or adapter parameters: training on experience-shaped trajectories, distilling experience-conditioned policies, or withdrawing guidance during training.

Key findings

• The four decisions share one budget. Seeking, selecting, and adapting experience all cost resources before the guidance helps, so inexpensive checks that could change the reuse decision belong before costly actions.
• Recurrence decides whether experience is worth it. On largely self-contained web tasks, giving an executor without experience more interaction steps can match or exceed online skill and memory augmentation at approximately matched token cost.
• Adapting before reuse pays. With DeepSeek-V4-Pro, QCR's query-conditioned notes raise average success by 10.7 points over the full trajectory while using 48.9% fewer online tokens.
• Check outcomes independently of the explanation experience proposed; when the same experience shapes both the action and its check, the check can inherit the action's mistaken premise.

Experience Maintenance

Browse 68 papers →

Each episode leaves new candidates and evidence about experience already retained. Simply appending candidates accumulates noise alongside useful guidance; maintenance decides what to admit, combine, correct, or retire so that later reuse stays reliable and efficient.

Operation	Problem signal	Maintenance choice	Main risk
Admission	A new candidate is proposed	Accept supported value; defer uncertainty; reject unsupported content	Excluding rare useful guidance or accepting a false explanation
Consolidation	Equivalent or fragmented guidance accumulates	Merge equivalents; abstract or share structure while keeping distinctions	Compression removes an action-changing condition
Refinement	Contradictory outcomes or changed prerequisites	Diagnose; adjust reliability, scope, or actions; validate before replacement	Repairing the wrong cause or breaking valid behavior
Retirement	Suspected loss of added benefit	Downweight, archive, or delete according to evidence and recoverability	Confusing poor access with low value; losing a rare recovery
Coordination	Repeated formation defects or mismatched access	Change operation selection, formation, or retrieval	Delayed rewards obscure which change helped
Safeguards	A premise loses support or contamination is detected	Restrict affected guidance; trace, repair, and validate successors	Missing descendants or removing independently valid guidance

Evidence for these decisions comes at three check levels: a support check examines content against recorded evidence; a paired comparison measures the local gain from using it; a collection-level test evaluates it together with the experience it would join.

Key findings

• Admission quality can decide whether updating helps at all. In the Experience-Following study, adding every trajectory ends below a fixed copy of the starting memory, while selective addition with a strict ground-truth evaluator ends above it — but deployed agents rarely have such labels.
• Updates can reverse earlier gains and spread. Consolidation can drop an action-changing detail, and a mistaken update can propagate into experience derived from it, so maintenance must also test unaffected behavior and keep what is needed to repair descendants.
• Open questions: how broadly to update, which interactions a collection-level test must cover, how to value units whose usage data earlier decisions have filtered, and where to spend verification effort.

Evaluation: Three Targets, Three Claims

Task success alone does not show whether stored experience is correct or whether the agent benefited from using it: an agent may ignore a useful lesson, complete a task despite errors in a stored procedure, or score higher simply because of additional model calls.

Higher performance with experience does not by itself show that an agent keeps improving. Each improvement claim needs more demanding evidence than the one before:

Browse 43 evaluation papers and benchmarks →

Applications

Seven application settings, compared on the four reuse conditions that determine what experience is worth retaining and how it can be checked.

Four design decisions

Browse 31 application papers →

Main Conclusions

1. Keep guidance connected to its conditions and evidence.

   The central design principle: throughout formation, reuse, and revision, keep guidance tied to its applicability conditions and to evidence that can support or challenge it. A more compact or executable representation need not preserve clearer conditions or stronger support.

2. Condensed guidance can lose the conditions that made it work.

   “On import errors, check dependencies,” condensed further into “reinstall dependencies,” skips the step that confirms a dependency is missing. Keeping the source trajectory with the lesson makes such omissions recoverable.

3. Shared experience must fit the agent that receives it.

   LEGOMem's ablations show that where task and subtask memories are placed affects success, and a more capable teacher does not always produce the most helpful memory for a given student.

4. Entries harmless individually can be harmful together.

   Entries that a model-based auditor mostly judges benign separately can elicit harmful behavior when a query activates them together, so units must be tested in the combinations later queries activate.

5. Rising task-stream scores mix learning with changing task demands.

   Separating the two requires comparing the updating agent with a copy whose starting repository stays frozen, on comparable tasks.

6. Fixed experience reuse is the best-supported claim.

   Evidence is clearest where tests or reference answers check outcomes. Sustained improvement is supported only partially, and improvement of the learning mechanism further requires testing the revised mechanism's contribution to subsequent learning.

7. Reuse shapes what the agent learns next.

   A familiar strategy may suppress exploration, and a mistaken premise may propagate into later experience. Preserving sources and conditions lets later evidence challenge earlier guidance.

Limits: the survey reviews a selective, mechanism-oriented corpus rather than a systematic screen; much evidence comes from recent preprints; assigning evidence roles and improvement claims is the authors' reading of each experimental design; and the proposed evaluation protocols have not yet been implemented.

Eight Open Problems

Toward improvement that is cumulative, transferable, and correctable. Each problem pairs why it remains open with what would count as progress.

Acquiring and extending experience

Keeping experience useful, revisable, and safe

Evaluating cumulative progress

Cite

If the survey or this list helps your research, please cite:

@misc{yan2026trajectories, …}