From AutoResearch to Continual AutoResearch: Do Agents Learn Across Tasks?: what it means for business leaders

A paired test of five frontier coding agents finds that keeping session state across related ML tasks produces strong solutions sooner, but rarely produces better final ones.

1. What the method is

Continual AutoResearch is an evaluation environment, not a new agent or model. It measures whether a coding agent doing ML research work improves across tasks when it keeps its working state and no model weights change. Each ordered stream of three to five related tasks is run with the same model, data, order and wall-clock budget in two ways: a stateful run resumes the same native CLI session and workspace, so transcript, files and results carry over, while a reset run starts each task with a fresh session and a clean workspace. The paper applies this to five agents across nine task families and adds a memory test that isolates written insights from session continuity. The advance is a paired measurement that separates the quality of the first solution, how early strong solutions appear, and the quality of the final one.

2. Why the method was developed

Agent research loops, from Karpathy's autoresearch onward, have mostly been evaluated one problem at a time, with the agent reset before each new task. That discards what was learned, so those evaluations cannot say whether lessons about experimental design, modeling choices or execution failures carry forward, and each run may pay to rediscover them. Carrying state has its own risk: irrelevant or misleading experience can follow the agent, and experience helps only where tasks share structure. Existing continual-learning benchmarks for language agents cover areas such as software engineering, database querying and games. The authors' distinction is that every task here is an executable ML research loop, with hypotheses, code, multiple candidates and a final selection under a fixed budget. For anyone paying for agent time, the open question was whether a persistent session buys better results or only faster ones.

3. Who should care

Heads of ML and data science using coding agents for modeling work should read this as evidence on speed rather than model quality: persistence may deliver a usable baseline sooner, but final results still need independent checks. Platform engineers deciding whether sessions and workspaces persist, and what memory to add, should note that the effect varies by agent and task family, and that stored summaries can anchor work on a weaker approach. Finance and engineering leaders paying for agent time should note that the authors find persistence most valuable when budgets are tight. Security and risk teams should weigh a concern the authors raise but did not test: misleading or malicious content retained from one task could affect later ones. Those buying agent products should ask vendors for paired evidence behind any claim about memory.

4. How the method works

Each task gives the agent a description, training and development data, and a fixed time budget. Agents work through their native command-line tools with no added memory or planning harness, may submit several solutions, and must select a final one. Every valid submission is scored after the run on a hidden test set; intermediate test scores stay hidden to prevent reward hacking, and dataset names are anonymized to block recognition shortcuts. Metrics are normalized to a common higher-is-better scale, and gains are stateful minus reset. Three gains are reported: final selected solution, first valid solution, and time-convergence, the normalized area under the best-so-far reward curve, which rewards finding good solutions early. The memory test starts every task in a fresh session and transfers only agent-written insight cards recording claims, evidence, confidence, scope, recommended actions and failure conditions: matched cards, cards from an unrelated stream, or none.

5. How it was evaluated

The authors tested Sonnet 4.6 and Opus 5 through Claude Code, GPT-5.4 and GPT-5.6-Sol through Codex, and Kimi K3 through OpenCode. Each agent faced 32 transfer tasks, excluding each stream's first task, usually with 30 minutes per task. The nine families comprise three synthetic AutoML streams, one symbolic regression stream, three forecasting streams built on energy, traffic and electricity-transformer data, and two tabular streams drawn from the SELA evaluation subset and MLE-Bench Lite. The baseline is the same agent reset between tasks. The evaluation is thin in places: only GPT-5.6-Sol was repeated, three times, and every other comparison rests on one matched run pair. The memory test used only the energy stream, and the 15, 30 and 60 minute budget test used only GPT-5.6-Sol on one synthetic family. Code is promised but not yet released, and our copy of the appendix was truncated.

6. How it performed

Time-convergence gain was positive for all five: Kimi K3 +21.3 normalized points, Sonnet 4.6 +20.9, Opus 5 +7.5, GPT-5.6-Sol +6.1, GPT-5.4 +3.0. Final outcomes did not follow: Sonnet gained +7.6, the others ranged from -0.9 to +0.4, and on best reward at the deadline only two agents stayed ahead. The early lead came from a sooner or stronger first solution; after also subtracting first-solution reward, the gain turned negative for every agent. Against unrelated cards, matched cards improved starts for three agents but reduced their time-convergence. Repeated GPT-5.6-Sol runs varied by up to 7.14 time-convergence points, which the authors call small but which is comparable to the three smallest gains. Streams were short and budgets brief. Our call: pilot persistent sessions where tasks are related and time is tight, keeping reset runs as a control, and do not expect persistence alone to deliver better final models. (Source: arXiv 2611.05108, 2026)