AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents: what it means for business leaders

Deleting a record from an AI agent's memory does not delete what the agent knows: in controlled tests, one plain question recovered the deleted secret about two thirds of the time from summaries built before the deletion.

1. What the method is

AfterDeleteBench is a controlled audit for one specific failure: an agent with external persistent memory deletes a record, but information from that record survives in state derived from it, such as a summary. For each synthetic secret the benchmark builds two matched worlds. In the forget world the agent sees the secret, consolidates memory, and then the source record is deleted. In the clean world the agent sees a neutral record instead and goes through the same steps. Model weights never change. Comparing exact recovery between the two worlds attributes any recovery to the original exposure rather than to guessing. The benchmark also separates three events that are usually blurred together: reconstructing the secret during an audit, writing it back into memory, and producing it again in a fresh session. It also tests two remedies, provenance-based deletion and a write-time filter.

2. Why the method was developed

Most work on forgetting in language models concerns the weights: machine unlearning, editing parameters, and extraction attacks against training data. Agents now carry a second store of state outside the weights, made of conversation records, summaries and fact ledgers, and deletion there is usually treated as a storage operation: remove the row, confirm it is gone. The authors' point is that a deletion receipt says nothing about what the agent can still say. A summary written before the deletion is a separate object, so it stays. Earlier agent-memory studies showed that memory can be extracted or poisoned when intact, but they did not attribute recovery after deletion to a specific exposure with a matched control. The authors acknowledge concurrent work on execution-state unlearning and provenance-guided replay. They claim a specific combination of methods, not a new problem.

3. Who should care

This matters to anyone shipping or buying an agent that remembers users across sessions. Privacy and legal teams handling deletion requests should note that removing the source record may not satisfy what a user reasonably expects, because derived summaries can still disclose the data. Platform engineers who own memory pipelines get a concrete design requirement: record which source records each summary was built from, so that deleting a source also retires or rebuilds its descendants. Security and red-team staff get a test protocol with a matched control and exact scoring that can sit in our release checks. ML evaluation teams should note the measurement warning: a mismatch between the requested answer format and the parser hid real disclosures entirely in some models. Procurement should ask memory-agent vendors how deletion reaches derived state, and should ask for behavioral evidence rather than storage logs.

4. How the method works

Each synthetic profile holds a target code, a public attribute naming it, background facts, and matched secret-bearing and neutral records. Flat memory keeps only raw records. Hierarchical memory also writes an immutable extractive summary that joins the text of the records verbatim, and evaluator-side metadata records which sources fed each summary. Exact deletion removes the addressed record and leaves the summary alone. Recorded-lineage closure retires every summary whose sources include the deleted record and rebuilds it from the records that survive. A sealed gate stores an HMAC-SHA256 descriptor of the deleted value and blocks any later write that exactly matches it. A separate suite stores only the three components of a code of the form REGION-MNEMONIC-NUMBER, with a public rule for assembling them, so that no complete copy exists. Auditors range from one DIRECT question to scripts, bandits, PPO policies and an LLM selector, with at most ten queries chosen from seven templates.

5. How it was evaluated

The primary model is Qwen2.5-7B-Instruct. Mistral-7B-Instruct-v0.3 and Phi-3-medium-4k-instruct appear in the cross-model and component-storage studies. The primary matrix uses 200 profiles per split. One split draws from the training inventories and one holds out target families, templates and vocabulary as a bounded synthetic shift. Profiles are balanced across four domains and four structures: atomic, duplicated, correlated and compositional. A second suite uses new panels of 104 profiles per split per model to test answer-format alignment, a matched comparison of write, no-write and gate-blocked branches, and the component-storage conditions. Scoring needs an exact string match in a parsed top-level field, and raw replies are kept in later studies to catch disclosures the parser misses. Intervals come from 10,000 paired bootstrap resamples. A 100-case benign-utility check looks for obvious side effects. Many follow-up studies were designed after the primary run, and the authors say so.

6. How it performed

With hierarchical memory, one direct question recovered the deleted code in 68% of profiles in distribution and 64% under shift, against 0 of 200 in the clean worlds. Atomic and correlated structures reached 88 to 100%, while compositional profiles reached zero. With only separated components stored, and the full string absent from all 1,248 model-visible states, exact reconstruction ran from 62 to 99% across the three models. Lineage closure brought every measured endpoint to zero. The gate blocked write-back, but fresh-session recall stayed at 68% because the residue remained. Format mismatches scored Mistral and Phi-3 at zero even when 77 and 78 of 104 raw replies contained the target. Of 3,088 recoveries, 3,081 came on the first query. The limits are synthetic exact strings, extractive summaries and complete provenance. Our call is to pilot: add this paired audit and source tracking to our memory pipeline now. (Source: arXiv 2610.31366, 2026)