AutoSim: A Self-Improving User Simulator for Conversational Search Evaluation: what it means for business leaders

A user simulator for conversational search shows that prompting cannot make simulated users behave like real ones, while fine-tuning the model can, and that only the fine-tuned version stops overstating how well a search system serves people.

1. What the method is

AutoSim is a simulated user for testing conversational search systems. It stands in for real people, asking questions, refining them and judging the passages it gets back. That lets a team compare retrieval and response designs without running a human study for each one. The paper adds a self-improvement loop that compares the simulator's behaviour with held-out real user sessions and edits the simulator to close the gap. The loop can edit either the prompt or the model weights. The paper's main finding is that the choice matters. Prompt edits stall well short of realistic behaviour, while fine-tuning the weights produces a simulated user that neither a trained classifier nor human readers can tell from a real one. The authors also show that the more realistic simulator gives a more accurate estimate of retrieval quality, which a prompted simulator overstates. Code, protocol and trained adapters are released.

2. Why the method was developed

Simulated users are the usual way to test interactive search and assistant systems at scale, but they are known to be unreliable stand-ins. Prior work found that swapping only the underlying user model moves a measured task-success rate by about nine percentage points, and that general-purpose LLM simulators overstate success. The cause shows up in the text. Asked to act as a search user, a strong LLM writes long, polite, complete requests, averaging 37.5 words per message against 12.8 for real users on TREC iKAT. Real users are brief and lean on earlier turns. The obvious fix, rewriting the prompt to ask for terse messages, moves the summary statistics but not the underlying behaviour. Earlier work had shown that trained user models beat prompted ones on realism, but nobody had asked whether that realism changes the evaluation numbers a company would act on.

3. Who should care

This matters most to teams that build or buy retrieval-augmented assistants, enterprise search or customer-facing chatbots and use synthetic users to decide whether a change ships. ML evaluation and QA owners should take away that a simulator which looks right on average message length can still overstate quality, and do so mostly on the hard queries where users fail. Product and engineering leads who approve deployments against a quality threshold are directly exposed, because the paper shows a prompted simulator pushing a system above a modest threshold that real users would not reach. Procurement should ask vendors who claim simulation-based evaluation how their simulated users were validated. Teams that build agents and testing tools have a reusable recipe: keep a strong LLM as the relevance judge and fine-tune a small open model as the user.

4. How the method works

AutoSim wraps an utterance generator in user state: a persona of background statements, an information need, the dialogue history and a latent mood covering satisfaction, frustration and patience. At each turn a policy chooses between deterministic components (reformulate, click, stop) and LLM functions (answer a clarifying question, give feedback, write the next utterance). Each loop cycle records traces, clusters the differences from real behaviour, proposes edits by reflective population search and keeps an edit only if it does not regress on a calibration split. For weight-level edits the authors warm-start Qwen2.5-3B with a LoRA adapter on real iKAT utterances. They then run four adversarial rejection-sampling rounds: sample several utterances per turn, keep those a frozen real-versus-simulated classifier rates most human, fine-tune on them and refresh the classifier. The relevance judge comes from a different model family than the generator, and every component runs on open weights.

5. How it was evaluated

Validity was tested on three TREC iKAT collections over ClueWeb22-B: 2023 (176 judged turns), 2024 (116) and multi-session 2025 (45). On each, a pool of twelve constructed retrieval pipelines (BM25, SPLADE, rank fusion, with and without a cross-encoder reranker, over raw, context-resolved or persona-stuffed queries) was ranked by AutoSim's satisfaction and compared with NIST's ranking by Kendall's tau. Two real organizer runs extended the 2023 pool to fourteen systems, and TREC CAsT 2020 over MS MARCO tested a second corpus. Realism was measured with held-out real-versus-simulated classifiers (an embedding model and a fine-tuned XLM-RoBERTa), with frontier LLMs and three human annotators reading full dialogues, and with retrieval overlap and conversational-move mix. Baselines were the default prompt, a hand-written terse prompt, an iterative prompt loop and few-shot prompting, run over twelve seeds. The bias test compared nDCG@5 from each simulator's queries with the real user's.

6. How it performed

Ranking agreement with NIST reached tau 0.66 on iKAT 2023, 0.65 with the real runs added, 0.91 on 2024, 0.91 on 2025 and 0.56 on CAsT. A persona-free judge scored 0.79, so validity comes from relevance judging, not the persona. On near-tied systems it fell to 0.33 on 2023, which was not significant. No prompt variant brought the classifier close to chance, but fine-tuning did, and the result replicated on Qwen2.5-7B and Mistral-7B. Where real users' own queries retrieve little, prompted simulators, including terse and length-matched ones, overstated nDCG enough to flip a deployment decision, while the fine-tuned simulator showed no detectable bias. Our extracted text lost many exact figures, so we cite only those that survived. The limits: organizer-written utterances serve as the reference for real users, only three annotators took part, and clicks, satisfaction and stopping were not modelled. Our call: pilot it in our assistant evaluation harness. (Source: arXiv 2610.28054, 2026)