Training User Simulators with Latent Intrinsic Feedback: what it means for business leaders

User simulators fine-tuned on real chat logs now emit a private rating of the assistant every turn, and that rating trains assistants as well as an external judge model at less than half the compute.

1. What the method is

The paper trains user simulators that, before every simulated user turn, write out a latent evaluation state: a short first-person thought about the assistant's last reply, a satisfaction rating from 1 to 5, whether the instruction was followed (yes, no, partially), a reaction label (critique, praise or neutral) and a two-to-five-word feedback summary. The simulator then writes the user's next message conditioned on that state. The models are Gemma3 base models at 4B, 12B and 27B parameters, with a Qwen2.5 replication, fine-tuned on real ChatGPT interaction logs from WildChat where the evaluation states were synthesized by Gemini 3.1 Pro. The advance is that the simulator now does two jobs at once: it imitates a user and it scores the assistant from the user's seat, so the rating can serve directly as a reward when training an assistant with reinforcement learning, with no separate judge model.

2. Why the method was developed

Assistants that look good on single-turn benchmarks still lose the thread in real conversations, and one cited analysis found users pushing back on assistant output 44 percent of the time. Training assistants to cope requires a realistic simulated user, and increasingly a multi-turn reinforcement learning loop in which that user supplies the reward. Earlier simulators came up short in two ways. Prompted ones, where an instruction-tuned model role-plays a user, drift from how people actually write. Fine-tuned ones imitate real utterances well but only generate text, so a separate judge model has to read every turn to estimate satisfaction. That judge is expensive, runs at every step of training, and often disagrees with how the user would have felt, partly because people frequently do not say that a reply fell short; they rephrase and move on. The authors wanted the assessment that drives that behaviour made explicit and reusable.

3. Who should care

This touches any team that trains or tunes a conversational assistant on its own traffic. Applied ML groups doing reinforcement learning from simulated conversations get a cheaper reward source and a more realistic environment. Evaluation and QA teams get a per-turn satisfaction score and a critique flag they can run over candidate assistants before a release, instead of waiting for live ratings. Product owners get a pre-launch proxy for user satisfaction, with the caveat that it is a model's guess. Infrastructure and finance should note the compute claim: removing the judge cut reinforcement learning cost by more than half in the authors' setup. Data governance and legal are implicated because the recipe is trained on real user-assistant logs; for us that means our own conversation data, with consent and retention terms that allow it. Vendors selling us assistants will start citing simulator scores, and we should ask how those simulators were built.

4. How the method works

Training is supervised fine-tuning of a base model on flipped logs, so the model learns to produce the user side. Each training example for turns after the first contains a JSON block between eval delimiters holding the five sub-states, followed by the real user utterance; the model is optimised on both. A one-line statement of the user's overall intent, also generated by Gemini, is injected as a system prompt so the simulator can be pointed at a task. The data is 129,897 cleaned, deduplicated English WildChat conversations yielding 238,636 evaluation states, of which 14 percent are critiques and 1.3 percent praise. For assistant training, a Qwen2.5-1.5B-Instruct model talks to the Gemma3-4B simulator over multiple turns on 100,000 intents and is trained with multi-turn GRPO to maximise the average per-turn rating the simulator emits. Because the rating comes out of the same forward pass as the user's reply, each turn costs one inference instead of two.

5. How it was evaluated

Simulator quality was measured on a held-out set of 4,000 WildChat conversations, half chosen to contain at least one user critique, giving 9,484 scored turns. Feedback behaviour was scored two ways: an F1 for whether the simulator critiques at the same turns real users did, with a Gemini judge classifying the generated text, and Spearman correlation between simulator ratings and the Gemini-labelled ground truth. We should keep in mind that this reference is itself synthetic. Realism was a ten-statistic alignment score against human text plus perplexity on the real utterances. Baselines were prompted Gemini 3.1 Pro, prompted Gemma3 instruction-tuned models, and plain fine-tuned simulators without latent states. Real human judgment entered through SimulatorArena, 909 conversations with end-of-conversation ratings from actual users on math tutoring and document tasks. The assistant-training claim was tested on WildBenchV2, 1,024 multi-turn tasks with objective checklists, judged by Gemini and averaged over three sampling runs.

6. How it performed

At 12B the latent-state simulator reached an F1 of 73.0 against 67.1 for the prompted model, a rating correlation of 0.43 against 0.37, and perplexity of 3.1 against 13.4. Plain fine-tuned simulators edged it on F1 because latent states make the model more cautious about raising a critique, but they emit no rating at all. On real human ratings the latent-state simulators led at every scale, 0.38 versus 0.24 at 4B and 0.40 versus 0.30 at 12B, though the 27B gap was not significant. Assistants trained against the simulator's own rating scored 41.4 percent checklist satisfaction, versus 41.1 with an external 14B judge and 28.8 before training, at under half the estimated FLOPs. The authors concede the accuracy gains are small and the correlations modest, and the recipe needs a large labelling model and real logs. Our call is to pilot: build one simulator on our own logs and compare its scores with live ratings before trusting it as a reward. (Source: arXiv 2610.13617, 2026)