UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users: what it means for business leaders

A Tencent Hunyuan benchmark tests coding agents in multi-turn bug-fixing conversations with realistic, shifting simulated users and runs the tests after every turn. Which agent looks best depends on how many turns we allow.

1. What the method is

UserSWE is a benchmark from a Tencent Hunyuan team. It measures how well a coding agent fixes real repository bugs over a conversation, rather than from one complete bug report. It has three parts. First, a construction pipeline builds multi-turn tasks in two ways: it mines repository histories for pairs of issues that a maintainer explicitly linked, or it restages tasks from existing single-turn benchmarks. Second, a simulated user speaks in one of eight personas. These cross two choices: whether the symptom is stated clearly or vaguely, and whether the user offers no cause, the right cause, a plausible wrong cause they drop when argued with, or a wrong cause they hold until shown evidence. Third, an evaluation framework runs the tests after every turn. A model from one family writes each verdict and a model from another family clears it. The released set is 72 tasks from 42 repositories.

2. Why the method was developed

Most coding benchmarks, SWE-bench and its successors, hand the agent a complete issue and grade a single patch. That is not how our engineers use these tools. They open with an incomplete complaint, sometimes guess the wrong cause, and clarify over several turns. The few multi-turn benchmarks that exist keep the simulated user in one fixed voice, usually precise and correct. The authors show this voice is unrepresentative. In 13,530 code-changing sessions from a commercial coding assistant, a clear symptom with a correct cause opens only 19.03% of sessions, and only 0.72% stay that way for five turns. Grading is the second problem. Tests written around the maintainer's own fix reject correct patches that use a different interface. Weak tests accept patches that fix nothing. Unrestricted LLM judges draw conclusions the evidence does not support. A benchmark for conversational debugging had to fix both problems.

3. Who should care

Engineering leaders choosing a coding assistant for their teams should care most. The paper shows that the vendor ranking we would read off a single-turn leaderboard changes with how many back-and-forth turns we allow. Platform and developer-experience teams that run internal evaluations get a design they can reuse: frozen information pools, a persona sequence drawn in advance so every agent meets the same user, and an execution check after every turn. Teams building their own agents or fine-tuning models get a clearer signal on whether an agent asks good questions and recovers from a misleading user. Anyone running LLM-as-judge pipelines, in coding or elsewhere, should note how often a second reviewer from another model family overturned the first judge. Procurement should treat a vendor's single-turn score as one point on a curve, not the whole picture.

4. How the method works

Each task carries an information pool generated from the issue and the gold patch, the maintainer's own fix. It holds a clear symptom, a vague one, a correct direction for the cause and a plausible wrong one. A cascade also carries follow-up material for its second issue. A check rejects the pool if it names any identifier the fix introduced. After every agent turn, the tests run in a container. The simulated user then picks what to say in a fixed order: answer a direct question; otherwise report the most blocking visible failure, then an uncovered core requirement, then an edge case. A persona sequence sets how each turn is phrased. It is drawn per task from turn-to-turn transition rates measured on 20,859 logged sessions. A Judge writes the turn's verdict and the next message, and a Reviewer from another family may rewrite them. If a test is tied to the gold patch's helper names, the agent can earn credit through a rewritten test. The rewrite must pass on the agent's code and still fail on the original code.

5. How it was evaluated

The authors tested eleven agents, each a model running on a scaffold, on 72 tasks. Twenty-two are two-issue cascades mined from repository history and 50 are single-issue tasks converted from published benchmarks. Cascades got a larger turn budget than single-issue tasks. The Judge and Reviewer roles rotated between GPT-5.6-Sol on Codex and Claude Opus 5 on Claude Code, swapped between tasks. The paper reports three measures. The first is the share of tasks resolved within one to eight turns. The second is the mean number of weighted turns, where a turn without a code edit counts as half. The third is a length-discounted pass rate (LDPR), which rewards finishing sooner and scores zero for a task not solved within budget. A separate set of 42 single-turn tasks from SWE-bench Pro and SWE-bench-Live gave a conventional comparison for the top four agents. An ablation reran the 50 single-issue tasks with a user who stayed clear and correct throughout. Every task's simulated user turns passed human review, about 50 hours in total.

6. How it performed

Rankings depended on the turn budget. Claude Fable 5.1 resolved the most tasks in one turn, 31.9%, and led the single-turn set at 78.6%. Yet Claude Opus 5 led on LDPR at 0.640, resolving 80.6% of tasks in an average of 2.99 weighted turns. GPT-6-Astra resolved the most by turn eight, 81.9%, but placed fourth on LDPR at 0.561. With a user who stayed clear and correct, three of four agents scored higher, by 0.020 to 0.076 LDPR. Idealized users flatter some agents more than others. The second reviewer rewrote 40.9% of Judge decisions, and 51.2% of requests for credit through a rewritten test were granted. The limits are real. There are only 72 tasks because sessions are costly, each covers one or two issues, and the session logs are proprietary. Fable's refusals count as failures, and the judges' model families also appear on the leaderboard. Our call: pilot this protocol on our own repositories before our next coding-assistant decision. (Source: arXiv 2610.29822, 2026)