Interaction-as-a-Judge: what it means for business leaders

A computer-use agent that clicks through a generated app before a judge scores it predicts human preferences and catches broken features better than reading code or a screenshot, and the same signal improves repair loops and reinforcement learning.

1. What the method is

Interaction-as-a-Judge is an evaluation paradigm in which a computer-use agent operates a generated application in a browser and the recorded trace of screenshots, actions and reasoning is handed to a judge, alongside or instead of the source code. The judge is either an off-the-shelf LLM or a reward model trained on human preference pairs. The authors, from Carnegie Mellon, Arena, Berkeley and UCLA, give the first controlled study of the idea. They compare three interaction policies: iterative, where the model picks each action after seeing its effect; one-shot, where it plans every action from the first screenshot and the code; and heuristic, where random actions are sampled over the visible controls with no model calls. They release two datasets, ArenaPreferences, pairwise human votes on React websites across seven categories, and ArenaCorrectness, game-development traces with 323 broken and 89 working hand-verified requirements, and show the signal also improves repair loops and reinforcement learning.

2. Why the method was developed

Coding agents now produce whole interactive applications, and the people who use them judge them by using them. Automated judges still read the code, run unit tests or at most look at one static screenshot. That gap lets through failures any user would see in seconds: a game that compiles but whose player falls through platforms, a scene transition that freezes, controls with no visible response. Earlier work such as AppEvalPilot and WebGen-Agent used a computer-use agent as one component of a pipeline but never asked how the interaction strategy or the agent's capability changes evaluation accuracy, and WebDevJudge found a GUI-agent workflow actually underperformed strong static judges. Code-only reward models also carry a known length bias that rewards longer output. Finally, there are settings where we can run an application but cannot see its code, and a code-based judge is useless there.

3. Who should care

This touches anyone in our business who evaluates software that a model wrote. The ML platform team choosing reward signals for a coding or app-generation model should read it as a recipe for a reward that sees behavior rather than text. QA and test leads get an automated acceptance tester for generated user interfaces, which is the layer our unit tests do not cover. Product managers and anyone running a vendor comparison of coding tools should note that preference judgments change when the judge actually uses the app, so demo screenshots are a weaker basis for a decision than we treat them as. Finance and infrastructure should expect a new cost line, because every evaluation now includes a browser session and a model query per step. Our data team gets two public datasets of agentic web-development traces to benchmark against.

4. How the method works

Interaction is modeled as a partially observable decision process over screenshots. Each candidate runs in its own Chromium context under Playwright, and the agent gets a tool set of mouse clicks, drags, key holds, key sequences, waits and an early finish, with a budget of 30 steps. Games are paused between actions and a timing shim advances their clock by 500 milliseconds per action, so model latency never alters game time. The heuristic policy enumerates up to 200 visible controls and samples uniformly; for games it draws from twelve generic actions. For preferences, the LLM judge receives the request, both codebases and both traces and is run twice with sides swapped to cancel position bias. The reward model is Qwen3.5-9B with a scalar head trained under a Bradley-Terry loss, video fed natively at two frames per second with the vision encoder frozen. For generation, the judge verifies repairs inside a five-round self-feedback loop or supplies the reward for DAPO reinforcement learning.

5. How it was evaluated

Preference accuracy is measured on ArenaPreferences, one month of Code Arena battles from July 2026, filtered to React sites that build, differ by more than five percent of characters, and were used by the voter for 60 seconds to ten minutes. Baselines are a code-only judge and a code-plus-screenshot judge. LLM judges are Gemini 3 Flash and GPT 5.6 Luna; the computer-use agent is GPT 5.6 Luna at medium reasoning with 30 actions per trace. The reward model is trained on earlier, temporally disjoint private data over five seeds with paired t-tests. Requirement verification and discovery use ArenaCorrectness against a code-only Agent-as-a-Judge across GPT 5.6 Luna, Terra and Sol. Repair runs five iterations with Luna as the agent and Sol as final grader. Training uses DAPO on Qwen-3.5-9B with heuristic interaction, build-failure penalties and a 600-site test set over six categories, games excluded. The arXiv rendering we read dropped most headline figures, so we report only the numbers that survived.

6. How it performed

The direction is consistent in every setting, but our copy of the paper lost the headline percentages, so we state magnitudes only where the appendix kept them. On 3,327 gaming battles, interaction beat a code-only reward model by 1.39, 1.55 and 1.61 points at 4B, 9B and 27B parameters. Across 28 category comparisons, iterative interaction significantly beat code-plus-screenshot in 14 and lost in one, with the largest gains in simulations and games. GPT 5.6 Luna as a preference judge gained nothing from interaction, and trace text without video added almost nothing. The correctness win comes from catching broken requirements; both judges verified working ones comparably, and requirement discovery stayed hard for both. Under reinforcement learning the interaction reward cut build failures and code length, with a quality gain the authors equate to over 50 Arena leaderboard points, measured only on builds that succeed. Cost confined them to one agent model, and traces hit the context ceiling on about three fifths of full-source inputs. Our call: pilot it as an acceptance tester on our generated interfaces before wiring it into any reward. (Source: arXiv 2610.17895, 2026)