ProductWebBench: Evaluating Website Continuity Beyond Task Completion: what it means for business leaders

A benchmark of 400 real frontend repository tasks shows that when coding agents complete the requested change, more than a quarter of the time they break something the site already did.

1. What the method is

ProductWebBench is a benchmark that grades a frontend coding agent on two obligations at once: did it build what was asked, and did the rest of the site keep working, which the authors call website continuity. It holds 400 tasks on real, runnable frontend repositories: 320 Change tasks that modify existing code, each organized around one facet where continuity can break, and 80 long-horizon Build tasks that grow a product site through 5 to 14 ordered milestones. Every task is a frozen package of repository snapshot, allowed paths, stage plan, captured browser evidence and 19 deterministic checks, admitted only once a reference patch passes every check and authored plausible-wrong patches fail. A simulated user reveals requirements one stage at a time, and every check a stage clears stays armed for the rest of the run, so a later edit that breaks an earlier state is recorded as a regression with exact provenance rather than a guessed label.

2. Why the method was developed

Frontend agents have so far been graded on one question, whether the requested target got built, and that misses most of what a real change costs. Adding a cart control touches shared state, checkout flows, responsive layouts and components on other routes, so a patch can satisfy its own DOM assertion while blanking a route or throwing console errors elsewhere. Existing benchmarks each cover a piece. Frontend generation suites such as Design2Code measure whether an interface gets produced, not whether an existing application survives. SWE-bench runs executable tests on repository changes but has no browser evidence about viewports, interaction histories or shared components. Browser-agent suites such as WebArena operate on live sites but never modify the source behind them. None evaluates incremental development, where accepted changes become the ground later ones build on and a later patch can silently invalidate an earlier state. The authors built the benchmark to measure that second obligation directly and with provenance.

3. Who should care

Our engineering leads deciding how far to let coding agents into frontend work should read this first: it says completing the ticket is not evidence the site still holds, and the failures are ordinary product health, overflow, dead routes, console errors. QA and release owners get a concrete pattern to copy, replaying every previously passing browser state after each agent edit and diffing against the untouched repository. Platform and tooling teams should note that the scaffold around a model moved scores as much as the model itself, so our harness is part of what we are evaluating. Procurement and anyone comparing vendor claims should treat single-number leaderboards with caution, since the two headline metrics here disagree at the top. Product managers should understand why staged, incremental requests cost more than they look. For any customer buying agent-built web changes from us, continuity is the promise that matters.

4. How the method works

Authors first run the repository and capture browser states keyed by route, viewport and action prefix, then write the request against that evidence, including which untouched states must survive. Admission applies the reference patch and the negative controls to the frozen package. At evaluation a persona, PM, QA or designer, reveals one authored work unit per stage. The solver gets scoped source and returns structured edits, and the harness applies them to a cumulative workspace, builds, serves, captures states and runs the checks, allowing 12 turns per stage and 200 per run. A cleared stage's assertions join a protected set replayed at every later checkpoint. The 19 gates split into 12 completion signals, six conservative continuity signals and one broad visual-regression signal held out. A second channel runs the same suite on the untouched repository, so any gate it passed and the final workspace fails is a regression with provenance. A graded continuity score keeps partial progress and penalizes each break.

5. How it was evaluated

The authors ran 13 coding models, closed ones such as Claude Opus 4.8 and GPT-5.5 and open ones such as DeepSeek-V4, Kimi-K3 and GLM-5.2, under the staged protocol from pristine repositories with identical budgets and no access to the reference or controls. After dropping verdicts older than their task's frozen specification, 8,737 cells remain, 6,979 Change and 1,758 Build, and every number comes from that one manifest, and nine models also have paired screenshot runs. A separate full-spec arm handed eight agent scaffolds, among them Claude Code, Codex, mini-swe-agent, Goose and Aider, the whole task at once on two backbones, and the authors keep it apart as an internal comparison, not a second leaderboard. Verifier validity was checked three ways: all 479 references pass and all 479 smart-wrong controls fail, a blind 720-verdict audit against model raters, and 15 replayed pairs that put a 0.093 noise floor on the graded score while the binary verdict held every time.

6. How it performed

Whole-task passes are rare: 198 of 8,737 cells, Claude Opus 4.8 ahead while GPT-5.5 leads Gate% at 61.71. The central finding is the gap: of 674 Change runs passing every requested-work check, 190, or 28.2%, failed a continuity check, 73 pushed a state they had already passed back to failing, and on the 586 runs with an untouched-repository baseline 18.3% were caught damaging a verified state, 13.3% still broken at the end. Interaction failed in 29.5% of runs whose static DOM passed. Swapping the scaffold moved scores as much as swapping the model, and a do-not-break-unrelated-behavior clause lifted Gate% while screenshots and user wording moved nothing. The authors caution that the graded score carries a 0.093 per-cell noise floor, the continuity false-fail audit rests on seven verdicts, and the baseline covers 234 of 320 Change slots. Our call is to pilot: add protected-state replay and an untouched-baseline diff to the acceptance gates on one agent-assisted frontend surface. (Source: arXiv 2610.31118, 2026)