When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP: what it means for business leaders

Five LLMs asked to modernize 100 legacy WordPress PHP files each finished with fewer working files than they started with, while the rule-based tool Rector kept almost all of them working.

1. What the method is

This paper is an empirical study, not a new model or tool. The authors gave five LLMs the same task: modernize a legacy PHP file to PHP 8.3 and keep its behavior. The models were Claude Sonnet 4, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-5 Codex and Llama 3.3 70B. Each model saw one whole file, with no project context and no execution feedback, and only its first complete answer counted. The new part is the scoring. Every output is compared with the unchanged original using 630 hidden behavioral checks across 100 WordPress 4.0 files. The 511 modernization findings that the rule-based tool Rector detects are split into changes PHP 8.3 requires, changes that only clear deprecation notices, and optional style changes. Measured this way, every model ended with fewer fully working files than leaving the code alone.

2. Why the method was developed

Developers often paste a legacy file into a chat assistant and ask it to modernize the code. The result looks like progress: type declarations, newer syntax, cleaner helpers. A modern-looking diff says nothing about whether callers still get the behavior they rely on. In one example from the paper, Gemini 2.5 Pro rewrote a diff method so that it throws an InvalidArgumentException where the original returned the PEAR error object callers expect, and nothing in the source flags the change. Existing migration benchmarks such as MigrationBench, RepoMod-Bench and JMigBench score the migrated code, but they never ask whether it beats leaving the code unchanged, and they count required and optional changes alike. In same-language modernization the original is itself a runnable baseline. The authors argue that leaving it out of the comparison hides real losses behind a good modernization score.

3. Who should care

Engineering leaders with old codebases in PHP or any language that has moved on are the first audience, because this is the shortcut their teams are already taking. Platform and developer-experience teams who set policy for coding assistants should read it as evidence that single-file, first-answer modernization needs a gate in front of it. QA and release owners get a clear failure profile: files that no longer load at all, and newly added type declarations that reject values the old code accepted. Procurement, and anyone evaluating vendors who sell AI-driven modernization, should ask for results against a no-change baseline, with preservation and modernization reported separately, rather than a single percentage of constructs updated. Product owners who fund modernization projects should know that most of the edits these models make were optional, and that the optional edits caused most of the damage.

4. How the method works

Every model got the same fixed prompt: act as a senior PHP developer, migrate the file to PHP 8.3 standards, and preserve inputs, outputs, side effects and call compatibility. Temperature was 0.2, except for GPT-5 Codex, whose API does not expose the setting. Incomplete answers were regenerated, and the first complete one was kept unedited. Behavior was defined by hidden PHPUnit checks, written by one author from WordPress documentation and the original code before any output was seen, then reviewed by a second author. The checks were 109 reachability, 441 output or state, and 80 implementation-sensitive fingerprint checks. They ran with all errors reported, and deprecation notices counted as failures. Each check was classed as preserved, regressed, repaired or still failing. To measure modernization, the authors reran Rector's rules on each output, and a classifier attributed each regression to its first recorded error.

5. How it was evaluated

The corpus is WordPress 4.0, from 2014, which has 482 PHP files. Rector 2.1.0 found modernization targets in 397 of them. The authors deliberately picked 100 files rich in outdated patterns, spread across four size bands (31 files up to 200 lines, 31 of 201 to 500, 26 of 501 to 1,000, and 12 longer) and covering all 48 rule types. Because the sample is purposive, the results describe these files, not PHP in general. There were two baselines: the unchanged originals, and Rector running one file at a time with no feedback, the same constraints the models had. Pairwise comparisons used exact McNemar tests with Holm–Bonferroni correction over 21 pairs, plus bootstrap intervals over 10,000 resamples. Two sensitivity analyses dropped the fingerprint checks or swapped back the original whenever an output failed PHP's syntax check. A replication package is public on Zenodo.

6. How it performed

Unchanged, 83 of 100 files passed every check. The best model, Claude Sonnet 4, left 59; GPT-5 Codex left 58, Gemini 2.5 Pro 55, Flash 50 and Llama 33. Rector kept 82. The models broke 32 to 51 working files each and fixed at most 10, against 7 broken and 6 fixed for Rector. Gemini 2.5 Pro made the most modernization changes (72.8%) without ending up more compatible. Only 37 findings (7%) were required, yet in working files that needed no required change, the models broke 36 to 60% against Rector's 8%. The largest single cause was files PHP refused to load, often because of invalid declarations such as a mixed return type on __set(). Falling back to the original when an output fails the syntax check recovers 8 to 15 files per model. Limits: one repository, one prompt, one generation per file. Our call: act now, and gate any LLM modernization behind load and behavioral tests run against the unchanged code. (Source: arXiv 2610.40269, 2026)