Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations: what it means for business leaders

After a runtime upgrade, LLM-repaired code can run cleanly and still be wrong; adding one edge-case input from the release note to the pre-release check cut wrong repairs released from 41 percent to 6.5 percent.

1. What the method is

This paper is a preregistered measurement of release gates, not a new repair tool. It asks how often a pre-release check lets through an LLM-repaired procedure that runs cleanly but returns wrong results after a runtime or API upgrade. The authors built 73 tasks covering 60 documented changes in CPython 3.9 to 3.12, Node.js 20 to 24 and the SQLite shell 3.51 to 3.54, executed every candidate on the real runtimes, and judged it against contracts withheld from the repairing model. Their central finding is that a check fails mainly through its inputs. Adding one edge-case input derived from the release note to an all-assertions gate cut the share of wrong repairs released from 0.410 to 0.065. Three inputs from an isolated GPT-5.6-luna writer that never saw the program reached 0.070.

2. Why the method was developed

Some upgrades break loudly, with a removed function raising an error. Others break quietly: an enumeration renders as 3 instead of Priority.HIGH, sum() returns a more accurate float that no longer matches stored ledger totals, or Buffer.poolSize grows eightfold and changes how every upload is chunked. Existing dependency-repair systems mostly accept a fix when the client compiles or its own tests pass, which is the known overfitting problem in program repair. The closest system, BBCFixer, compares old and new execution as evidence for making a repair, not as a release criterion. The authors' own earlier study pointed to inputs as the weak point, but reviewers objected that the edge inputs were author-written, the repair space was a finite menu, no human signed off the contracts, and the test partition had only 34 clusters. This study fully answers the first and last objections and only partly the other two.

3. Who should care

Engineering and platform leaders who run runtime upgrades or dependency bumps with coding agents are the primary audience, because this failure produces no error and a green check. QA and test owners get a concrete change to their release checklist: for each documented change, add an input that exercises it, taken from the release note, with the expected output taken from the old program on the old runtime. ML leads choosing models should note that test-writing ability varied widely, with GPT-5.6-luna writing valid tests for 71 of 73 tasks and Qwen3-8B for 30, so a model that repairs adequately may not be the one to trust as a test writer. Owners of finance and data procedures have the most exposure, since a changed float total or rendered value can pass unnoticed. Nothing here touches model training or hosting.

4. How the method works

Each task pairs a fixed business requirement and an old procedure with a finite set of documented code fragments to choose from, a visible input set and a hidden one. The gate runs a candidate on the new runtime and applies every requirement assertion to every visible input. Inputs are the variable: typical only, typical plus one author edge input, typical plus up to three blind-writer inputs, or typical plus the repairing model's own self-tests. The blind writer sees the requirement, release-note excerpts, input keys, the typical input with its old output, and the output checks, but never the template, fragments, old program or hidden inputs. Its inputs are data only, and must run cleanly and deterministically on the old program, which supplies the expected output, so no model-written oracle is trusted. A candidate failing the gate gets one revision, with pass or fail feedback and no expected values.

5. How it was evaluated

The test partition holds 52 affected tasks in 49 change clusters, 15 of them new since the authors' earlier study, with the design frozen by SHA-256 hash sets before any repair call. Four model families repaired under equal information: GPT-5.6-luna through the Codex CLI, Qwen3-8B, gpt-oss-20b and qwen3.6-35b-a3b, with eight configurations per task and arm, giving 456 attempts per arm. A fifth, gemma-4-12b, failed its admission check on what the authors call a serving problem, so no conclusion attaches to it. Statistics use the change cluster as the unit, with sign-flip permutation tests, bootstrap intervals and Holm correction. Two later frozen stages enlarged the menu from 405 to 1491 fragments, then added 233 hard distractors that match legacy output on typical inputs. Two blind AI audits per contract stood in for human sign-off. Repairs are choices among author-written fragments; no model-written code ran.

6. How it performed

Across all possible repairs, the edge input cut wrong releases from 0.410 to 0.065 with false rejection at 0.010, and from 0.471 to 0.061 on the 15 unseen clusters. On model repairs, wrong releases per attempt fell from 0.233 to 0.057 for Qwen3-8B, 0.181 to 0.020 for gpt-oss-20b and 0.120 to 0.018 for qwen3.6-35b-a3b, without losing correct completion. Self-tests helped less and no contrast survived correction; a guard keeping only inputs the writer called legacy-binding reached 0.157. On the hardest menu, 0.168 under the edge input and 0.153 under the frontier writer of wrong programs matching legacy output on typical inputs still passed, and two local-model writers failed their tests. An AI audit flagged 34 of 73 contracts; dropping the disputed checks changed no verdicts, though class-label disputes remain. Rates depend on menu construction, free-form repair was never run, and our copy stops before the authors' threats section. Our call: pilot release-note edge inputs on our next runtime upgrade. (Source: arXiv 2611.08269, 2026)