AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes: what it means for business leaders

Meta's AutoRefine recalibrates an LLM code judge's prompt from outcomes the workflow already records, using a teacher model that explains each label it keeps and discards the ones the judge's inputs cannot explain.

1. What the method is

AutoRefine is a loop from Meta that rewrites the prompt of an LLM code judge so that its verdicts reproduce the outcomes a development workflow already records: tests pass or fail, changes land or are abandoned. Its distinguishing part is an informed teacher, a second model call that reads exactly what the judge reads plus the outcome. The teacher writes a justification for each label it keeps, and abstains when the outcome cannot be explained from the judge's inputs, which drops that sample from calibration. The result is a judge prompt tuned without hand labeling, and a second output that matters as much: the teacher's abstention rate, which tells a team when the judge lacks the information to decide and needs better inputs, not a better prompt. No model weights are trained; only the prompt changes.

2. Why the method was developed

Coding agents now produce changes at volume, and much of that output, such as refactors, performance work and cleanups, has no test that proves it correct. Teams put an LLM judge in front of human reviewers to filter it, but a judge is only reliable once its prompt is fitted to the task, and that fit decays whenever the agent, the model or the codebase changes. Hand-labeled calibration sets are expensive and go stale, so many judge prompts are never validated at all. Workflow outcomes are free and continuous, but noisy: a change can fail on a hidden test the judge never sees, or be abandoned because it duplicated other work, was superseded, or was closed by a cleanup bot. A judge fitted to those labels learns noise. Earlier calibration methods such as AutoCalibrate and EvalGen depend on human labels.

3. Who should care

Engineering leaders running coding agents or automated code review are the first audience: this is a maintenance procedure for the gate that decides which machine-written changes reach our reviewers, and it prices that maintenance in model calls instead of labeling hours. Platform and developer-productivity teams would own it, since it needs recorded agent outputs and terminal review status from the code-review system. ML and evaluation teams should read it as a pattern beyond code, because the loop needs only the judge's view of an output and an accept or reject outcome. Quality and risk owners should note what the abstention rate offers: a measured statement of how many outcomes the judge could not have predicted from what it is shown. Anyone buying an agent product with a built-in judge can ask the vendor how, and how often, that judge is recalibrated.

4. How the method works

The teacher first reviews the training samples once, keeping labels with a justification, abstaining on those it cannot explain, and very occasionally correcting one (2 of 512 in the merge study). Each iteration then runs the current judge prompt on the kept samples and scores it with Scott's pi, an agreement measure corrected for chance, so a judge that always predicts the majority class scores at or below zero. For every disagreement, a model reads the judge's reasoning beside the label and the teacher's justification and proposes prompt edits; edits must be balanced, pairing each new reason to fail with a reason to pass, so the judge does not drift toward rejecting everything. A second call applies the edits under a length cap. An uncurated validation split picks the final prompt, and the loop stops after at most five iterations. Agent outputs are recorded once and replayed, so calibration never reruns the agent.

5. How it was evaluated

All experiments are from Meta's internal code. The first uses trustworthy labels: one agent attempted 579 tasks built from landed changes that have fail-to-pass tests, leaving 417 records (353 pass, 64 fail), split into 127 training, 85 validation and 85 test tasks, the test split holding 13 failures. Two judges that compare the agent's patch with the production reference patch, a five-check rubric judge and a pass/fail judge, were calibrated with the tests hidden; the baselines were the uncalibrated prompts, including one with its threshold tuned on the training split. A second study flipped 25% of labels at random to measure what the teacher removes. A third ran the teacher on 512 landed or abandoned diffs from a performance-optimization agent, then recalibrated that agent's judge on 125 training diffs and tested it on 17 current and 16 older held-out diffs. Engineers also reviewed 56 tasks blind to the outcome.

6. How it performed

With trustworthy labels the rubric judge's chance-corrected agreement rose from 0.36 to 0.60 and failure precision from 0.42 to 0.68; a threshold tuned on training data alone gave 0.33. On the deployed agent, agreement on the 17 independent diffs rose from 0.16 to 0.51 and landed-diff recall on 43 diffs outside training from 12 of 20 to 18 of 20, while the old judge stayed ahead on the 16 older diffs. The teacher cut injected noise from 24.9% to 20.4% and abstained on 60% of diffs abandoned after review, but lets failures labeled as passes through. Without a reference patch the teacher abstained on 12 of 13 failures and human reviewers missed nearly all of them: no prompt fixes missing inputs. Samples are small, from one company and one agent, each loop ran once, and judge and teacher share a model family. Our call: pilot it wherever we gate agent-written changes with a judge and already log merge or test outcomes. (Source: arXiv 2611.09291, 2026)