Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents: what it means for business leaders

Failure predictors for coding agents mostly recognize the issue and the model rather than the run, which undercuts the evidence for stopping, switching or retrying on their alarms.

1. What the method is

This paper offers no new predictor; its advance is a way of testing whether failure predictors for coding agents learn anything about the run in front of them. It separates three causes of failure: a hard issue that most models fail, a weak model that fails many issues, and a bad run, in which the same model solves the same issue on other attempts. Three tests that need only existing data make the split measurable: comparing a predictor's within-issue AUROC against run-blind reference scores, comparing the gap between the best single model and an any-model oracle against a simulated world with no specialization, and judging an alarm by whether the action it triggers fixes the failure. Applied to public trajectories, the tests find that reading the trajectory adds nothing measurable beyond issue difficulty and model ability.

2. Why the method was developed

Agent runs are long and often fail, so a growing body of work reads a partial trajectory and then stops the run to save tokens, restarts it, or hands the issue to another model. Each action suits only one cause: stopping or a human for a hard issue, a stronger model for a weak one, and a retry only for a bad run. The authors argue that current practice rests on three untested assumptions: that a high AUROC means run-specific learning, that the oracle gap reflects strengths switching can exploit, and that an alarm marks a run a retry can fix. Standard evaluations cannot test these, because scores pooled across issues and models blend difficulty and ability into one number, and leaderboard data gives each model one attempt per issue, which makes a bad run indistinguishable from the other two causes.

3. Who should care

Engineering leaders and platform teams running coding agents should care first, because any plan to stop, retry, or reroute runs on a predictor's alarm depends on what that alarm means. Whoever owns model routing and agent spend should read the switching results before funding a per-issue router, since the case for one usually cites the oracle gap, and should compare escalation and retry policies at equal numbers of attempts. ML and evaluation teams are touched most directly: we would need to report within-issue scores beside pooled ones and collect repeated runs, which cost more than a single leaderboard pass. Anyone buying agent tooling can use the same tests in procurement, asking a vendor claiming early failure detection what its score adds beyond issue difficulty and model identity, and whether its gains hold on held-out runs.

4. How the method works

The predictors copy the design of an earlier system, Fail-Fast, but see neither the issue text nor the model name. They read the trajectory after a set number of steps, either the last eight steps or the full prefix, and output a failure score, using TF-IDF logistic regression with issue-grouped five-fold cross-validation, plus fine-tuned Qwen3-0.6B models as a check. Run-blind references set the bar: issue difficulty is the fraction of 16 outside models that fail the issue, a diagnostic built on outcomes unavailable at deployment, and model identity is the model's failure rate on training issues. AUROC is then split into cross-issue and within-issue pairs, and the latter into consistent pairs, where the weaker model failed, and upsets, where it won. A predictor tracking the run should beat chance on both. Rasch and two-parameter logistic simulations supply a world where every issue ranks the models identically.

5. How it was evaluated

The main data are public mini-swe-agent runs on the 500 issues of SWE-bench Verified, where each model attempts each issue once: 3,492 runs with known outcomes from seven models, including Claude Opus 4.5 and 4.6, Gemini 3 Flash, and GLM-5, of which 994 failed. Most results cover the 2,840 runs still active at step 20, since only those can be acted on. A companion release supplies outcomes for 23 models to test breadth. Eighteen predictor configurations entered equivalence tests, Holm-corrected over 50 comparisons, and a conditional logit tested within-issue value with issue difficulty held fixed. Because single-run data cannot separate specialization from luck, a pilot used a public SWE-agent corpus: two Llama models, each run at least four times on each of 522 issues, with runs split into halves to choose and then measure, averaged over 200 random splits with bootstrapped intervals.

6. How it performed

Difficulty alone reached 0.92 pooled AUROC against 0.62 to 0.63 for trajectory predictors, because 99.66% of scored pairs compare different issues. Within an issue, model identity reached 0.72 against 0.60 to 0.61, and predictors scored 0.42 on upsets, below chance. In all 18 configurations the pooled gain beyond both was statistically equivalent to zero. The 10.0-point oracle gap was no larger than chance produces, and 45% of failures fell on 64 issues no model solved. In the pilot, run-to-run variation explained 42% of outcome variation against 4.6% for specialization, so retries have room that alarms cannot target. Escalating every failure to one fixed second model gained 4.67 to 9.15 points (95% interval), assuming a perfect detector. The authors note one benchmark and scaffold, older pilot models, and that non-rejection does not prove specialization absent. Our call: act now on these tests, and fund no alarm-driven routing or retries until a predictor beats run-blind references on repeated runs. (Source: arXiv 2611.08285, 2026)