Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers: what it means for business leaders

SPLIT uses a small labelled pilot and an exact success formula to decide how many samples to draw, which models to mix and how much to verify, landing within a fraction of a point of the best allocation where fixed rules lose up to 20.

1. What the method is

SPLIT is a planner for inference-time compute in systems that generate many candidate answers and submit the one a verifier prefers. It rests on an exact formula for the probability that the submitted answer is correct under two selectors used in practice, a uniform pick among accepted candidates and the top-scoring candidate. The formula accounts for three things fixed rules ignore: tasks differ in difficulty, different models fail on the same problems, and repeated calls to the same verifier repeat the same mistakes. SPLIT fits the formula's ingredients from a labelled pilot of 8 to 64 samples per task by empirical Bayes, then searches a grid of configurations, generators and mixing weights, sample counts, verifier calls or acceptance thresholds, for the one with the highest predicted success under a cost budget. The paper also proves when majority voting across LLM judges lowers attainable precision and shows real judges meet that condition on one benchmark and not another.

2. Why the method was developed

Drawing more samples raises coverage, the chance that at least one candidate is correct, smoothly and predictably. But a system is graded on the single answer it hands in, and that answer is chosen by an imperfect verifier: unit tests, a majority vote, an LLM judge or a reward model. Real verifiers accept wrong answers and reject right ones, and their errors are correlated, so verified success stalls far below coverage. On MATH, majority voting over Llama-3-8B-Instruct samples sits near 43% while coverage reaches 98%. Practitioners today split a budget between sampling, model choice and verification by rules of thumb such as use the strongest model or ask more judges. The author shows each rule is badly wrong somewhere: the right split changes with budget, models and verifier. Earlier compute-optimal work fitted power laws to one setting and modelled generation, not selection, so it did not transfer to new verifiers or budgets.

3. Who should care

Any team we run that puts a best-of-N loop in production touches this: coding agents that draft several patches and keep the one an LLM reviewer scores highest, retrieval and extraction pipelines with a judge model, math or reasoning features with self-consistency voting. Platform and inference engineers own the budget split and can replace heuristics with a calibrated plan. Evaluation and quality owners should note the finding that verifier error must be re-measured whenever the generator changes, since stronger generators produce wrong outputs that pass fixed tests more often. Finance and procurement care because the planner stays near-optimal under per-token prices and latency, not only raw compute, and online early stopping cuts spend 5 to 25% at equal accuracy. Anyone relying on juries of LLM judges for safety or compliance decisions should read the result that adding judges can raise false acceptance.

4. How the method works

Given a task, each generator's sample is correct with some probability, and the verifier accepts correct candidates at a true positive rate and wrong ones at a false positive rate, both possibly task dependent. Under conditional independence of candidates given the task, the author derives the exact probability that a uniformly chosen accepted candidate is correct; it rises with samples and converges to the verifier's precision, which is the ceiling. Task difficulty gets a beta or non-parametric prior fitted to pilot counts. Shared blind spots between generators are a Gaussian copula with one correlation per pair, fitted by maximising the pilot's bivariate likelihood. Repeated judge calls on one candidate share a per-candidate acceptance propensity drawn from a beta distribution whose concentration is fitted from the spread of verdicts. For scored verifiers, class-conditional score distributions are binned with a per-task random intercept. The planner evaluates the formula for each configuration and returns the best; because it only needs to rank configurations, its regret is bounded by differential rather than absolute forecast error.

5. How it was evaluated

Everything uses publicly released samples and verdicts; no new API calls were made. Seven benchmarks cover math, science QA, code and software-engineering agents: MATH and CodeContests from the Large Language Monkeys release with up to 10,000 samples per task, HumanEval and MBPP across 17 and 10 models with the original tests as verifier and EvalPlus as truth, MATH-500 and GPQA with eight LLM judges and 19 reward models from the Weaver release, 492 SWE-Bench Verified issues with 22 R2E-Gym agent rollouts each and an execution-free patch verifier, and a pool of 175 submitted SWE-bench systems. For CodeContests the author re-executed 80,403 programs locally. The allocation protocol calibrates on half the tasks from a 32-sample pilot, plans per budget, and scores the choice on the other half against the hindsight-optimal configuration. Baselines are strongest model, most accurate per cost, uniform ensemble, an adaptation of the Singhi et al. compute-optimal rule, generation only, maximum verification, a fixed threshold and SPLIT with independence assumed. Paired Wilcoxon tests back the headline comparisons.

6. How it performed

On unseen tasks SPLIT's best variant lands within 0.02 to 0.65 points of the hindsight optimum and every variant within 0.8, while each fixed rule loses at least 2.5 points somewhere: strongest model and the Singhi rule lose about 12 points on MATH at small budgets, generation only loses 2.6 on GPQA, and a fixed 0.5 threshold on SWE-bench cuts single-rollout resolve rate from 32.5% to 25.1%. Assuming independent errors multiplies regret 1.2 to 4.8 times. Regret stays between 0.45 and 0.62 under five pricing models, and online early stopping with a pooled budget matches the best static plan in hindsight. The author is candid that as a point forecaster SPLIT beats power laws clearly only with simulated verifiers and SWE-bench thresholds, by 0.4 to 0.6 points with real tests, and loses on pure coverage and argmax selection. Limits include a 16-rollout same-agent horizon, the need for a pilot representative of deployment, unmodelled duplicate candidates, and a vacuous finite-sample guarantee. Our call: pilot it on one production best-of-N pipeline where we already pay for a verifier. (Source: arXiv 2610.23255, 2026)