Reliable AI-Assisted Decisions via Selective Auditing: what it means for business leaders

A stream of AI decisions can be spot-checked by a human or stronger model so that no more than a chosen fraction of the AI's mistakes slip through unreviewed, with the audit rate driven down as low as the data allow and no assumptions about the AI or its confidence scores.

1. What the method is

The paper from Gibbs, Han, Tibshirani and Zrnic defines selective auditing: an AI system emits a decision and a confidence score each round, and a policy decides whether to accept the decision or send it to an auditor who corrects it if wrong. The target is a cap, set by us, on the fraction of the AI's errors that go unaudited, while auditing as rarely as possible. They give three algorithms. Additive thresholding moves a confidence threshold up or down by a fixed step as audited errors arrive, auditing everything below it plus a small random spot-check. A second version recalibrates the confidence scores online before thresholding. The third, a primal-dual method, is proven optimally efficient: it reaches the best audit rate any valid online policy can achieve, on adversarial streams and on independent, identically distributed streams alike. The error guarantee holds for any data sequence and any confidence score, calibrated or not.

2. Why the method was developed

Nobody can review every output of an agent or classifier at production volume, so the question is which outputs to escalate. Earlier work on learning to defer or reject framed this as a classifier trained with a loss function, which means the reliability of the escalation rule depends on how well that classifier generalizes, and firm guarantees need strong assumptions. Calibration-based alternatives mostly assume an offline setting where training and test data come from the same distribution, which fails when decisions arrive sequentially, the input mix drifts, or the AI itself adapts to feedback from past audits. The closest prior method, from Kiyani and colleagues, controls the same error criterion online but was never analyzed for efficiency, and the authors show it audits more than necessary. The paper's contribution is an online guarantee with no distributional assumptions and a matching proof of how few audits are enough.

3. Who should care

This touches any team that runs a human-in-the-loop or model-in-the-loop review step over AI output. Trust and safety leads can allocate reviewer time on LLM responses using an automated judge as the confidence signal. Data and annotation managers can decide which model-proposed labels a human must verify and how many annotator hours a dataset will cost. Product owners of agentic systems can set the oversight budget as one number, the tolerated fraction of unreviewed errors, rather than a hand-tuned confidence cutoff. Risk and compliance can state a measurable oversight guarantee that does not depend on the model being calibrated. Platform and MLOps engineers would implement the policy as a thin online layer: it needs only the confidence score, the audit decision, and the auditor's verdict when one was requested. Finance should note that the audit rate, hence reviewer headcount, becomes a tunable output.

4. How the method works

Additive thresholding keeps one threshold. Each round, if the confidence is below the threshold the item is audited; otherwise it is audited with a small fixed probability. The threshold then shifts by a step size times a quantity whose average the method drives to zero: the audited error, reweighted by the inverse of the probability it was observed, minus the target fraction. This inverse-probability weighting corrects for only seeing errors when an audit happens. The calibrated variant learns a monotone step function from raw scores to estimated correctness using exponentiated gradient descent on a grid of thresholds, again with inverse-weighted gradients, and feeds the recalibrated score to the same threshold rule. The optimally efficient method maintains weights over candidate thresholds with AdaHedge in the primal and a dual variable tracking a regularized estimate of cumulative constraint violation, updated only from audited rounds. Batched updates and total-error or false-negative-rate variants follow the same pattern.

5. How it was evaluated

Three settings were tested, each with a fixed stream and only the audit randomness resampled across trials. For safety verification the authors used SORRY-Bench and StrongREJECT, both with human safety labels, with GPT-5.4 as a weak judge whose confidence came from self-consistency across repeated refusal-or-fulfillment votes. For qualitative coding they built an annotation pipeline over eight UCSB interview transcripts, with Qwen2.5-14B or GPT-4o-mini coding each sentence and emitting a verbalized confidence, GPT-5.2 standing in as the auditor, and a second refinement pass doubling the rounds. For data labeling they used GPT-4o annotations with verbalized probabilities on political-bias and climate-stance text classification, plus ImageNet and ImageNet-V2 with maximum softmax probability as confidence. Baselines were random auditing at the target rate and an oracle fixed threshold chosen in hindsight. Reported metrics were the unaudited-error fraction and audit effort saved, both as percentages.

6. How it performed

The rendered text we worked from lost the numeric values in the result figures, so we report direction rather than magnitude. Across all tasks the realized error fraction tracked the chosen target closely. On StrongREJECT, whose obfuscation jailbreaks defeat automated judges, the calibrated and optimal methods ran slightly above target, and the optimal method was somewhat anti-conservative on ImageNet. On savings, the optimal method and the calibrated variant consistently beat plain additive thresholding, and the optimal method often approached the hindsight oracle, notably with GPT-4o-mini as annotator. The authors note the guarantees weaken when errors are rare, that the error bound converges slower than the usual parametric rate because feedback arrives only on audited rounds, and that the method assumes a perfect auditor, uniform audit cost and a scalar confidence signal. Our call: pilot it on one review queue where we already log reviewer verdicts and a confidence score. (Source: arXiv 2610.31662, 2026)