CascadeFilter: Cost-Efficient Semantic Filtering with Joint Threshold Selection: what it means for business leaders

A three-stage LLM filtering cascade picks its two routing thresholds together, choosing the cheapest pair whose agreement with a frontier model passes a statistical certificate, and reports higher benchmark accuracy than seven baselines in 12 of 13 settings.

1. What the method is

CascadeFilter is a three-stage method for LLM semantic filtering, which means selecting the records in a corpus that satisfy a natural-language condition such as whether a review mentions a billing issue. Stage 1 runs a small open model (Qwen3-4B) on short passages pulled from each document. Records it is unsure about go to Stage 2, where the same small model reads the full document. Anything still unresolved goes to a frontier model, GPT-5.5, which the paper calls the oracle. The new part is how the two acceptance thresholds are chosen. Earlier methods tune each stage on its own or shift the thresholds along a fixed sequence after the cascade is built. CascadeFilter instead searches pairs of thresholds together over a grid fixed in advance, prices each pair by its modeled token cost, and picks the cheapest pair that passes a statistical test bounding its disagreement with the oracle. The test needs only a limited sample of oracle labels.

2. Why the method was developed

Running a frontier model on every document means paying for the full input and the output each time, so cost grows with both corpus size and document length. Cascades such as FrugalGPT, RouteLLM, BARGAIN and Task Cascades already send easy records to cheap models and hard ones to the oracle, and Task Cascades already offers an accuracy guarantee. Its calibration, however, moves the thresholds together along a prescribed sequence after the cascade is assembled. It never compares threshold combinations by what they cost. The two thresholds interact. Loosening the first one accepts more records cheaply but uses up error budget, which the second stage must make up by sending more records to the oracle. Whether that trade saves money depends on which records move between stages and how long they are, and tuning each stage separately cannot answer that. The authors also wanted a guarantee that stays valid after the selection itself, which testing each candidate on its own does not provide.

3. Who should care

The main audience is data and analytics platform teams that run LLM classification over large text collections such as support tickets, reviews, transcripts, claims or audit logs. For them this is a recipe for cutting frontier-model spend while keeping a stated level of agreement with that model. ML engineers who own inference budgets get a cost model that counts document length and token prices, not just the number of calls. Finance should note that the method shifts spend from per-token API fees toward small-model GPU time we host ourselves, and that the paper's cost figures are nominal, not audited bills. Product and risk owners need to know exactly what the guarantee covers. It covers overall agreement with the frontier model on one specific corpus. It does not cover correctness against ground truth, and it does not cover precision or recall for individual classes, so rare-category filters such as abuse or fraud flags need their own checks.

4. How the method works

Each document is split into overlapping sentence windows. Keyword (BM25) and embedding retrieval results are merged with reciprocal rank fusion, then reranked to keep compact evidence. Twenty to forty evidence snippets are batched into each small-model call. Every record gets a routing score that combines the model's token-level confidence with how much its neighbors in embedding space agree with its prediction. Stage 1 and Stage 2 predictions are computed once and cached, so every threshold pair can be priced from the cache without rerunning any model. Candidate thresholds are quantiles of the scores, up to 32 for the first stage and 64 for the second. A uniform random sample drawn without replacement is labeled by the oracle. For each first-stage threshold, second-stage thresholds are tested from strict to lenient with a finite-population betting test, stopping at the first failure. The allowed failure probability is split across these sequences. The cheapest pair that passes is deployed, and if none passes, every record goes to the oracle.

5. How it was evaluated

The authors used six datasets in 13 settings. One is a Yelp benchmark they built: 50 aspect queries over 2,000 reviews, giving 100,000 records, labeled by three LLMs with only the records where all three agreed kept. The others are Codebase user profiles, IMDB reviews, AirDialogue outcomes, Twitter hate speech and FEVER claim verification, with populations from 9,400 to 361,800 records. The seven baselines were running the small model on everything, LOTUS, BARGAIN, ScaleDoc, two clustering-and-voting methods, and Task Cascades with its guarantee extension. All used the same small model and oracle, and each baseline was run at a single operating point. To check how often the certificate fails, calibration was repeated over 20 random draws per setting. Ablations removed each stage, neighbor support, rank fusion, the reranker, joint search and the sequential testing scheme, and the small model, embedding model and reranker were swapped for other sizes and families. Small-model cost used a nominal token price, not measured GPU cost.

6. How it performed

CascadeFilter had the highest benchmark accuracy and macro-F1 in 12 of 13 settings. It averaged 0.938 accuracy and 0.887 macro-F1, against 0.900 and 0.830 for Task Cascades and 0.894 and 0.817 for BARGAIN. Its one loss was to BARGAIN on one dialogue setting, by 0.001 accuracy. On average 57.6 percent of records stopped at Stage 1, 27.8 percent at Stage 2 and 14.6 percent at the oracle. Tuning the stages separately instead of jointly lost 3.2 accuracy points and cost 2.65 times as much. The authors are unusually frank about the limits. Cost totals are unaudited and may double-count cached work. Baselines were not compared at matched accuracy. The certificate failed in 9 of 260 draws, including 6 of 20 on one setting. The released code does not match the reported runs. Macro-F1 fell to 0.594 on one setting with a rare positive class. Our call: pilot it on a high-volume filtering job where we can audit costs and per-class recall ourselves. (Source: arXiv 2611.08367, 2026)