Cost-Aware Abstention for LLM-Based IT Ticket Classification: what it means for business leaders

On a public set of 47,837 IT tickets, a cost-based rule for when to hand a ticket to a person cut routing cost by 38 percent for trained classifiers, while prompted LLMs earned almost no permission to route on their own.

1. What the method is

This paper is an evaluation framework, not a new algorithm. It asks one operational question: when costs are asymmetric, when should an automated classifier be allowed to route an IT support ticket, and when should it hand the ticket to a person? The author wraps each classifier in an abstention policy. The policy routes a ticket only when a confidence score clears a threshold. That threshold is chosen on a held-out calibration split under an explicit cost model, then frozen and tested once on unseen data. Prediction, confidence, rejection scoring and cost are kept as separate stages that can each be checked. The same protocol is applied to four trained classifiers and to four prompted LLMs: GPT-4o-mini, Claude Haiku 4.5, Qwen2.5-7B and Llama-3-8B, each with and without examples in the prompt. The output is a go or no-go verdict for each model, not an accuracy score.

2. Why the method was developed

Ticket routing has uneven costs. A correct automatic route is nearly free. A human triage costs something: the industry figure the paper cites is about US$17 for a voice ticket. A misroute costs more, because the ticket still ends up with a person after a delayed handoff, plus rework and missed service levels. Accuracy and macro-F1 count all errors the same way and say nothing about when to trust a single decision. At the same time, teams are tempted to replace trained classifiers with prompted LLMs that need no labeled data, and LLMs are known to report confidence well above their real accuracy. The building blocks are decades old: Chow's trade-off between errors and rejections, Elkan's cost-sensitive decisions, and standard calibration checks. They are rarely combined into a reproducible readiness test on a public ticket set, and this paper does that.

3. Who should care

The direct audience is anyone running or selling automation on a service desk or support queue: IT operations leaders, support managers and the teams building ticket triage into service-management tools. The lesson carries over to any classification task where a wrong automatic action costs more than asking a person, such as claims intake, document routing, content moderation or alert triage. For our ML engineers it is a reusable protocol: set aside calibration data, write the cost ratio down, pick the threshold on that data and test once. For product and procurement it is a warning about vendor demos that show LLM classification accuracy without a deferral policy or a calibrated confidence. Finance owns one input the method cannot supply: what a triage really costs us compared with a misroute. That ratio sets the threshold, and so the share of work that can be automated.

4. How the method works

Each decision has one of three costs: zero for a correct route, one for human triage and, in the main setting, five for a wrong route. For the trained classifiers, confidence is the probability of the top class. The policy routes when that probability exceeds a threshold picked from a grid on the calibration split, and the threshold is then frozen. The gap between the top two classes and the spread across all classes are tested as alternative signals. For the LLMs, the prompt asks for JSON with a label and a stated confidence, and malformed output counts as wrong with confidence zero. Stated confidences are not probabilities, so a monotone mapping (isotonic regression) fitted on 500 calibration tickets per setup converts them into probabilities. The LLM threshold is not tuned but derived from the costs: route only when the chance of being right exceeds 0.80, the break-even point for a 1-to-5 ratio. Agreement across models is tested as a further confidence signal.

5. How it was evaluated

The data is the public Kaggle IT Service Ticket Classification Dataset: 47,837 short internal tickets, with a median length of 175 characters. They fall into eight uneven categories such as Access, Hardware, Purchase and a Miscellaneous catch-all, split 70/10/20 into training, calibration and test. The trained models are logistic regression, a linear SVM and a random forest on word-frequency features, plus DistilBERT fine-tuned for two epochs. The LLMs ran at temperature zero with the label names and up to 24 examples, but no category definitions. To save cost they were scored on a 1,000-ticket test subset, and the trained models were rescored on the same tickets to confirm the ranking holds. Metrics include accuracy, calibration error, how well confidence separates right answers from wrong ones, and expected cost, with bootstrap confidence intervals. Robustness checks cover 16 combinations of triage and misroute cost, a prompt-length test, a second inference provider and a check for near-duplicate tickets leaking between training and test.

6. How it performed

The trained classifiers were far ahead. DistilBERT reached 0.879 accuracy on the shared subset, against 0.477 for the best LLM, Claude Haiku with examples. For logistic regression, the threshold of 0.60 chosen on calibration data cut the modeled cost per ticket from 0.763 to 0.472. That is a 38.1 percent saving (interval 35.7 to 40.4 percent), and wrong routes fell from 15.3 to 3.5 percent while 70.5 percent of tickets were still routed automatically. The other trained models saved 35.5 to 38.1 percent, and frozen policies beat both routing everything and deferring everything in 14 of 16 cost settings. Recalibration fixed the LLMs' stated confidence levels but did not improve how well they told right answers from wrong ones; GPT-4o-mini with examples reported confidence 1.0 on every ticket. At the 0.80 break-even, seven LLM setups deferred every ticket and Haiku with examples routed 19 of 1,000. The limits are illustrative costs, one dataset and no test of drift. Our call: act now on the protocol for any routing automation, and watch prompted LLMs as routers. (Source: arXiv 2610.31657, 2026)