A new method for classifying IT support tickets demonstrates a substantial reduction in routing errors by strategically abstaining from automatic classification. Researchers introduced "cost-aware abstention," a technique that allows automated systems to defer tickets to human support when the potential cost of an incorrect automated routing exceeds a defined threshold. This approach aims to balance the efficiency of automated systems with the accuracy of human oversight.
The study, published on arXiv, evaluated this method on a corpus of 47,837 IT support tickets across eight categories. When employing a cost model where incorrect routing incurs a penalty five times higher than human triage, the proposed abstention strategy, combined with a TF-IDF logistic regression classifier, achieved a 77.3% reduction in wrong routes compared to a baseline system. The overall cost of ticket routing also saw a relative reduction of 38.1%.
Traditional IT support systems often rely on automated classifiers to route incoming tickets to the appropriate department or technician. However, misrouting can lead to delays, increased resolution times, and higher operational costs. The cost-aware abstention method addresses this by setting a readiness assessment based on explicit costs. A routing policy is selected on calibration data under these explicit costs, and then this frozen policy is evaluated on new, unseen data.
The research compared four traditional machine learning classifiers with four large language models (LLMs) under both zero-shot and few-shot prompting scenarios. While LLMs showed promise, a simpler TF-IDF model, when tuned with the cost-aware abstention threshold, outperformed them in terms of cost reduction and error rate reduction for this specific task. This suggests that for certain classification problems, simpler, well-tuned models can be more effective when integrated with cost-sensitive decision-making.
The illustrative cost model used in the study assigned a cost of zero for correct routing, one unit for human triage, and five units for incorrect routing. By setting a threshold of 0.60 for the TF-IDF logistic regression classifier, the system learned to abstain from classifying tickets where its confidence was below this level, thereby sending them for human review. This selective abstention proved more cost-effective than forcing a classification on every ticket.
This approach aligns with broader research into cost-sensitive machine learning, where the financial implications of model predictions are explicitly factored into the decision-making process. For instance, other studies have explored cost-saving LLM cascades with early abstention and cost-aware model selection for text classification, highlighting the importance of balancing performance with operational expenses. The ability to dynamically decide when a task is too costly or uncertain for automation and should be handed off to a human is becoming increasingly valuable in practical applications.
The researchers have made their analysis tools and data publicly available, aiming to encourage further research and development in cost-aware AI systems for IT support and similar classification tasks.