Building Automation Ticket Classification Using Zero- and Few-Shot LLMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The complexity and diversity of issues in building automation systems make it challenging to efficiently classify and prioritize report tickets, leading to potential misclassification and delayed resolution, which affects operational efficiency and customer satisfaction.
Innovation Solution
A large language model (LLM) is employed for zero-shot and few-shot classification of building automation tickets, supplemented by hybrid annotation and minimal manual correction, to enhance accuracy and scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional supervised machine learning techniques are used for ticket classification, then classification accuracy can be improved with sufficient training data, but manual annotation of data becomes labour-intensive and time consuming
Solution Approach 1:
The system uses heuristic-based automatic labelling to generate training data without manual annotation. The classification system automatically creates labelled samples by applying predefined heuristics to unlabelled tickets, enabling the model to train on self-generated data rather than requiring manual human labelling of each sample.
Solution Approach 2:
The system performs preliminary classification using heuristics before formal model training. By pre-processing tickets with heuristic rules to generate initial labels and pseudo-labels, the system prepares training data in advance, reducing the need for manual annotation during the actual training phase.
2Productivity
If heuristic-based automatic labelling is used to generate training data, then manual labor is reduced, but the generated training data lacks diversity and leads to classifiers with reduced accuracy
Solution Approach 1:
The system dynamically adjusts the mixing ratio of labelled and unlabelled data during training. Instead of using a fixed composition of training data, the system adaptively balances heuristic-generated labels, pseudo-labels, and manually annotated samples based on performance feedback, allowing the training strategy to evolve and improve diversity while maintaining accuracy.
Solution Approach 2:
The system creates a composite training dataset that combines multiple data sources with different characteristics. By integrating heuristic-labelled data, pseudo-labelled data from pre-training, and manually annotated diverse samples, the system forms a heterogeneous training corpus that leverages the strengths of each source to improve overall classification accuracy while maintaining efficiency.
3Device complexity
If manual classification is performed, then classification can be performed without complex systems, but the process is time consuming and subject to user error
Solution Approach 1:
The classification process is segmented into multiple stages: initial heuristic-based classification, pre-training with pseudo-labels, and fine-tuning with diverse labelled data. This segmentation allows the system to use simple heuristic rules for quick initial classification while applying more complex machine learning methods only where needed, balancing simplicity and speed across different processing stages.
Solution Approach 2:
The system introduces pseudo-labels as an intermediary between manual classification and final automated classification. These pseudo-labels generated during pre-training serve as a bridge, providing structured supervision signals that guide the model without requiring extensive manual annotation, thereby improving speed while maintaining reasonable accuracy.
4Measurement precision
If a classification system is designed to handle diverse and complex issues, then classification accuracy improves, but the system complexity increases
Solution Approach 1:
The system employs a universal classification framework that handles multiple issue types and complexities through a single integrated model. Rather than creating separate specialized classifiers for different ticket types, the system uses one multi-functional classification system trained on diverse data, reducing overall system complexity while maintaining high accuracy across various issue domains.
Data Source
AI summary
For ticket classification in a building automation system, a large language model (LLM) is used to classify. In one approach, a prompt is generated for zero-shot classification, and a prompt is generated for few-shot classification. In another approach, a hybrid annotation provides corrections (review) by an expert to correct LLM classification for sample tickets to be used as examples in the few-shot classification. The LLM may operate on a diverse and complex range of tickets in an efficient and scalable manner.