Anomaly Detection Model Training via Multi-Level Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems in IT environments face challenges in achieving satisfactory accuracy and efficiency due to the overwhelming volume of monitoring data, leading to an overabundance of false alerts that obscure more serious issues, making it difficult for expert reviewers to provide sufficient feedback to improve model accuracy.
Innovation Solution
The system generates an initial set of training data by combining classifications from multiple pretrained classifiers, performs multi-level clustering of data samples, and creates an expert review list for targeted feedback, allowing reviewers to focus on representative samples and apply feedback to closely related data samples, thereby improving the accuracy of anomaly detection models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple pretrained classifiers are used to generate training data, then the accuracy of anomaly detection is improved, but the device complexity increases
Solution Approach 1:
The system segments the anomaly detection process into multiple independent pretrained classifiers, each specializing in different aspects of anomaly detection. This segmentation allows each classifier to focus on specific patterns, improving overall accuracy while maintaining manageable complexity through modular design
Solution Approach 2:
Multiple pretrained classifiers are merged into an ensemble system where their classifications are combined to generate training data. This merging leverages the strengths of different classifiers to achieve higher accuracy than any single classifier could achieve alone
2Measurement precision
If expert reviewers review all data samples, then the model accuracy is improved, but the loss of time increases
Solution Approach 1:
The dataset is segmented into multiple clusters based on similarity, and experts only need to review representative samples from each cluster. This segmentation reduces the total number of samples requiring expert review while maintaining model accuracy through targeted feedback on diverse representative cases
Solution Approach 2:
Automated preprocessing and clustering are performed before expert review to organize and prioritize data samples. This preliminary action prepares the data in a way that enables experts to focus their efforts on the most valuable samples, significantly reducing review time
3Productivity
If clustering is performed on data samples, then the productivity of expert review is improved, but the device complexity increases
Solution Approach 1:
The system performs self-service through automated clustering algorithms that automatically organize data samples into groups based on their characteristics. This self-service capability eliminates the need for manual data organization, significantly improving review efficiency while the automation manages the processing complexity
Data Source
AI summary
An embodiment generates an initial set of training data from monitoring data. The initial set of training data is generated by combining outputs from a plurality of pretrained classifiers. The embodiment trains a new classification model using the initial set of training data to identify anomalies in monitoring data. The embodiment performs a multiple-level clustering of the data samples resulting in a plurality of clusters of sub-clusters of data samples, and generates a review list of data samples by selecting a representative data sample from each of the clusters. The embodiment receives an updated data sample from the expert review that includes a revised target classification for at least one of the data samples of the expert review list. The embodiment then trains another replacement classification model using a revised set of training data that includes the updated data sample and associated revised target classification.