Communication Network Anomaly Detection With Semi-Supervised Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing communication networks face challenges in accurately detecting and classifying anomalies in performance indicator data due to the scarcity of labeled training data and the inherent noise and variability in KPI time series, leading to high false positive rates and difficulty in distinguishing between problematic and non-problematic anomalies.
Innovation Solution
A three-step approach involving unsupervised, semi-supervised, and supervised machine learning models, where unsupervised models initiate anomaly detection, semi-supervised models enhance accuracy with user-provided labels, and supervised models refine the classification using both user and semi-supervised labels, with expert validation at each stage to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If unsupervised machine learning models are used for anomaly detection, then anomaly detection capability is provided, but accuracy and false positive rate are problematic due to lack of labeled data
Solution Approach 1:
The anomaly detection system is segmented into three distinct stages: unsupervised anomaly detection, semi-supervised refinement, and supervised classification. Each stage processes anomalies at different levels of detail and uses appropriate labeling strategies, thereby improving overall accuracy while maintaining automation.
Solution Approach 2:
The system performs preliminary anomaly detection using unsupervised models before applying more resource-intensive semi-supervised and supervised methods. This preliminary action filters the data early, reducing the volume of data that requires expensive labeled processing while maintaining detection capability.
2Measurement precision
If more labeled training data is collected manually, then supervised model accuracy improves, but time and resource consumption increase significantly
Solution Approach 1:
Instead of manually labeling all anomalies, the system applies partial labeling strategies where only a subset of anomalies (those with highest uncertainty or most critical ones) are manually labeled. The semi-supervised model then propagates these limited labels to generate additional training data, achieving good accuracy with minimal manual effort.
Solution Approach 2:
The semi-supervised machine learning model serves itself by automatically generating labeled training data from the limited manually labeled examples. This self-service mechanism reduces dependency on continuous manual labeling while maintaining model accuracy through automated label propagation and model retraining.
3Measurement precision
If semi-supervised and supervised models are trained iteratively, then detection accuracy improves, but computational complexity and training time increase
Solution Approach 1:
The complex training process is segmented into distinct phases: unsupervised model training, semi-supervised model training with uncertainty sampling, and supervised model training. Each phase has a specific objective and uses appropriate data subsets, making the overall complex process more manageable and efficient.
Solution Approach 2:
The unsupervised model performs preliminary anomaly detection and feature extraction before the semi-supervised and supervised models are trained. This preliminary action prepares the data in advance, reducing the computational burden on subsequent training stages and organizing the training pipeline for better efficiency.
4Productivity
If uncertainty sampling is used to select anomalies for labeling, then labeling efficiency improves, but selection complexity increases
Solution Approach 1:
The semi-supervised machine learning model acts as an intermediary that bridges the gap between raw anomaly data and manually labeled data. It uses uncertainty sampling to identify which anomalies最需要 manual labeling, thereby improving labeling efficiency while the model itself handles the complexity of selection algorithms.
Data Source
AI summary
Example embodiments may relate to training of a supervised machine learning model for anomaly detection in a communication network. A computer-implemented method may comprise: detecting, by an unsupervised machine learning model, a plurality of anomalies in performance indicator data of a communication network; receiving labels for a first subset of the plurality of anomalies and labelling the first subset of the plurality of anomalies with the labels; training, based on the labelled first subset of the plurality of anomalies, a semi-supervised machine learning model for labelling anomalies; labelling, by the semi-supervised machine learning model, a second subset of the plurality of anomalies; and training, based on the labelled first and second subsets of the plurality of anomalies, a supervised machine learning model for detecting and/or classifying anomalies in the performance indicator data.