Auto-encoder Network Monitoring Abnormality Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for detecting abnormalities in network performance data using machine learning models face challenges such as overfitting, reliance on Gaussian distribution assumptions, and the need for domain expert intervention, particularly in 4G/5G performance data with distorted distributions, and struggle with unsupervised learning and prior classification of abnormal types.
Innovation Solution
A network monitoring apparatus and method utilizing an auto-encoder-based abnormality detection model trained through unsupervised learning, which receives performance management data and an initial abnormality probability value to generate reconstructed data and an abnormality score, allowing for the detection of abnormal samples without prior classification of abnormal types or definition of normal data periods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing machine learning methods (density-based, distance-based, trend-based) are used for abnormality detection, then detection capability is provided, but the methods require prior classification of abnormal types, domain expert intervention, and assume Gaussian distribution which does not match 4G/5G performance data
Solution Approach 1:
The autoencoder model performs unsupervised learning to automatically identify abnormal patterns in network performance data without requiring manual labeling or domain expert intervention. The model learns normal behavior patterns autonomously and detects deviations as abnormalities, eliminating the need for prior classification of abnormal types and reducing training complexity.
2Adaptability or versatility
If mixed data of normal and abnormality data is used for model training, then comprehensive training is achieved, but the model tends to overfit and requires balancing between known and unknown samples
Solution Approach 1:
The autoencoder architecture separates the learning of normal patterns from abnormal detection. By training exclusively on normal data, the model extracts and learns only normal behavior patterns, automatically ignoring abnormal samples. This extraction approach prevents overfitting to mixed data while maintaining comprehensive adaptability to normal variations.
3Measurement precision
If traditional clustering or distance-based methods are used, then abnormality detection is provided, but these methods fail when feature distribution deviates from Gaussian distribution as in 4G/5G performance data
Solution Approach 1:
The patent transforms the approach by changing from statistical methods assuming Gaussian distribution to a neural network-based autoencoder that learns arbitrary distributions from data. The model adapts to the actual distorted distributions of 4G/5G performance data by learning representations directly from the data without imposing distributional assumptions.
Data Source
AI summary
A network monitoring apparatus includes a memory configured to store a performance management data sample of a network; and at least one processor configured to input the performance management data sample and an abnormality probability value to an auto-encoder of an abnormality detection model that is trained based on performance management data of the network, wherein the abnormality probability value is a default value, obtain, from the auto-encoder of the abnormality detection model, a reconstructed performance management data sample and an abnormality score value, and detect an abnormal sample based on the abnormality score value, wherein the auto-encoder is configured to receive the performance management data sample and the abnormality probability value, and output the reconstructed performance management data sample and the abnormality score value.