Autoencoder Clustering for Network Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection models in computer networks often suffer from overfitting or underfitting, leading to reduced sensitivity to anomalies and excessive false-positive determinations, particularly in variable or consistent network traffic.
Innovation Solution
A multi-phase approach using autoencoders for clustering network traffic, generating reconstruction error models, and employing a statistical model to identify anomalies by comparing reconstruction errors against a Gaussian mixture model, combined with machine learning techniques like one-class SVM for enhanced anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If principal component analysis and one-class support vector machines are used for feature extraction and learning normal network traffic characteristics, then anomaly detection capability is improved, but the model suffers from overfitting variable network communication which reduces sensitivity to anomalies
Solution Approach 1:
The patent segments network traffic into multiple clusters based on communication patterns, protocols, and behaviors. Each cluster is modeled separately using autoencoders, allowing the system to capture the unique characteristics of different traffic types without overfitting. This segmentation enables the model to generalize better across variable traffic while maintaining sensitivity to anomalies within each cluster.
Solution Approach 2:
The patent employs dynamic clustering that adapts to changing network traffic patterns over time. The autoencoder models are trained on evolving traffic data and can adjust to new communication patterns, making the anomaly detection system responsive to dynamic network environments without losing sensitivity to genuine anomalies.
2Reliability
If models reflect relatively consistent traffic to reduce false positives, then sensitivity to anomalies becomes excessive and includes an excessive number of false-positive determinations
Solution Approach 1:
The patent applies local quality by creating cluster-specific autoencoder models that are tailored to the characteristics of each traffic cluster. Each model learns the normal patterns specific to its cluster, allowing for precise anomaly detection within that local context. This prevents false positives caused by applying a single global model to diverse traffic types while maintaining high sensitivity to cluster-specific anomalies.
Solution Approach 2:
The patent changes parameters by using multiple autoencoders with different architectures and training configurations for different clusters. Each autoencoder is optimized for its specific cluster's traffic characteristics, enabling the system to adjust its detection sensitivity locally rather than using a fixed threshold across all traffic types.
3Device complexity
If a single anomaly detection model is used for all network traffic, then device complexity is reduced, but the model cannot effectively handle both variable and consistent traffic patterns simultaneously
Solution Approach 1:
The patent achieves universality by using autoencoders as a unified framework that can handle both variable and consistent traffic patterns through clustering. The same autoencoder architecture is applied across multiple clusters, each adapted to its specific traffic type. This provides a universal solution that maintains manageable complexity while effectively processing diverse network traffic patterns through the clustering mechanism.
Data Source
Figure 1~2
Figure 3
AI summary
A method of anomaly detection for network traffic communicated by devices via a computer network, the method comprising: receiving a set of training time series each including a plurality of time windows of data corresponding to network communication characteristics for a device; training an autoencoder for the first cluster based on time series in the cluster, wherein a state of the autoencoder is periodically recorded after a predetermined fixed number of training examples to define a set of trained autoencoders for the cluster; receiving a new time series including a plurality of time windows of data corresponding to network communication characteristics for a device; for each time window of the new time series, generating a vector of reconstruction errors for the device for each autoencoder based on testing the autoencoder with data from the time window; evaluating a derivative of each vector; training a machine learning model based on the derivatives so as to define a filter for identifying subsequent time series for a device being absent anomalous communication.