Autoencoder Clustering for Network Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection models in network traffic often suffer from overfitting or underfitting, leading to reduced sensitivity to anomalies and excessive false-positive determinations, particularly in variable or consistent communication scenarios.
Innovation Solution
A method involving clustering of time series data using autoencoders, generating reconstruction errors, and creating probabilistic models to detect anomalies in network traffic, where autoencoders convert time series into feature vectors for clustering, and probabilistic models are aggregated to identify anomalies based on reconstruction errors exceeding a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection models (PCA, one-class SVM) are used to learn normal network traffic characteristics, then the system can identify anomalous traffic, but the models suffer from overfitting variable network communication which reduces sensitivity to anomalies
Solution Approach 1:
The patent segments the network traffic data into multiple time series representations (e.g., different temporal granularities, protocol layers, or traffic dimensions). Each time series is processed separately through autoencoders, allowing the system to capture patterns at multiple scales without overfitting to any single representation. This segmentation enables the model to generalize better across variable network communication patterns.
Solution Approach 2:
The patent transforms the anomaly detection problem from learning directly from raw network traffic to learning from reconstructed time series data. By introducing the time series dimension and using autoencoders to compress and reconstruct the data, the system operates in a transformed feature space that separates normal variation from true anomalies, thereby improving sensitivity without overfitting.
2Measurement precision
If traditional anomaly detection models reflect relatively consistent traffic to avoid overfitting, then sensitivity to anomalies is excessive but includes an excessive number of false-positive determinations
Solution Approach 1:
The patent changes the parameters used to represent network traffic from raw consistent patterns to reconstructed time series features. By transforming the data through autoencoders and analyzing reconstruction errors across multiple time series, the system adjusts its sensitivity parameters dynamically, allowing it to distinguish between normal consistent traffic variations and true anomalies, thereby reducing false positives while maintaining appropriate sensitivity.
3Adaptability or versatility
If clustering is performed on time series data using autoencoders, then the system can group similar traffic patterns, but the device complexity increases due to multiple autoencoders and probabilistic models
Solution Approach 1:
The patent makes each autoencoder universal by designing them to handle multiple functions: feature extraction, data compression, and anomaly scoring. The same autoencoder architecture is reused across different time series and clusters, reducing the need for separate specialized models. The probabilistic models also serve dual purposes by both clustering similar traffic patterns and providing anomaly detection, thereby reducing overall system complexity while maintaining versatility.
Data Source
AI summary
A method of anomaly detection for network traffic communicated by devices via a computer network, the method including clustering a set of time series, each time series including a plurality of time windows of data corresponding to network communication characteristics for a device; training an autoencoder for each cluster based on time series in the cluster; generating a set of reconstruction errors for each autoencoder based on testing the autoencoder with data from time windows of at least a subset of the time series; generating a probabilistic model of reconstruction errors for each autoencoder; and generating an aggregation of the probabilistic models for, in use, detecting reconstruction errors for a time series of data corresponding to network communication characteristics for a device as anomalous.