Self-Taught Learning Network for Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network anomaly detection systems face challenges in distinguishing malicious network behaviors from normal traffic, especially with dynamic and diverse network characteristics, and struggle to efficiently classify anomalies using both labeled and unlabeled data, particularly when dealing with both numerical and categorical features.
Innovation Solution
A system comprising a data pre-processing module that extracts time-series features and tokenizes categorical features into dimensional embedding vectors, combined with an alert module featuring a trained autoencoder and classifier neural network, which uses a bootstrapping method to detect and classify potential malicious network behaviors, enabling efficient classification even without fully labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional network anomaly detection mechanisms analyze individual traffic records to identify known characteristics, then detection of specific malicious behaviors is possible, but the system cannot distinguish anomalous traffic from normal traffic without referring to associated traffic records of the past
Solution Approach 1:
The patent transitions from analyzing individual traffic records in isolation to analyzing sequences of traffic records by introducing a temporal dimension. The recurrent neural network processes traffic records as time-series data, capturing patterns and relationships across multiple records. This dimensional transformation enables the system to distinguish anomalous traffic from normal traffic by examining behavioral patterns over time rather than isolated characteristics.
2Adaptability or versatility
If deep learning methods such as recurrent neural networks are applied to detect network traffic associated with cyber-threats, then the system can handle large volumes of data and learn variations of network behavior, but the system requires extensive labeled training data which is often unavailable
Solution Approach 1:
The patent applies preliminary action by first pre-training the recurrent neural network on unlabeled network traffic data using autoencoder-based unsupervised learning. During this pre-training phase, the network learns normal network behavior patterns and representations without requiring labeled data. After pre-training, the system can then perform anomaly detection by comparing new traffic against the learned normal patterns, enabling deployment in environments with limited labeled data while maintaining adaptability to various network behaviors.
3Measurement precision
If clustering methods based on deep learning autoencoders are used to detect anomalies, then the system can flag anomalous network traffic, but the system is unable to make sense of the clusters without the presence of labelled data
Solution Approach 1:
The patent implements feedback by creating a closed-loop system where the recurrent neural network continuously monitors network traffic, compares it against learned normal patterns, and adjusts its anomaly detection thresholds based on feedback from detected anomalies and false positives. The system learns from its detection performance over time, refining its understanding of what constitutes anomalous behavior. This feedback mechanism enables the system to interpret clusters of anomalies meaningfully without relying on pre-existing labeled data, as the feedback loop progressively builds contextual understanding of network behaviors.
Data Source
AI summary
This document discloses a system and method for detecting and classifying potential malicious network behaviours or characteristics contained within data traffic. In particular, this document discloses a system comprising a data pre-processing module for processing the received data traffic before the processed data traffic is provided to an alert module communicatively connected to the data pre-processing module. The alert module, which comprises a trained autoencoder and a classifier neural network trained via self-taught learning, then determines, based on a set of partially labelled training data, whether potential malicious network behaviours that typically present themselves as network traffic anomalies are contained within the processed data traffic.