Anomaly Detection Using Gradient-Based Unsupervised Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomalous behavior detection systems in computing systems often inaccurately classify behavior as normal or anomalous, leading to false positives or false negatives, which can result in undetected security breaches or inappropriate remediation measures.
Innovation Solution
Implementing a combination of unsupervised and supervised machine learning models trained on event data with feature generation and threshold-based labeling to accurately detect anomalous behavior, using an unsupervised training dataset to identify gradient ranges and select entries for supervised model training, and applying these models to production data for real-time anomaly detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection systems are used, then detection speed is maintained, but detection accuracy deteriorates leading to false positives and false negatives
Solution Approach 1:
The patent segments the anomaly detection process into two distinct phases: an unsupervised training phase that establishes baseline behavior patterns, and a supervised detection phase that classifies new events. This segmentation allows the system to learn normal behavior without labels first, then use that knowledge to improve supervised detection accuracy while reducing false positives.
Solution Approach 2:
The system performs preliminary unsupervised learning to establish baseline behavior patterns before conducting supervised anomaly detection. By pre-training on unlabeled data to understand normal system behavior, the system prepares feature representations and behavioral models that significantly improve subsequent detection accuracy and reduce false alarms.
2Measurement precision
If more sophisticated detection algorithms are implemented, then detection accuracy improves, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary unsupervised learning component that bridges raw system events and supervised classification. This intermediary layer extracts meaningful features and establishes behavioral baselines from unlabeled data, making the subsequent supervised detection simpler and more accurate without requiring complex labeled training datasets.
Solution Approach 2:
The system performs self-service by automatically learning normal behavior patterns through unsupervised learning without requiring manual labeling or intervention. The system autonomously identifies baseline behaviors, extracts features, and prepares detection models, reducing the need for complex manual configuration and simplifying overall system operation.
Data Source
AI summary
A training dataset is used to train an unsupervised machine learning trained model. Corresponding gradient values are determined for a plurality of entries included in the training dataset using the trained unsupervised machine learning model. A first subset of the training dataset is selected based on the determined corresponding gradient values and a first threshold value selected from a set of threshold values. A labeled version of the selected first subset is used to train a first supervised machine learning model to detect one or more anomalies.