Event-Driven Anomaly Detection Model Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex computer systems face challenges in detecting anomalous behavior due to increased complexity, which leads to difficulties in distinguishing between normal and abnormal behavior, especially with advanced persistent threats, and results in high false positive detections with existing deterministic or probabilistic analysis methods.
Innovation Solution
A model-based event-driven anomaly detection system that generates an expected behavior model using statistical classifiers and behavioral classifiers to create a separation between normal and abnormal behavior, and trains models for probabilistic system behavior, allowing for more accurate and sensitive threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deterministic or probabilistic analysis methods are used for anomaly detection, then the detection process is simple to implement, but the false positive rate increases and sophisticated threats are missed
Solution Approach 1:
The system segments the anomaly detection process into distinct phases: data collection from multiple sources, feature extraction and selection, model training with labeled data, and detection execution. This segmentation allows each phase to be optimized independently, improving reliability while managing complexity through structured organization of the detection pipeline.
Solution Approach 2:
The system performs preliminary actions by collecting and labeling training data beforehand, extracting relevant features in advance, and training detection models before actual threat detection is needed. This preliminary preparation creates ready-to-use models that can quickly and accurately detect threats without requiring complex real-time analysis, thereby improving reliability while controlling operational complexity.
2Adaptability or versatility
If the detection scope is expanded to cover more system features, then more sophisticated threats can be detected, but the difficulty of detecting anomalies increases due to the law of large numbers
Solution Approach 1:
The system applies local quality by focusing detection efforts on specific, relevant features rather than attempting to analyze all system features uniformly. Through feature extraction and selection, the system identifies and concentrates computational resources on the most discriminative features for detecting sophisticated threats, thereby maintaining high detection coverage while reducing the effective search space and detection difficulty.
Solution Approach 2:
The system changes parameters by transforming raw system features into extracted features through mathematical transformations and statistical analyses. This parameter transformation converts complex, high-dimensional system data into a reduced feature space with more manageable dimensions, enabling comprehensive threat detection while overcoming the computational challenges posed by the law of large numbers.
3Measurement precision
If minute variations in system behavior are detected to identify advanced persistent threats, then threat detection sensitivity increases, but false positive detections increase
Solution Approach 1:
The system implements feedback mechanisms by using labeled training data that includes both normal and anomalous behavior examples. The detection models learn from this feedback during training, adjusting their sensitivity thresholds and decision boundaries to distinguish true threats from normal variations. This feedback-driven approach enables high measurement precision for detecting minute behavioral variations while maintaining reliability by reducing false positives through learned discrimination.
4Ease of operation
If a single detection method is used, then the system is easy to operate, but critical anomalies are missed or too many false positives are identified
Solution Approach 1:
The system merges multiple detection approaches by combining supervised learning models trained on labeled data with unsupervised anomaly detection techniques. This combination allows the system to leverage the strengths of each method: supervised models provide accurate classification of known threat patterns, while unsupervised methods detect novel anomalies. The merged approach maintains ease of operation through unified system architecture while significantly improving detection reliability and reducing false positives.
Data Source
AI summary
An embodiment of the present invention is directed to an optimal event-driven anomaly detection scheme. The present invention recognizes that anomaly detection based solely on rules-based (deterministic) or probabilistic analysis alone are insufficient to capture and respond to ever evolving, highly sophisticated threats that tend to persist within a system undetected for long periods of time. According to an embodiment of the present invention, a tiered detection scheme composed of behavioral analysis and machine-learned probabilistic system behaviors provides an optimal level of sensitivity to detect and respond to threats, and further limits the number of false positives identified.