Processing Pipeline for SIEM Threat Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
SIEM systems face challenges in identifying emerging threats due to evolving event nature and dynamic information environments, as conventional rules-based anomaly detection often fails to detect true threats and can generate excessive false positives, especially with categorical non-numeric data and lack of labeled training data.
Innovation Solution
A processing pipeline with two branches is developed to support advanced analytics, including machine-learning processes, for data fitting and live data evaluation, which preprocesses data to convert categorical data into a numeric format, enabling feature engineering and anomaly detection without static signatures or predefined rules, allowing for continuous optimization and attribution of anomalies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If rules-based anomaly detection is used to identify events of interest, then the system can detect deviations from expected behavior, but it generates excessive false positives and fails to identify emerging threats
Solution Approach 1:
The patent transforms categorical non-numeric data into numeric features through encoding schemes (one-hot encoding, label encoding, frequency encoding). This parameter transformation enables the application of machine learning algorithms that require numeric input, fundamentally changing how the system processes and analyzes security events to reduce false positives and improve threat detection.
Solution Approach 2:
The patent replaces traditional rules-based anomaly detection mechanisms with machine learning-based detection. Instead of relying on predefined static rules that generate false positives, the system uses trained models (random forest, gradient boosting, neural networks) that dynamically learn patterns from data, substantially reducing false positives while improving emerging threat identification.
2Reliability
If static rules and signatures are used for anomaly detection, then the system can identify known threats, but it fails to detect emerging threats that masquerade as benign activity
Solution Approach 1:
The patent implements dynamic anomaly detection through machine learning models that continuously learn from new data. The system adapts to evolving threats by retraining models with recent security events, allowing it to detect emerging threats that masquerade as benign activity rather than relying on static signatures that become obsolete.
Solution Approach 2:
The patent performs preliminary feature engineering and data preprocessing to create meaningful numeric representations of security events before applying machine learning models. This preliminary transformation of categorical data into engineered features enables the models to learn patterns and detect emerging threats more effectively.
3Reliability
If machine-learning processes are applied to categorical non-numeric data, then the system can improve anomaly detection accuracy, but the data must be preprocessed to convert categorical data into numeric format
Solution Approach 1:
The patent segments the data preprocessing task into distinct encoding schemes (one-hot encoding for nominal variables, label encoding for ordinal variables, frequency encoding for categorical variables). This segmentation of the preprocessing complexity into manageable, well-defined encoding strategies makes the overall process more systematic and easier to implement despite the inherent complexity.
4Productivity
If advanced analytics with two-branch processing pipeline are implemented, then the system can efficiently model baseline behavior and detect anomalies, but the processing pipeline complexity increases
Solution Approach 1:
The patent divides the security analytics processing into two distinct branches: a baseline modeling branch that learns normal behavior patterns, and an anomaly detection branch that identifies deviations from baseline. This segmentation of the processing pipeline into specialized branches improves efficiency by allowing each branch to optimize for its specific function while managing overall complexity through modular architecture.
Data Source
AI summary
A processing pipeline for supporting advanced analytics for network monitoring and information management as well as specific analytics for particular use cases. The processing pipeline 500 takes in system data (502) and pre-processes (504) the system data. The system data (502) may include any of the types of data described above including text log files, and categorical data from various sources. The illustrated processing pipeline 500 includes two branches; a data modeling branch (506) where a model is developed for the data and a data evaluation branch (508) where the developed model is leveraged to evaluate live data. For certain event detection use cases, the output of the data evaluation branch (508) includes a score (510) (e.g., a threat level score) and context information for evaluating the threat.