Hybrid Threat Detection Using Real-Time and Batch Anomaly Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security systems struggle to detect unknown threats and insider threats, as well as scale and process vast amounts of data, often failing to identify malicious activities that use valid access and evade traditional security technologies.
Innovation Solution
A combined approach using both machine-learning based and rules-based analysis techniques to detect anomalies and threats, integrating a data intake and query system like SPLUNK® ENTERPRISE to analyze machine-generated data, enabling real-time and batch processing for scalable and insightful threat detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional security technologies are used to detect threats, then known threats can be identified, but unknown threats and insider threats cannot be detected
Solution Approach 1:
The system dynamically adapts its detection capabilities by combining static rules-based analysis with dynamic machine learning models that continuously learn from new data, enabling the system to evolve its threat detection patterns and identify previously unknown threats while maintaining reliability for known threat detection
Solution Approach 2:
The patent creates a composite detection system that integrates two different analytical approaches (rules-based and machine learning) into a unified framework, where each component contributes its strengths: rules-based systems provide reliable pattern matching for known threats, while machine learning provides adaptability for detecting novel and insider threats
2Measurement precision
If vast amounts of data are processed to improve threat detection, then detection accuracy improves, but system scalability and processing efficiency deteriorate
Solution Approach 1:
The system segments the data processing workload by dividing it into rules-based analysis components and machine learning components, allowing parallel processing of different data types and reducing the computational burden on any single system, thereby maintaining both precision and productivity when processing vast amounts of data
Solution Approach 2:
The patent replaces traditional mechanical data processing methods with machine learning algorithms that can automatically learn patterns and anomalies from vast datasets without requiring manual rule configuration, significantly improving processing efficiency while maintaining or enhancing detection precision
3Reliability
If more data is analyzed to reduce false positives, then false positive rate decreases, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary filtering using rules-based analysis to quickly identify and eliminate obvious false positives before subjecting remaining cases to more computationally intensive machine learning analysis, reducing overall processing time while maintaining low false positive rates through staged analysis
Data Source
AI summary
First event data, indicative of a first activity on a computer network and second event data indicative of a second activity on the computer network, is received. A first machine learning anomaly detection model is applied to the first event data, by a real-time analysis engine operated by the threat indicator detection system in real time, to detect first anomaly data. A second machine learning anomaly detection model is applied to the first anomaly data and the second event data, by a batch analysis engine operated by the threat indicator detection system in a batch mode, to detect second anomaly data. A third anomaly is detected using an anomaly detection rule. The threat indictor system processes the first anomaly data, the second anomaly data, and the third anomaly data using a threat indicator model to identify a threat indicator associated with a potential security threat to the computer network.