Real-time Threat Alert Forensic Analysis via Anomaly Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Advanced persistent threat (APT) detection systems face challenges in efficiently tracking and correlating highly anomalous events over long periods, due to the overwhelming amount of data generated, which hinders early detection and real-time investigation.
Innovation Solution
The implementation of efficient in-memory data structures for real-time correlation of APT events, using anomaly scores to prioritize and store only the most suspicious events, and employing asynchronous eviction strategies to maintain high throughput and low-latency forensic analysis, while generating a smart hyper-alerts graph to cluster and visualize correlated alerts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all events are stored in main memory for real-time analysis, then detection speed and throughput are improved, but memory consumption increases significantly
Solution Approach 1:
The patent applies local quality by assigning different storage locations to different events based on their anomaly scores. High-anomaly events are stored in main memory for fast access, while low-anomaly events are stored in slower memory. This creates a heterogeneous storage system where each event receives storage quality proportional to its security importance, resolving the contradiction between throughput and memory consumption.
Solution Approach 2:
The patent segments the storage system into multiple memory levels (main memory and slower memory) and segments events into different categories based on anomaly scores. This segmentation allows the system to prioritize storage resources for critical events while maintaining overall system throughput, effectively balancing memory consumption against detection speed.
2Measurement precision
If anomaly scoring is performed on all events, then detection accuracy is improved, but processing time increases
Solution Approach 1:
The patent implements partial action by performing comprehensive anomaly scoring only on a subset of high-priority events stored in main memory, rather than scoring all events. This partial application of the scoring function maintains high detection accuracy for critical events while significantly reducing overall processing time and computational resources required.
Solution Approach 2:
The system applies different levels of anomaly scoring effort to different events based on their characteristics and priority. High-anomaly events receive thorough analysis while low-anomaly events receive minimal processing, creating a quality-differentiated processing approach that optimizes the balance between detection accuracy and processing time.
3Quantity of substance
If cold events are evicted from main memory to reduce memory usage, then memory capacity for hot events is improved, but detection of historical threats is worsened
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing anomaly scores for events before they are potentially evicted from main memory. This advance preparation ensures that even when cold events are moved to slower memory, their anomaly assessment is already complete and can be quickly retrieved, maintaining historical threat detection capability without compromising memory capacity for active events.
Data Source
AI summary
Methods and systems for security monitoring and response include assigning an anomaly score to each of a plurality of event paths that are stored in a first memory. Events that are cold, events that are older than a threshold, and events that are not part of a top-k anomalous path are identified. The identified events are evicted from the first memory to a second memory. A threat associated with events in the first memory is identified. A security action is performed responsive to the identified threat.