Anomaly Detection Pipeline for Fast Large-Scale Event Screening
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data intake and query systems require long processing times to detect anomalies in large datasets, leading to delayed responses to potential issues such as fraudulent activity.
Innovation Solution
Implementing high-speed anomaly detection on a subset of events during ingestion, followed by full-scale detection on the entire dataset, allowing for rapid anomaly detection and mitigation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-scale anomaly detection is performed on the entire dataset, then detection accuracy is improved, but processing time increases significantly
Solution Approach 1:
The patent segments the anomaly detection process into two distinct phases: (1) high-speed anomaly detection performed on a subset of events during data ingestion, and (2) full-scale anomaly detection performed on the entire dataset afterward. This segmentation allows the system to quickly identify potential anomalies using a fast but less comprehensive method, then apply a more accurate but slower method only when needed, thereby resolving the contradiction between detection accuracy and processing time.
Solution Approach 2:
The patent applies preliminary action by performing high-speed anomaly detection on a subset of events during the data ingestion phase, before the full-scale detection is executed. This preliminary detection identifies suspicious patterns early, allowing the system to prioritize or pre-process certain data segments for the subsequent full-scale analysis, thus reducing the overall time required while maintaining accuracy.
2Reliability
If conventional anomaly detection methods are used on large datasets, then comprehensive analysis is achieved, but the mean time to detect anomalies increases
Solution Approach 1:
The patent divides the dataset processing into two segments: a subset of events processed through high-speed anomaly detection during ingestion, and the complete dataset processed through full-scale anomaly detection. This segmentation enables the system to maintain comprehensive detection coverage while significantly reducing the mean time to detect anomalies by handling the majority of data with a faster method.
Solution Approach 2:
The patent applies partial action by performing high-speed anomaly detection on only a subset of events rather than processing every single event with the slowest, most comprehensive method. This partial processing approach achieves sufficient detection for most cases while reserving full-scale detection for verification and complex scenarios, thereby reducing the overall mean detection time.
3Productivity
If high-speed anomaly detection is performed on a subset of events, then processing speed is improved, but detection coverage may be reduced
Solution Approach 1:
The patent segments the detection process into high-speed detection on a subset of events and full-scale detection on the entire dataset. This ensures that while the high-speed phase improves processing productivity, the full-scale phase subsequently restores comprehensive detection coverage, eliminating the trade-off between speed and coverage.
Solution Approach 2:
The patent uses high-speed anomaly detection as an intermediary step that processes a subset of events quickly to identify potential anomalies. This intermediary detection then guides or informs the subsequent full-scale detection, ensuring that no anomalies are missed while maintaining high overall processing speed. The intermediary phase acts as a filter that prepares the data for more thorough analysis.
Data Source
AI summary
The various implementations provide for techniques to ingest raw data into a set of events and sample a subset of the events for initial processing via high-speed anomaly detection. The high-speed anomaly detection techniques use a first set of sensitivities that enable fast detection of potential anomalies within the subset of events. The subset of events is also included in a full set of events that are stored in an index and processed via full-scale anomaly detection. The full-scale anomaly detection uses a second set of sensitivities that enable the system to detect anomalies from the full set of events accurately. Upon detecting an anomaly using the high-speed or full-scale detection techniques, the system performs actions to notify entities or automatically mitigate the effects of the anomaly such as mitigation actions to quell potentially fraudulent activity.