Recurring Event Scheduler for Real-Time Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems face challenges in analyzing and detecting anomalies in real-time or near real-time from large datasets, particularly in ensuring data is processed quickly enough to provide relevant notifications and ingesting data from diverse sources.
Innovation Solution
A recurring event-based scheduler is employed to support continuous or real-time/near real-time anomaly detection by generating schedules for anomaly detection tasks based on specified data ingestion intervals, incorporating techniques like jitter to distribute processing and ensure timely analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data is analyzed sequentially to detect time-based anomalies, then anomaly detection accuracy is improved, but processing speed deteriorates
Solution Approach 1:
The patent segments the anomaly detection process into distinct components: data ingestion, sequential analysis, and notification generation. By using multiple workers that can process different data streams simultaneously while maintaining sequential analysis within each stream, the system achieves both accuracy and speed. The segmentation allows parallel processing of multiple metrics without compromising the temporal integrity of individual time-series analyses.
Solution Approach 2:
The system performs preliminary actions by pre-configuring detection rules, thresholds, and worker assignments before data arrives. The scheduler pre-allocates computational resources and prepares analysis templates, so that when data is ingested, the sequential analysis can begin immediately without setup delays. This preliminary preparation enables faster processing while maintaining accurate anomaly detection.
2Loss of time
If data is processed quickly to provide relevant notifications, then notification timeliness is improved, but analysis completeness deteriorates
Solution Approach 1:
The patent implements partial action by having workers process data at different depths based on configuration. Some workers perform rapid shallow analysis for immediate notifications, while others perform deeper sequential analysis for comprehensive understanding. The system sends notifications based on the fastest available analysis that meets confidence thresholds, achieving timely alerts without requiring complete analysis of all data dimensions before notification.
Solution Approach 2:
The system uses feedback loops where notification outcomes and anomaly patterns are fed back into the detection system. This allows the system to learn from past anomalies and improve future detection accuracy. The feedback mechanism ensures that even quick notifications are refined over time, maintaining analysis completeness through iterative improvement rather than requiring exhaustive initial analysis.
3Adaptability or versatility
If data is ingested from different sources continuously, then data coverage is improved, but system complexity deteriorates
Solution Approach 1:
The patent implements universality through a unified worker architecture that can handle multiple data sources and metric types through a common interface. Workers are designed to be multi-functional, capable of processing various data formats and applying different detection algorithms through configuration rather than requiring separate processing pipelines for each data source. This universal design achieves broad data coverage while minimizing system complexity.
Solution Approach 2:
The scheduler acts as an intermediary layer between diverse data sources and the analysis workers. It standardizes incoming data from different sources into a common format, manages worker allocation, and coordinates processing across multiple data streams. This intermediary abstraction shields workers from data source complexity, allowing them to focus on analysis while the scheduler handles source-specific variations, thereby achieving broad coverage without proportionally increasing worker complexity.
Data Source
AI summary
Systems and methods are described for using a recurring event based scheduler to continuously monitor data to detect anomalies within the data. In some aspects, an anomaly detection schedule may be determined for monitoring time series data to detect anomalies based on a data ingestion interval. A plurality of anomaly detection events may be sequentially generated and stored in an event queue at times specified by the anomaly detection schedule. The anomaly detection events may then be processed sequentially from the event queue to trigger execution of a plurality of anomaly detection workflow tasks at the times specified by the anomaly detection schedule. In some cases, execution of individual anomaly detection workflow tasks causes individual portions of time series data to be obtained from a customer data source and processed by an anomaly detection model to detect anomalies in the time series data.