Decoupled Cloud Anomaly Detection for Streaming Data Alerts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection systems in cloud environments struggle with scalability and efficiency when handling high-velocity and high-volume data streams, leading to overwhelming numbers of alerts and challenges in maintaining up-to-date models, particularly in complex systems with numerous data streams.
Innovation Solution
A cloud-based anomaly detection system decouples data stream processing from anomaly detection models, using modular architecture with dedicated virtual machines for model maintenance and stream processing, enabling flexible resource allocation and efficient generation of smart alerts based on priority, persistence, pervasiveness, and recency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional anomaly detection systems process high-velocity and high-volume data streams, then more data can be monitored, but the system becomes less scalable and more complex
Solution Approach 1:
The system segments the anomaly detection architecture into separate components: a stream processing layer that handles data ingestion and a dedicated model maintenance layer that handles model training and updates. This segmentation allows each layer to be optimized independently and scales better with data volume.
Solution Approach 2:
The patent introduces an intermediary layer between data stream processing and model maintenance. This intermediary enables decoupled operation where stream processing can continue uninterrupted while models are updated in the background, reducing the complexity of coordinating model changes with active data processing.
2Quantity of substance
If the system processes more data streams, then monitoring coverage increases, but the number of alerts increases overwhelming operators
Solution Approach 1:
The system extracts and separates the alert generation function from raw anomaly detection. By taking out the alert generation process and applying multiple filtering layers (priority, persistence, pervasiveness, recency), the system reduces the volume of alerts while preserving critical information about actual issues.
Solution Approach 2:
The system implements feedback mechanisms where alert characteristics (priority, persistence, pervasiveness, recency) are continuously refined based on observed patterns. This feedback loop enables the system to learn what constitutes a genuine issue versus noise, improving the signal-to-noise ratio over time.
3Measurement precision
If anomaly detection models are continuously updated, then detection accuracy improves, but system performance and scalability deteriorate
Solution Approach 1:
The system performs preliminary actions by pre-computing and caching statistical parameters (mean, variance, quantiles) during normal operation. When model updates are needed, these pre-computed values can be quickly applied without reprocessing the entire data stream, maintaining throughput while improving accuracy.
Solution Approach 2:
The patent implements dynamic model updates where models can be trained in the background and applied without interrupting stream processing. The system dynamically switches between model versions based on readiness status, allowing continuous improvement of detection accuracy while maintaining stable processing throughput.
4Productivity
If more virtual machines are deployed for processing, then processing capacity increases, but resource allocation becomes less efficient
Solution Approach 1:
The system designs virtual machines with multi-functionality, where each VM can handle both stream processing and model maintenance tasks. This universality reduces the total number of VMs needed compared to dedicated separate systems, improving resource allocation efficiency while maintaining high processing capacity.
Data Source
AI summary
Aspects of the subject disclosure may include, for example, detecting data streams by a processing system including a processor, wherein the processing system is associated with an anomaly detection and alerting system in which stream processing and model maintenance is decoupled from one another, and wherein one or more dedicated virtual machines (VMs) store and maintain anomaly detection and alerting models, based on the detecting, causing, by the processing system, a plurality of stream-processing VMs to be instantiated for processing the data streams, and managing, by the processing system, data stream assignments for the plurality of stream-processing VMs based on monitoring of one or more conditions, wherein the plurality of stream-processing VMs process assigned data streams by executing instances of the anomaly detection and alerting models, and provide model outputs to the one or more dedicated VMs for updating of the anomaly detection and alerting models. Other embodiments are disclosed.