Kernel Density Estimator Event Stream Deviation Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current complex event processing (CEP) systems face challenges in detecting deviations in event streams due to high-volume, low-latency data processing requirements, limited expressiveness of standard SQL aggregates, and the assumption of normal distribution, which can lead to suboptimal results for non-normal distributions and multiple modes.
Innovation Solution
The use of kernel density estimators (KDEs) for real-time deviation detection, combining long-term and short-term statistics with an ideal probability density function, and adaptive threshold settings to identify deviations in event streams, allowing for online, real-time analysis without prior knowledge of the stream characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If standard SQL aggregates are used for deviation detection, then the processing is simple and fast, but the expressiveness is limited and cannot detect irregularities or multiple modes of distribution
Solution Approach 1:
The patent transitions from using standard SQL aggregates (which provide limited summary measures) to employing kernel density estimation with multiple parameters (bandwidth, kernel function selection, and time window parameters) to capture complex distribution characteristics including multiple modes and irregularities, thereby resolving the contradiction between processing simplicity and distribution detection capability
Solution Approach 2:
The patent introduces kernel density estimation as an intermediary computational layer between raw event streams and deviation detection results. This intermediary provides a flexible statistical model that can adapt to various distribution types while maintaining real-time processing capabilities, bridging the gap between simple aggregation and complex distribution analysis
2Ease of operation
If Bollinger bands are used for deviation detection, then the implementation is straightforward, but reliable results are only obtained for normal distributions
Solution Approach 1:
The patent replaces the static assumption of normal distribution with dynamic kernel density estimation that adapts to the actual distribution characteristics of the event stream. The kernel density estimator dynamically adjusts to capture multiple modes and irregularities, maintaining implementation simplicity while significantly improving reliability for arbitrary distributions
Solution Approach 2:
The patent changes the fundamental parameter assumption from fixed normal distribution parameters to adaptive kernel density parameters that are estimated from the data itself. This allows the system to maintain ease of operation while achieving reliable deviation detection for non-normal and multi-modal distributions
3Measurement precision
If database systems are used to analyze event streams, then comprehensive analysis is possible, but the data arrives faster than the system can process
Solution Approach 1:
The patent segments the analysis into incremental updates of kernel density estimates as events arrive, rather than requiring batch processing of complete datasets. This segmentation allows continuous stream processing with comprehensive distribution analysis, resolving the contradiction between analysis depth and processing speed by processing data in manageable increments as it arrives
4Measurement precision
If data mining algorithms are used, then thorough data analysis can be performed, but multiple runs over the data are required which is not feasible for continuous streams
Solution Approach 1:
The patent implements continuous kernel density estimation that updates incrementally as new events arrive in the stream, eliminating the need to stop and re-run analysis. This continuous action maintains thorough analysis depth while preventing time loss by processing data in a single continuous pass rather than requiring multiple discrete runs
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Certain example embodiments described herein relate to systems and/or methods for event stream deviation detection. More particularly, certain example embodiments described herein relate to maintaining short and long-term statistics of an incoming stream of event data. In certain example embodiments, a deviation is calculated based at least in part on the long-term and short-term statistics. The deviation may then be compared to a threshold value. In certain example embodiments, the estimations required for the statistics are done with Kernel Density Estimators (KDEs).