Real-time Streaming Data Analytics with Statistical Moment Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data analytics systems for machine-to-machine control struggle with real-time analysis of streaming data, requiring storage of historical data to set accurate thresholds, which is resource-intensive and inefficient.
Innovation Solution
A system that updates statistical moments of a data stream in real-time, discards each value after analysis, and sets dynamic thresholds based on these moments to trigger remedial actions, eliminating the need for data storage and reducing computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If historical data is stored to set accurate thresholds, then measurement precision is improved, but device complexity and resource consumption increase
Solution Approach 1:
The patent extracts only the essential statistical characteristics (moments) from the data stream needed for threshold calculation, rather than storing the complete historical data. By computing moments like mean and variance online and discarding individual data points after processing, the system achieves accurate threshold setting with minimal data retention, directly resolving the contradiction between measurement precision and device complexity
Solution Approach 2:
The patent transforms the raw data stream into statistical moment parameters (mean, variance, skewness, kurtosis) that capture the essential characteristics needed for threshold calculation. This parameter transformation allows accurate threshold determination without storing original data points, reducing device complexity while maintaining measurement precision
2Reliability
If historical data is stored for analysis, then reliability is improved, but loss of substance increases due to storage resource consumption
Solution Approach 1:
The system extracts only the necessary statistical information (moments) from the data stream for reliable analysis, discarding individual data points after moment calculation. This approach maintains analysis reliability by preserving essential statistical characteristics while minimizing hardware resource consumption by not storing complete historical data
Solution Approach 2:
The patent treats individual data points as temporary, disposable elements that are processed to extract statistical moments and then discarded. This disposable approach to data points enables reliable real-time analysis without the burden of long-term data storage, reducing hardware resource consumption while maintaining analysis accuracy
3Productivity
If real-time analysis is performed without data storage, then productivity is improved, but measurement precision may deteriorate
Solution Approach 1:
The patent pre-calculates and continuously updates statistical moments (mean, variance, skewness, kurtosis) as data arrives, preparing the necessary statistical foundation in real-time. This preliminary computation of moments enables immediate threshold evaluation and anomaly detection without requiring historical data storage, maintaining both productivity and measurement precision
Solution Approach 2:
By transforming raw data into statistical moment parameters in real-time, the system enables immediate threshold-based decision-making. The parameter transformation to moments provides sufficient statistical information for accurate threshold setting while enabling fast real-time processing, resolving the contradiction between productivity and measurement precision
Data Source
AI summary
A computer-implemented method includes receiving a plurality of values of a data stream, where the plurality of values are generated by a monitored device and received in real time. One or more statistical moments of the data stream are updated, by a computer processing device, based on each value of the plurality of values. Each value of the plurality of values is discarded, after updating the one or more statistical moments of the data stream based on that value. A threshold is set for the data stream based on the one or more statistical moments. It is detected that the threshold has been passed by the data stream. A remedial action is performed on the monitored device, responsive to the threshold being passed, where the remedial action is associated with the threshold.