Incremental Autocorrelation Calculation for Streamed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data, particularly streamed data, are inefficient and costly, as they require extensive computational resources and often result in slow response times and redundant calculations, especially when calculating autocorrelations on large data sets.
Innovation Solution
A computing system that incrementally calculates autocorrelation at a specified lag by using pre-calculated components from a previous computation window, avoiding the need to revisit all data elements in the adjusted computation window, thereby reducing redundant computations and conserving resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional autocorrelation calculation methods are used on streamed data, then complete autocorrelation values can be obtained, but calculation time increases significantly and computing resources are wasted due to redundant computations
Solution Approach 1:
The patent pre-calculates and stores autocorrelation components (such as sum of products, sum of squares) as data elements are added to the computation window. These pre-computed components are then reused when new data arrives, avoiding redundant calculations. For example, when a new data point is added, the system updates previously calculated components incrementally rather than recalculating all autocorrelation values from scratch.
Solution Approach 2:
The patent divides the autocorrelation calculation into multiple independent components (e.g., different lag values, different statistical moments). Each component is calculated and stored separately, allowing the system to only update and recalculate the specific components affected by new data arrivals, rather than recomputing the entire autocorrelation function.
2Measurement precision
If traditional autocorrelation calculation methods are used on streamed data, then accurate statistical values can be obtained, but computing resources and power consumption increase due to visiting all data elements repeatedly
Solution Approach 1:
The system pre-computes and caches intermediate statistical components (such as running sums, running products, and running variances) as data flows through the computation window. When new data arrives, these pre-computed components are updated incrementally using efficient recurrence relations, significantly reducing the computational work required compared to recalculating all statistics from scratch.
Solution Approach 2:
The patent optimizes the calculation by focusing computational effort only on the local changes caused by new data arrivals. Instead of globally recalculating all autocorrelation values across the entire computation window, the system identifies and updates only the specific components and lag values that are actually affected by the new data point, leaving unchanged components untouched.
3Adaptability or versatility
If complete autocorrelation calculation is performed on large computation windows, then comprehensive statistical analysis is achieved, but device complexity and computational overhead increase
Solution Approach 1:
The patent decomposes the complex autocorrelation calculation into multiple simpler, independent computational stages. Each stage handles a specific aspect (e.g., calculating sum of products for different lags, calculating normalized values, handling window updates). This modular segmentation reduces the complexity of any single computational step while maintaining the comprehensive analytical capability.
Solution Approach 2:
The system dynamically adjusts its computational approach based on the current state of the computation window and the specific requirements of the analysis. As data flows through the system, the computational complexity adapts to the window size and data characteristics, using optimized algorithms for different scenarios (e.g., incremental updates vs. full recalculations, different lag ranges).
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for incrementally calculating autocorrelation for streamed data in real time. Embodiments of the invention include incrementally calculating one or more components of autocorrelation at a specified lag for an adjusted computation window based on the one or more components of the autocorrelation at the specified lag calculated for a previous computation window and then calculating the autocorrelation the specified lag using the components. Incrementally calculating autocorrelation avoids visiting and storing all data elements in the adjusted computation window and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.