Iterative Autocorrelation Calculation for Streamed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data and streamed data are inefficient, requiring significant computational resources and failing to provide real-time decision-making capabilities, especially in applications like financial analysis, due to the need for large buffer sizes and redundant calculations during autocorrelation calculations.
Innovation Solution
The method iteratively calculates autocorrelation by updating only the components affected by new data elements in the computation window, reducing the need to revisit all data elements and minimizing redundant computations, thereby enhancing calculation efficiency and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional autocorrelation calculation methods are used on streamed data, then calculation accuracy is maintained, but computational load and processing time increase significantly
Solution Approach 1:
The patent segments the autocorrelation calculation into distinct components: mean calculation, variance calculation, and covariance calculation. Each component is computed separately and then combined to produce the final autocorrelation result. This segmentation allows for optimized computation of each part and enables parallel processing, thereby improving calculation efficiency while reducing overall computational load.
Solution Approach 2:
The patent performs preliminary calculations of the mean and variance before computing the covariance. By pre-computing these statistical parameters and storing them, the method avoids redundant calculations during the autocorrelation computation phase. This preliminary action significantly reduces the computational burden, especially when dealing with large buffered datasets.
2Measurement precision
If large computation windows are used for autocorrelation analysis, then statistical accuracy is improved, but buffer requirements and memory usage increase
Solution Approach 1:
The patent divides the large computation window into smaller manageable segments or blocks. Instead of loading the entire large window into memory at once, the method processes data in smaller chunks, computing statistical parameters for each segment and then aggregating the results. This approach maintains statistical accuracy over the full computation window while significantly reducing the buffer size required at any given time.
Solution Approach 2:
The patent implements a nested processing structure where smaller computation windows or blocks are nested within the larger computation window. Each nested block is processed independently, and their results are combined to produce the final autocorrelation for the entire large window. This nesting strategy enables accurate analysis of large datasets with limited memory resources.
3Speed
If autocorrelation is recalculated frequently for real-time decisions, then responsiveness is improved, but computational overhead and power consumption increase
Solution Approach 1:
The patent establishes continuous computation streams where statistical parameters (mean, variance, covariance) are updated incrementally as new data arrives, rather than performing complete recalculations. This continuous update mechanism maintains real-time responsiveness by always having current statistical values ready, while minimizing computational overhead through incremental updates instead of frequent full recalculations.
Solution Approach 2:
The patent pre-computes and maintains running statistics (mean, variance) that are updated continuously as data flows through the system. These pre-computed parameters are readily available when autocorrelation calculations are needed, eliminating the need for expensive full recalculations and enabling rapid response to new data while keeping power consumption low.
4Reliability
If all data elements in the computation window are accessed for each autocorrelation calculation, then calculation completeness is ensured, but processing time and computational complexity increase
Solution Approach 1:
The patent segments the data access pattern by organizing computations around pre-calculated statistical parameters rather than accessing individual data elements. Instead of visiting all n data elements for each autocorrelation calculation, the method accesses the already-computed mean, variance, and covariance parameters, which encapsulate the information from all data elements. This ensures calculation completeness while dramatically reducing processing time.
Solution Approach 2:
The patent creates and uses copies of statistical parameters (mean, variance, covariance) that represent the entire data window. Rather than accessing the original data elements repeatedly, the system works with these computed parameter copies, which contain the essential statistical information. This copying approach maintains calculation reliability while minimizing data access time and computational complexity.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for iteratively calculating autocorrelation at a specified lag for streamed data in real time by iteratively calculating one or more components of autocorrelation at the specified lag l for a computation window of size n. Embodiments of the invention include iteratively calculating one or more components of autocorrelation at the specified lag l for an adjusted computation window based on the one or more components of the autocorrelation at the specified lag l calculated for a previous computation window and then calculating the autocorrelation at the specified lag l using the components. Iteratively calculating autocorrelation avoids visiting all data elements in the adjusted computation window and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.


