Decremental Autocorrelation Calculation for Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data, particularly for real-time decision-making in the financial industry, are inefficient and costly due to the high computational resources required for calculations like autocorrelation, which often involve accessing and recalculating large data sets, leading to slow response times and resource wastage.
Innovation Solution
The method involves decrementally calculating autocorrelation by adjusting the computation window size and using previously calculated components to minimize the number of data elements accessed, thereby reducing redundant computations and conserving resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional autocorrelation calculation methods are used on Big Data sets, then calculation accuracy is maintained, but computational resource consumption increases significantly and response time slows down
Solution Approach 1:
The patent pre-calculates and stores autocorrelation components (sum of products, sum of squares) for data elements before they are needed for final autocorrelation calculation. When a computation window adjusts, these pre-computed components are reused instead of recalculating from scratch, significantly reducing computational resource consumption while maintaining calculation accuracy.
Solution Approach 2:
The autocorrelation calculation is divided into separate components (sum of products, sum of squares, mean values) that can be independently computed and stored. This segmentation allows each component to be calculated once and reused multiple times across different computation windows, reducing redundant calculations and energy consumption.
2Speed
If computation window size is decreased to exclude accessed data elements, then real-time decision making capability is improved, but the number of data elements to be accessed increases repeatedly
Solution Approach 1:
Autocorrelation components are pre-computed and stored for each data element before the computation window needs to adjust. When the window size decreases and data elements are removed, the pre-computed components are immediately reused to calculate the new autocorrelation values, avoiding repeated access and recalculation of the same data elements.
Solution Approach 2:
The patent creates and stores copies of autocorrelation components (sum of products, sum of squares) for data elements. These copied component values can be retrieved and reused when computation windows change, eliminating the need to repeatedly access and recalculate the original data elements, thus improving response time while reducing recalculation overhead.
3Reliability
If all data elements in the computation window are accessed for autocorrelation calculation, then complete statistical analysis is achieved, but computing resources are wasted on redundant computations
Solution Approach 1:
The autocorrelation calculation is segmented into distinct components (sum of products, sum of squares, means) that can be independently computed, stored, and reused. This allows the system to maintain complete statistical analysis by computing all necessary components once, while avoiding redundant computations by reusing these components across multiple calculation iterations and computation window adjustments.
Solution Approach 2:
All necessary autocorrelation components are pre-computed and stored before the final calculation is needed. This preliminary computation ensures complete statistical analysis is available while eliminating redundant computations when computation windows adjust, as the pre-computed components can be directly reused without re-accessing all data elements.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for decrementally calculating autocorrelation for Big Data. Embodiments of the invention include decrementally calculating one or more components of autocorrelation at a specified lag for an adjusted computation window based on the one or more components of an autocorrelation at the specified lag calculated for a previous computation window and then calculating the autocorrelation at the specified lag based on one or more of the decrementally calculated components. Decrementally calculating autocorrelation avoids visiting all data elements in the adjusted computation window and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.