Decremental Correlation Calculation for Streamed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data, particularly streamed data, are inefficient and resource-intensive, as they require recalculating correlations for large datasets, leading to high computational costs and power consumption.
Innovation Solution
The method involves decrementally calculating correlation components by removing and updating only the affected components of the correlation calculation, rather than recalculating the entire dataset, using a computing system that accesses and modifies computation subsets to directly or indirectly calculate the necessary components.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional correlation calculation methods are used on Big Data or streamed data, then accurate correlation results can be obtained, but computational resources and time consumption increase significantly
Solution Approach 1:
The patent segments the correlation calculation into incremental updates. Instead of recalculating the entire correlation from scratch, the system divides the computation into small incremental steps that process only the newly added data elements, maintaining accuracy while dramatically improving efficiency.
Solution Approach 2:
The system performs preliminary calculations by maintaining running statistics (sums, means, variances) as data elements are added to the computation subset. These preliminary results are then used as the basis for subsequent correlation calculations, avoiding redundant computations and reducing overall processing time.
2Measurement precision
If all data elements are accessed and stored for correlation recalculation, then complete and accurate results are achieved, but memory requirements and storage costs increase
Solution Approach 1:
The patent extracts only the essential statistical components (sums, means, variances) needed for correlation calculation from the full data set. By maintaining only these extracted statistics rather than storing all原始 data elements, the system achieves complete correlation results with minimal memory and storage requirements.
Solution Approach 2:
The system performs partial action by calculating only the specific statistical components necessary for correlation rather than processing or storing all data elements. This selective approach maintains calculation completeness while reducing resource consumption to the minimum necessary level.
3Reliability
If correlation is recalculated whenever data elements are removed from computation subsets, then up-to-date correlation values are maintained, but computational overhead and power consumption increase
Solution Approach 1:
The system performs preliminary maintenance of statistical components (sums, means, variances) as data elements are removed. By keeping these components continuously updated in an incremental manner, the system ensures correlation values remain current without requiring expensive full recalculations, thereby reducing power consumption while maintaining reliability.
4Measurement precision
If large computation subsets are used for correlation calculation, then statistical significance is improved, but processing time and computational complexity increase
Solution Approach 1:
The patent segments the large computation subset processing into incremental updates. By dividing the work into small sequential steps that process data elements individually or in small batches, the system maintains statistical significance from large data sets while avoiding the computational bottleneck of processing all elements simultaneously, thus reducing processing time.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for decrementally calculating correlation for Big Data or streamed data. Embodiments of the invention include decrementally calculating one or more components of a correlation for two modified computation subsets based on one or more components of the correlation calculated for two previous computation subsets and then calculating the correlation for the modified computation subsets based on the decrementally calculated components. Decrementally calculating the components of a correlation avoids visiting all data elements in the modified computation subsets and performing redundant computations thereby increasing calculation efficiency, saving computation resources, and reducing computing system's power consumption.