Incremental Correlation Calculation for Streamed Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for processing and analyzing Big Data, particularly streamed data, are inefficient and costly, leading to slow response times and significant waste of computing resources due to the need to recalculate correlations for large data sets, which is exacerbated by the rapid growth and complexity of data in real-time applications like finance and online gaming.
Innovation Solution
A computing system that incrementally calculates correlation components for modified computation subsets based on previous subsets, avoiding the need to revisit all data elements and store entire subsets, thereby improving efficiency and reducing resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional correlation calculation methods are used on Big Data or streamed data, then accurate correlation results can be obtained, but the computational load and processing time increase significantly
Solution Approach 1:
The patent segments the computation subset into manageable components and processes data elements incrementally rather than all at once. The correlation calculation is divided into sequential steps where each new data element is processed individually, updating the correlation result progressively. This segmentation allows the system to maintain accuracy while reducing the computational burden of processing entire large datasets simultaneously.
Solution Approach 2:
The patent performs preliminary actions by initializing correlation components with the first pair of data elements before processing subsequent elements. The system pre-establishes the baseline correlation state and then incrementally updates it, avoiding the need to recalculate from scratch each time new data arrives. This preliminary setup enables faster subsequent processing while maintaining calculation accuracy.
2Measurement precision
If all data elements in computation subsets are accessed for recalculating correlation, then accurate results are achieved, but computing resources are wasted and response time slows down
Solution Approach 1:
The patent extracts and processes only the newly added data element pair instead of accessing and processing all data elements in the computation subsets. By taking out just the relevant new information and integrating it into the existing correlation calculation, the system achieves accurate results while minimizing computing resource consumption. This extraction approach avoids the energy waste of repeatedly processing the entire dataset.
3Adaptability or versatility
If large computation subsets are used to process Big Data, then comprehensive analysis is achieved, but buffer requirements and storage needs increase significantly
Solution Approach 1:
The patent segments the computation subset processing into incremental steps, handling data elements sequentially rather than loading entire large subsets into buffer memory at once. This segmentation allows comprehensive analysis of Big Data while keeping buffer requirements manageable, as the system processes and discards data elements as they are incorporated into the correlation calculation.
4Reliability
If correlation is recalculated whenever data subsets change, then real-time accuracy is maintained, but processing time and computational overhead increase
Solution Approach 1:
The patent performs preliminary actions by establishing the initial correlation state with the first data pair and then incrementally updating it with each new element. This preliminary setup allows the system to maintain real-time accuracy without performing complete recalculations, as the incremental updates build upon the pre-established baseline. The system achieves reliability while minimizing the time loss associated with recalculation.
Solution Approach 2:
The patent maintains continuity of useful action by continuously updating the correlation result as new data elements arrive, rather than performing discrete complete recalculations. The incremental update process ensures the correlation remains accurate in real-time while avoiding the time overhead of repeated full recalculations, as each update builds continuously on the previous state.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for incrementally calculating correlation for Big Data or streamed data. Embodiments of the invention include incrementally calculating one or more components of a correlation for two modified computation subsets based on one or more components calculated for two previous computation subsets and then calculating the correlation based on the incrementally calculated components. Incrementally calculating the components of a correlation avoids visiting all pairs of data elements in the two modified computation subsets and performing redundant computations thereby increasing calculation efficiency, saving computing resources and reducing computing system's power consumption.