Dynamic Correlation Batch Calculation for Big Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for calculating correlations in Big Data are inefficient, especially when dealing with large and dynamically changing data sets, as they require recalculating correlations for entire sets, leading to significant redundant data access and computation, which is time-consuming and resource-intensive.
Innovation Solution
A computing system dynamically calculates correlations by modifying the pre-existing correlation calculations by excluding and including data elements, using pre-calculated components to efficiently update the correlation without recalculating from scratch, especially when the changes are minor compared to the total data set size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional correlation calculation methods are used on modified computation sets, then complete and accurate correlation results are obtained, but significant redundant data access and computation occur leading to inefficiency
Solution Approach 1:
The patent segments the computation set into three parts: the original computation set, the excluded data elements, and the included data elements. By dividing the correlation calculation into these segments, the system can reuse pre-calculated correlation components from the original set and only compute correlations for the excluded and included elements, avoiding redundant calculations on the entire modified set.
Solution Approach 2:
The patent performs preliminary correlation calculations on the original computation set before modification occurs. These pre-calculated correlation components are stored and reused when the computation set is modified, eliminating the need to recalculate correlations for unchanged data elements and significantly reducing computational overhead.
2Measurement precision
If complete recalculation of correlations is performed on modified data sets, then accuracy is maintained, but calculation time increases significantly
Solution Approach 1:
The patent segments the correlation calculation into reusable components from the original set and new components from excluded/included elements. This segmentation allows the system to maintain accuracy by computing only the necessary new correlations while preserving the accuracy of pre-calculated components, thereby reducing total calculation time.
Solution Approach 2:
The patent introduces intermediate correlation components as mediators between the original correlation results and the final modified correlation results. These intermediate components represent the excluded and included data elements and serve as building blocks to construct the final accurate correlation without requiring complete recalculation.
3Adaptability or versatility
If traditional methods are used for dynamically changing data sets, then all data elements are processed, but redundant computation occurs when changes are minor
Solution Approach 1:
The patent implements a dynamic correlation calculation approach that adapts to changes in the computation set. Instead of static complete recalculation, the system dynamically identifies excluded and included data elements and adjusts the calculation scope accordingly, maintaining adaptability to changing data while improving efficiency by avoiding redundant computations.
Solution Approach 2:
The patent changes the calculation parameters from processing the entire modified computation set to processing only the difference set (excluded and included elements). This parameter change in the calculation scope maintains the ability to handle dynamic data sets while significantly improving calculation efficiency when changes are minor.
Data Source
AI summary
The present invention extends to methods, systems, and computing system program products for dynamic correlation batch calculation for Big Data. Embodiments of the invention include calculating a correlation for a modified computation set based on a group of components calculated for the pre-modified computation set and one or more groups of components calculated for a computation set to be excluded from the pre-modified computation set and a computation set to be included in the pre-modified computation set, where the size of the to-be-included computation set may or may not be equal to the size of the to-be-excluded computation set. When the size of the to-be-excluded computation set is smaller than half the size of the pre-modified computation set, dynamic correlation batch calculation may reduce computations thereby increasing calculation efficiency, saving computation resources, and reducing computing system's power consumption.