Multi-pass Algorithm for High Cardinality Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Calculating accurate summary statistics for large data sets with high cardinality is inefficient due to the need to process every data point and require significant memory for intermediate computations, leading to exponential increases in processing and storage demands.
Innovation Solution
A method that divides the data processing into multiple phases using a hash function to assign identical data elements to the same phase, allowing for in-memory processing and rapid erasure of irrelevant intermediate results, and utilizes a grid-computing system to distribute and parallelize data processing, reducing memory usage and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods process every data point in large high-cardinality data sets, then accurate summary statistics can be calculated, but processing time and memory usage increase exponentially
Solution Approach 1:
The patent divides the large high-cardinality data set into multiple partitions or buckets based on hash values of the data elements. Each partition contains a subset of the data, allowing parallel processing across multiple computing devices. This segmentation reduces the processing time for each individual partition while maintaining the accuracy of overall summary statistics through proper aggregation of results from all partitions.
2Measurement precision
If traditional methods process every data point in large high-cardinality data sets, then accurate summary statistics can be calculated, but memory usage for intermediate computations increases exponentially
Solution Approach 1:
The patent segments the data into multiple partitions that can be processed independently in parallel. Each partition fits within the memory capacity of individual computing devices, avoiding the exponential memory growth that would occur if all data were processed simultaneously. The memory-efficient approach allows accurate summary statistics to be calculated by aggregating results from multiple smaller in-memory partitions.
Solution Approach 2:
The patent introduces intermediate summary statistics from each partition as mediators that are aggregated to produce the final summary statistics. Instead of loading all raw data into memory, the system computes intermediate results for each partition and then combines these intermediates, significantly reducing peak memory usage while maintaining computational accuracy.
3Productivity
If a grid-computing system parallelizes data processing across multiple devices, then processing time is reduced, but system complexity increases
Solution Approach 1:
The patent divides the data processing task into independent partitions that can be assigned to different grid-computing devices. Each device processes its assigned partition independently using the same algorithm, which simplifies the coordination complexity. The segmented approach allows linear scaling of processing speed with the number of devices while keeping individual device logic relatively simple.
Solution Approach 2:
The patent implements a universal processing algorithm that can be executed identically across all grid-computing devices. Each device performs the same multi-pass algorithm on its local partition, making the system highly parallelizable. This universality reduces system complexity by eliminating the need for different processing logic on different devices, while still achieving high processing speeds through parallel execution.
Data Source
AI summary
This disclosure describes methods, systems, computer-readable media, and apparatuses for calculating a summary statistic. Calculating the summary statistic can be performed by identifying multiple subsets of a set of variable observations and assigning the subsets to grid-computing devices such that no two of the subsets are assigned to a same one of the grid-computing devices. A parallel processing operation that involves multiple processing phases at each of the grid-computing devices is then coordinated. The parallel processing operation includes each of the grid-computing devices inventorying the respectively assigned subset and generating inventory information representative of the respectively assigned subset. Subsequently, the inventory information generated by the grid-computing devices is received, and a summary statistic is determined by synthesizing the received inventory information.