Data Pre-aggregation via Two-Pass Parallel Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multidimensional analysis with large data sets experiences low performance due to inefficient pre-aggregation, requiring excessive storage for all possible dimension combinations.
Innovation Solution
A system utilizing dynamic batch size, parallel thread processing, and reduced computations to enhance data pre-aggregation, performing two passes to determine subkey and key rollup data, and storing these into data blocks to form a pre-aggregated data cube.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all possible dimension combinations are pre-aggregated, then query response speed is improved, but storage capacity requirement increases excessively
Solution Approach 1:
The patent segments the pre-aggregation process into two distinct passes: a first pass that performs aggregation with a dynamic batch size, and a second pass that performs rollup operations. This segmentation allows the system to process data in manageable chunks rather than attempting to pre-aggregate all dimension combinations at once, thereby reducing storage requirements while maintaining query performance.
Solution Approach 2:
The patent introduces dynamic batch size adjustment in the first pass of the two-pass algorithm. The batch size is dynamically modified based on processing progress and system conditions, allowing optimal resource utilization during pre-aggregation. This dynamic approach enables efficient processing without requiring excessive storage capacity for all possible dimension combinations.
2Productivity
If traditional single-thread pre-aggregation is used, then system complexity is low, but processing speed and throughput are insufficient
Solution Approach 1:
The patent implements a two-pass algorithm where the first pass performs preliminary aggregation operations with dynamic batch sizing before the second pass performs rollup operations. This preliminary action in the first pass prepares the data in an optimized format, enabling faster processing in the second pass while distributing the computational workload efficiently across multiple threads.
Solution Approach 2:
The patent ensures continuous processing through its two-pass algorithm design. The first pass continuously processes data batches with dynamic sizing, and the second pass continuously performs rollup operations on the aggregated results. This continuous action across both passes maximizes processor utilization and throughput without requiring complex parallel processing coordination.
3Measurement precision
If excessive computations are performed during aggregation, then aggregation accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies partial action by performing aggregation in two distinct passes rather than attempting to do everything in one pass. The first pass performs partial aggregation with dynamic batch sizing, and the second pass performs the remaining rollup operations. This partial action approach maintains aggregation accuracy while significantly reducing the computational burden and processing time compared to performing all operations in a single pass.
Data Source
AI summary
A system for enhanced data pre-aggregation is provided. In one embodiment, a method is provided that includes receiving data formatted in a key/subkey format and distributing a data batch of the data to a plurality of processing threads. Each processing thread performs operations of: performing a first pass on the data batch to determine subkey rollup data; performing a second pass on the data batch to determine key rollup data; and storing the subkey rollup data and the key rollup data into data blocks. The method also includes outputting the data blocks to form a pre-aggregated data cube.