Horizontal Data Partitioning for Aggregation Speed
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing techniques, such as those using lightweight indexes, are limited in their ability to handle complex queries, particularly when records are not fully included in the target set for aggregation processing, leading to invalid results.
Innovation Solution
A data processing device and method that horizontally partitions table data into blocks, calculates statistical values for each block, determines if all records in a block are selected based on the query, and uses the statistical value as the result for selected blocks, optimizing aggregation processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If lightweight indexes are used to accelerate aggregation processing, then processing speed is improved, but the technique becomes inapplicable to complex queries involving record selection
Solution Approach 1:
The table data is horizontally partitioned into multiple blocks, and for each block, separate statistical values are calculated and stored. This segmentation allows the system to handle different query types by selectively using statistical values from relevant blocks, thus maintaining both speed and versatility.
Solution Approach 2:
Statistical values for each block are pre-calculated and stored before query execution. This preliminary action enables fast query processing by allowing the system to directly use pre-computed statistical values when applicable, rather than computing them during query execution.
2Power
If statistical values from lightweight indexes are used for all blocks, then computation amount is reduced, but invalid results occur when records are not fully included in the target set
Solution Approach 1:
The system calculates and stores statistical values locally for each individual block rather than for the entire table. This local quality approach allows the system to determine result validity on a per-block basis, using statistical values only when all records in a block are included in the target set, thus ensuring reliability while maintaining computational efficiency.
3Measurement precision
If all blocks are processed using traditional scanning methods, then result accuracy is maintained, but processing time increases significantly
Solution Approach 1:
The system performs partial processing by using pre-calculated statistical values for blocks where all records are included in the target set, and only performs full scanning for blocks where some records are excluded. This partial action approach maintains accuracy for all blocks while reducing overall processing time by avoiding redundant full scans.
Data Source
AI summary
A data processing device according to the present invention includes: a partition unit that horizontally partitions records included in table data into a plurality of blocks, the horizontal partitioning indicating partitioning that uses records as a unit; a statistical value calculation unit that calculates, for each of the blocks, a statistical value of an attribute included in the records of the block; a determination unit that determines, when processing a query for performing aggregation processing after record selection processing with respect to the table data, based on the statistical value, for each of the blocks, whether all records in the block are selected or not based on the selection processing; and a query execution unit that uses, for a block determined that all records are selected based on the determination unit, the statistical value of the determined block as a result of the query for the determined block.