Hash Map Aggregation for High-Cardinality Parallel Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hash map data structures in database systems with high-cardinality database tables lead to inefficient memory usage and limited performance in multiple-core computing systems due to unnecessary memory allocations and probing functions.
Innovation Solution
Implementing a data aggregation method that uses duplicate functions instead of probing functions for hash map data structures, and storing pointers in merged hash maps to improve memory allocation efficiency, particularly in parallel computing environments with shared memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If probing functions (e.g., linear probing) are performed on hash maps for data aggregation, then data grouping capability is improved, but memory space allocation increases and efficiency decreases for high-cardinality tables
Solution Approach 1:
The patent changes the operational parameters of hash maps by switching from probing functions to duplicate functions based on cardinality thresholds. For high-cardinality tables, duplicate functions are used which allocate memory differently, avoiding the excessive memory allocation caused by probing functions while maintaining data grouping capability.
Solution Approach 2:
The system dynamically selects between probing functions and duplicate functions based on the cardinality of the database table. This dynamic adaptation allows the system to optimize memory allocation and performance according to the specific characteristics of the data being processed, rather than using a fixed approach.
2Productivity
If probing functions are performed on hash maps, then data aggregation is achieved, but performance of multiple-core computing systems is limited
Solution Approach 1:
By changing the function type from probing to duplicate based on cardinality parameters, the system achieves faster processing times. Duplicate functions require fewer computational steps and memory operations, directly improving data aggregation performance while reducing processing time in parallel computing environments.
3Adaptability or versatility
If hash maps with probing functions are used for high-cardinality tables, then data grouping is performed, but memory usage efficiency decreases
Solution Approach 1:
The patent applies parameter changes by selecting different hash map operations based on cardinality thresholds. For high-cardinality tables, duplicate functions are used which have better memory usage efficiency compared to probing functions, as they avoid unnecessary memory allocations and probes while maintaining the essential data grouping functionality.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for providing data aggregation based on hash map data structures. An embodiment operates by receiving a query specifying an input table and an aggregate function. The embodiment then generates a first thread-local hash map and a second thread-local hash map for the input table and performs a probing function associated with the first thread-local hash map. The embodiment then determines, based on the first thread-local hash map, an index cardinality associated with the input table. The embodiment then, in response to the determination that the index cardinality exceeds the threshold, performs a duplicate function associated with the second thread-local hash map and generates a second thread-local copy map. The embodiment then merges the first thread-local hash map and the second thread-local copy map, thereby generating a merged hash map.