Non-Power-of-Two Hash Partitioning for Database Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database management systems face challenges in efficiently processing large datasets due to limitations in operational memory, leading to performance degradation as data spills into slower memory, and the partitioning operation consumes significant computing resources, often defeating the gains from parallelization.
Innovation Solution
The implementation of hash distribution techniques, such as power-of-two partitioning and extended hash partitioning, to evenly distribute tuples among processing units, minimizing access to slower memory and optimizing resource utilization by generating partition identifiers from partition hashes, and using multiple iterations to achieve the target number of partitions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is loaded into operational memory for query processing, then query execution speed is improved, but memory capacity is exceeded and data spills into slower memory
Solution Approach 1:
The patent divides the large dataset into multiple partitions that can be processed separately. Each partition is smaller and can fit into operational memory, allowing parallel processing across multiple computing nodes while avoiding the need to load the entire dataset into memory at once.
Solution Approach 2:
The patent introduces the dimension of parallel processing across multiple computing nodes. By distributing data partitions across multiple nodes that can process simultaneously, the system achieves higher overall throughput without requiring each node to have excessive memory capacity.
2Productivity
If partition operation is performed to distribute data among processing units, then parallel processing efficiency is improved, but the partition operation itself consumes significant computing resources
Solution Approach 1:
The patent performs partitioning operations in advance, before query execution. The computed partition identifiers are stored with the data tuples, so that during query processing, no additional partitioning computation is needed. This preliminary action shifts the computational burden to a separate setup phase.
Solution Approach 2:
Each computing node independently determines its own partition identifiers using local computations based on the partitioning scheme. This self-service approach eliminates the need for centralized partition assignment and reduces communication overhead across the distributed system.
3Quantity of substance
If range distribution is used for partitioning, then data can be distributed among nodes, but additional computational steps are required for each tuple
Solution Approach 1:
The patent replaces the complex range-based computational approach with a simpler hash-based partitioning mechanism. Instead of computing which range a tuple falls into, the system computes a hash value and uses that to directly determine the partition identifier, significantly reducing the computational steps required per tuple.
Solution Approach 2:
The patent changes the partitioning parameter from range boundaries to hash values. This parameter transformation simplifies the partitioning logic from complex conditional checks to straightforward hash computation and mapping, reducing the computational overhead for each tuple.
Data Source
AI summary
Techniques for non-power-of-two partitioning of a data set as well as generation and selection of partition schemes for the data set. In an embodiment, one or more iterations of a partition scheme is for a non-power-of-two number of partitions. Extended hash partitioning may be used to partition a data set into a non-power-of-two number of partitions by determining the partition identifier of each tuple of the data set using the extended hash partitioning algorithm. In an embodiment, multiple partition schemes are generated for multiple data sets, based on properties of the data sets and/or availability of computing resources for the partition operation or the subsequent operation to the partition operation. The generated partition schemes may use non-power-of-two partitioning for one or more iterations of a generated partition scheme. The most optimal partition scheme may be selected from the generated partition schemes based on optimization policies.