Database Partition Splitting via Logical Midpoint
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Database partitioning in data centers often requires manual intervention and can lead to performance penalties, especially in consistent hash systems, as it struggles to manage uneven data distribution and maintain performance during split operations.
Innovation Solution
Implementing a logical split point for hash-range partitions, using the median of hash and range value pairs to determine the split point, which allows for automatic partitioning without user interaction and minimizes performance loss, while also considering the distribution skewness and range relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If manual partitioning is used to manage database splits, then partitioning can be performed, but it requires manual intervention and increases operational complexity
Solution Approach 1:
The system automatically detects when a partition should be split based on predefined criteria (such as partition size thresholds) and performs the split operation without requiring manual intervention. The client monitors partition health metrics and autonomously manages the partitioning process, eliminating the need for administrators to manually track and split partitions.
2Reliability
If consistent hashing is used to distribute data across partitions, then data distribution is maintained, but performance penalties occur during split operations
Solution Approach 1:
The system pre-calculates the split point and prepares the partition for splitting before actual data migration begins. By determining the optimal split point in advance based on current data distribution and access patterns, the system minimizes disruption to ongoing queries and maintains consistent hashing properties throughout the transition.
Solution Approach 2:
The partitioning system dynamically adjusts the hashing distribution during split operations, allowing queries to be routed appropriately to either the original partition or the new partition based on the current state of the system. This dynamic routing maintains query performance while the data distribution is being reorganized.
3Stability of the object's composition
If partitions are split frequently to maintain balance, then data distribution remains even, but the number of split operations increases system overhead
Solution Approach 1:
The system continuously monitors partition characteristics such as size, data distribution, and access patterns, and only initiates a split operation when predefined thresholds are exceeded. This feedback mechanism ensures that partitions are split only when necessary to maintain balance, avoiding unnecessary split operations that would consume system resources and time.
Data Source
AI summary
Techniques are disclosed for splitting a database partition into two partitions. In embodiments, where the partition is a hash partition, the partition is split at its logical midpoint, so that half of the partition's rows are placed in each split partition. Where the partition is a hash-range partition, a number of rows of the partition are sampled. Where enough samples fall on each side of the logical midpoint, the logical midpoint is again used as the split point. Where not enough samples fall on one side of the logical midpoint, then the median of the samples is used as the split point.