Database Partition Replication for Balanced Cluster Reconfiguration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed file systems, particularly in massively parallel processing (MPP) database systems, face challenges in workload balancing and data availability due to unbalanced distribution of data among nodes, especially during node failures or additions, which affects throughput and latency.
Innovation Solution
A method and apparatus that evenly distribute primary and secondary database partitions among computer nodes, ensuring each secondary partition is on a different node than its corresponding primary partition, with secondary partitions being upgraded to primary status upon node failure to maintain workload balancing and data availability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is distributed among nodes using consistent hashing algorithms, then data redundancy is achieved, but workload becomes unbalanced when nodes are added or fail
Solution Approach 1:
The patent segments data into primary partitions and secondary partitions, with each partition type serving distinct functions. Primary partitions handle active workload while secondary partitions provide redundancy and serve as backups, allowing independent management of each partition type to maintain both redundancy and workload balance
Solution Approach 2:
The system pre-distributes secondary partitions to different nodes before failures occur, ensuring that when a node fails, its primary partitions can be immediately replaced by pre-positioned secondary partitions from other nodes, maintaining workload balance without requiring complex real-time redistribution calculations
2Quantity of substance
If secondary partitions are stored on the same node as primary partitions, then storage efficiency is improved, but data availability and workload balancing suffer during node failures
Solution Approach 1:
The patent applies different quality requirements to different partition types: primary partitions are stored on nodes optimized for serving workload, while secondary partitions are deliberately placed on different nodes with different characteristics, creating local specialization that improves both storage efficiency and failure resilience
Solution Approach 2:
Secondary partitions act as intermediary copies that mediate between primary partitions and failed nodes. When a node fails, secondary partitions from other nodes serve as intermediaries to replace the lost primary partitions, ensuring data availability while maintaining the original storage efficiency benefits
3Productivity
If workload balance is prioritized during normal operation, then throughput is improved, but latency increases during node failures due to redistribution overhead
Solution Approach 1:
Secondary partitions are pre-distributed to appropriate nodes during normal operation, so when a failure occurs, the system can immediately promote secondary partitions to primary status without requiring time-consuming redistribution calculations or data movement, thus maintaining low latency during failures while preserving throughput during normal operation
Solution Approach 2:
The system creates copy partitions (secondary partitions) that are identical to primary partitions but stored on different nodes. These copies can be instantly activated when primaries fail, eliminating the need for complex real-time redistribution and minimizing latency during failure events
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method for distributing a plurality of data partitions among a cluster of computer nodes includes dividing a database into a plurality of primary partitions, storing the primary partitions on a plurality of nodes associated with a computer network, duplicating the primary partitions to create a plurality of secondary partitions and storing the secondary partitions on the nodes, wherein the primary and secondary partitions are substantially evenly distributed among the nodes, each secondary partition is stored on a different node than its corresponding primary partition, diversity exists with respect to each secondary partition and the node where it is stored, the primary partitions are available to serve a system workload, and one or more secondary partitions corresponding to one or more primary partitions of a failed node are available to be upgraded to primary status to maintain workload balancing and data availability after one of the computer nodes fails.