Workload Balancing During Failover in Share-Nothing Database Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed computing systems face scalability issues and performance degradation when a data processing host fails, as existing methods often require reassigning data partitions to a single host, leading to increased processing time and workload imbalance.
Innovation Solution
A method is introduced to reassess and reassign data partitions among surviving hosts and standby hosts based on processing power and requirements, using calculations such as weighted data quantities and normalized workloads to optimize workload balancing and minimize downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data partitions are reassigned to a single host after failure, then system simplicity is maintained, but processing time increases and workload imbalance occurs
Solution Approach 1:
The patent segments the failed host's data partitions and distributes them across multiple surviving hosts based on their available capacity and workload characteristics. Instead of consolidating all partitions to one host, the system divides the redistribution task among several hosts, thereby reducing processing time while maintaining manageable complexity through automated allocation algorithms.
Solution Approach 2:
The patent introduces a new dimension of workload distribution by considering multiple hosts simultaneously rather than single-host consolidation. It uses multi-dimensional metrics including host capacity, current workload, data partition size, and processing requirements to optimize the redistribution strategy across the cluster.
2Productivity
If data partitions are redistributed across multiple hosts, then processing time decreases and workload balance improves, but system complexity increases
Solution Approach 1:
The patent implements self-service mechanisms where the system automatically evaluates host capacities, calculates optimal partition assignments, and executes redistribution without manual intervention. The automated algorithms continuously monitor system state and dynamically adjust workload distribution, eliminating the need for complex manual management while maintaining high productivity.
Solution Approach 2:
The patent dynamically changes system parameters such as host capacity weights, partition assignment criteria, and load balancing thresholds based on real-time system conditions. These parameter adjustments allow the system to adapt to changing workloads and host performances, optimizing processing speed while keeping complexity manageable through configurable parameters.
3Stability of the object's composition
If existing workload assignment methods are used during failover, then system stability is maintained, but performance degradation occurs due to suboptimal workload balancing
Solution Approach 1:
The patent incorporates feedback mechanisms that continuously monitor host performance metrics, partition processing status, and system load conditions. This feedback information is used to dynamically adjust workload assignments during failover, ensuring that partitions are allocated to hosts that can process them most efficiently while maintaining system stability through controlled adaptation.
Solution Approach 2:
The patent transitions from static workload assignment to dynamic allocation during failover. The system continuously evaluates host capacities and workload conditions, adjusting partition assignments in real-time based on current system state. This dynamic approach maintains stability by using controlled adaptation while improving performance through optimized workload distribution.
Data Source
AI summary
Procedures and systems may be used for assigning data partitions to data-processing host computers, for example, to initially assign data partitions at the outset of a large data-processing job or during failover measures taken in response to a failed host in a share-nothing database management system (SN-DBMS). In one implementation, a method of managing exclusive access to a data partition within a database system assesses a first host and a second host that have exclusive access to a first and second data partition, respectively, within a database system. The method assigns exclusive access of the data partition to one of the first and second hosts based on factors that may include the processing powers of first and second the hosts, and on processing requirements (such as data quantity and data criticalness) for data on the first and second data partitions.