Balanced Consistent Hashing for Cluster Resource Redistribution

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In distributed computer clusters, existing methods for redistributing resource management responsibilities are inefficient, leading to increased downtime and failure to account for relative capabilities of nodes, resulting in suboptimal resource utilization and high redistribution costs.

Innovation Solution

A mechanism using balanced consistent hashing to redistribute resource management responsibilities, where the resource identification space is divided among nodes based on their relative capabilities, minimizing the number of resource masters relocated and conserving compute and network resources.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If resource management responsibilities are redistributed when a cluster member leaves, then the cluster maintains availability and eliminates single points of failure, but the redistribution process causes cluster downtime and consumes excessive compute and network resources

Engineering Contradiction:

Improvecluster availabilityVSAvoidredistribution downtime

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The resource identification space is segmented into multiple regions, with each region independently assigned to a specific cluster member. When a member leaves, only the regions it managed need to be redistributed, rather than redistributing all resource management responsibilities. This segmentation minimizes the scope of redistribution and reduces downtime.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system pre-establishes a mapping between resource identifiers and cluster members using consistent hashing. This preliminary arrangement ensures that when a member leaves, the system can quickly determine which resources need redistribution and assign them to appropriate members without causing widespread reconfiguration, thereby reducing redistribution time.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If resource management responsibilities are redistributed among remaining cluster members, then the cluster continues to operate without single points of failure, but the redistribution consumes excessive compute cycles and network bandwidth

Engineering Contradiction:

Improvefailure eliminationVSAvoidcompute and network resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

By dividing the resource identification space into distinct regions and assigning each region to a specific cluster member, the system ensures that only the regions previously managed by the departed member need to be redistributed. This localized redistribution significantly reduces the compute cycles and network bandwidth required compared to redistributing all resource management responsibilities.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies different assignment rules to different regions of the resource identification space based on the capabilities and status of cluster members. When a member leaves, only the local regions it managed are redistributed to appropriate members, rather than triggering a global reassignment. This local quality approach minimizes the resources consumed during redistribution.

Inventive Principle:

Principle #3Local quality

3Productivity

If resource management responsibilities are redistributed to maintain balanced load, then resource utilization is optimized, but the complexity of tracking and managing region assignments increases

Engineering Contradiction:

Improveresource utilization efficiencyVSAvoidassignment tracking complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system pre-establishes a consistent hashing function that deterministically maps resource identifiers to cluster members. This preliminary mapping structure allows the system to automatically determine which regions need redistribution when a member leaves, eliminating the need for complex manual tracking and assignment logic while maintaining balanced load distribution.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system uses consistent hashing to provide automatic feedback on the distribution of resource management responsibilities. When a member joins or leaves the cluster, the hashing function naturally indicates which regions should be reassigned, providing a simple feedback mechanism that maintains balanced load without requiring complex tracking infrastructure.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS8032637B2Balanced consistent hashing for distributed resource management

Publication Date: 2011.10.04 ARCTERA US LLC

AI summary

A method, system, computer-readable storage medium and apparatus for balanced and consistent placement of resource management responsibilities within a multi-computer environment, such as a cluster, that are both scalable and make efficient use of cluster resources are provided. Embodiments reduce the time that a cluster is unavailable due to redistribution of resource management responsibilities by reducing the amount of redistribution of those responsibilities among the surviving cluster members. Embodiments further provide redistribution of resource management responsibilities based upon relative capabilities of the remaining cluster nodes.