Consistent Hashing for Distributed Storage Node Selection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional storage technologies face challenges in meeting the requirements for mass data storage in terms of scalability, fault tolerance, availability, and deployment costs, particularly due to inefficiencies in node selection and redundancy in distributed storage systems.

Innovation Solution

A method using consistent hashing with a random selection algorithm and monitored weights to improve node selection efficiency by mapping object identities to hash subspaces and selecting storage nodes, reducing time complexity and minimizing conflicts during node selection.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If traditional storage technologies (NFS, SAN) are used for mass data storage, then data storage capacity can be achieved, but scalability and deployment costs are compromised

Engineering Contradiction:

Improvedata storage capacityVSAvoidscalability

Core Design Contradiction:

Quantity of substanceVSAdaptability or versatility

Solution Approach 1:

The patent segments the distributed storage system into multiple independent storage nodes, each capable of autonomous operation. Data is divided into fragments and distributed across these nodes, enabling the system to scale by simply adding more nodes without reconfiguring the entire system. This segmentation approach directly resolves the scalability limitation of traditional NFS/SAN storage technologies.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces consistent hashing as a mathematical mapping function that transforms the node selection problem from a linear search into a hash-based direct access mechanism. By mapping data fragments to storage nodes through hash functions, the system achieves O(1) time complexity for node selection, adding a dimensional transformation to the storage architecture that enables efficient scaling.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Quantity of substance

If traditional storage technologies are used, then data storage can be implemented, but fault tolerance and availability are insufficient

Engineering Contradiction:

Improvedata storage capacityVSAvoidfault tolerance

Core Design Contradiction:

Quantity of substanceVSReliability

Solution Approach 1:

The patent implements local quality by storing multiple copies of data fragments across different storage nodes based on their individual capacities and reliability characteristics. Each node can have different storage weights and reliability profiles, and the system selectively places data copies on nodes that provide optimal fault tolerance. This local differentiation enables the system to achieve high availability while accommodating heterogeneous node qualities.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent employs prior cushioning by pre-calculating and distributing redundant data copies across multiple storage nodes before any failure occurs. The consistent hashing mechanism proactively assigns data to N different nodes, creating a buffer against potential failures. When node failures occur, the system can immediately retrieve data from remaining nodes without service interruption, effectively cushioning against faults.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

3Productivity

If conventional node selection methods are used in distributed storage, then data can be stored, but node selection efficiency and convergence speed are poor

Engineering Contradiction:

Improvestorage operation speedVSAvoidnode selection time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The patent replaces the mechanical linear search mechanism for node selection with a mathematical hash function-based system. Instead of sequentially searching through storage node lists to find appropriate nodes, the system uses consistent hashing to directly compute the target node from the data fragment identifier. This substitution transforms the node selection process from an O(N) mechanical search to an O(1) mathematical operation, dramatically improving efficiency.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent changes the fundamental parameter of node selection from sequential position-based indexing to hash value-based addressing. By transforming the selection criterion from linear position to cryptographic hash output, the system achieves constant-time node selection regardless of the number of storage nodes. This parameter transformation enables the system to maintain high operation speed even as the cluster scales to thousands of nodes.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS11294601B2Method of distributed data redundancy storage using consistent hashing

Publication Date: 2022.04.05 HERE DATA TECH

AI summary

Disclosed is a method of distributed data redundancy storage using consistent hashing, which is used for selection of pre-stored data in the distributed position in storage cluster. The method is implemented by establishing a logical topology for each storage node in a distributed storage system, determining the position of each node in a storage node sequence corresponding to each hash subspace by using consistent hashing, and then determining a number of data copies and each piece of stored data according to the redundant storage policy of pre-stored data; and storing each piece of stored data in a different storage node according to storage rules.