Consistent Hashing for Distributed Deduplication Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed deduplication storage systems face challenges in efficiently managing fluctuating numbers of deduplication nodes, leading to resource consumption and impediment of other requests or services due to significant data redistribution.
Innovation Solution
The implementation of consistent hashing to assign backup data to deduplication nodes based on metadata, including the data itself, allows for efficient redistribution when node numbers change, minimizing resource consumption by updating fewer hash keys and reassigning less data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If traditional data redistribution is performed when deduplication nodes are added or removed, then the system can maintain data balance, but significant I/O resources are consumed and other requests or services are impeded
Solution Approach 1:
The system segments the data space into discrete hash key buckets that can be independently assigned to nodes. When nodes are added or removed, only the specific buckets affected by the change need to be redistributed, rather than redistributing all data. This segmentation enables partial updates that maintain data balance while minimizing I/O resource consumption.
Solution Approach 2:
The patent implements partial redistribution by identifying and updating only the subset of data buckets that need reassignment when node topology changes. Instead of performing a complete system-wide redistribution, the system calculates which specific hash keys are affected and redistributes only those portions, significantly reducing I/O overhead while maintaining overall data balance.
2Adaptability or versatility
If traditional data redistribution is performed when node numbers fluctuate, then data can be reassigned to new or remaining nodes, but the process impedes other requests or services
Solution Approach 1:
By segmenting data into independent hash key buckets, the system can perform localized redistribution operations that do not block other service requests. The segmented structure allows parallel processing where data reassignment occurs in the background for affected buckets while other services continue to access unaffected data, maintaining productivity during node fluctuations.
Solution Approach 2:
The system performs preliminary calculations to identify which specific hash keys need reassignment before actual data movement begins. This preliminary action allows the system to prepare reassignment plans that minimize disruption, pre-compute new node assignments for affected buckets, and execute changes with reduced impact on ongoing service requests.
3Stability of the object's composition
If all hash keys are updated when nodes change, then data can be evenly redistributed, but resource consumption increases significantly
Solution Approach 1:
The system extracts and identifies only the specific hash keys that are affected by node additions or removals, rather than processing all hash keys in the system. By taking out only the relevant subset of hash keys for update, the system maintains data distribution uniformity where needed while avoiding the computational overhead of updating unrelated hash keys, thus reducing resource consumption.
Solution Approach 2:
The patent implements partial hash key updates by calculating which specific buckets require reassignment based on the changed node topology. Only those specific hash keys are updated to maintain uniform data distribution, while the majority of hash keys remain unchanged, significantly reducing computational resource consumption compared to a full system-wide update.
Data Source
AI summary
A set of metadata associated with backup data is obtained. A consistent hash key for the backup data is generated based at least in part on the set of metadata. The backup data is assigned to one of a plurality of deduplication nodes based at least in part on the consistent hash key.