Consistent Hashing for Similarity Group Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional resource distribution algorithms in clustered data protection systems inefficiently redistribute resources when nodes become unavailable or available, consuming significant system resources and causing performance bottlenecks due to hotspots and reconfiguration events.
Innovation Solution
Implement an affinity-based load-balancing distribution of similarity groups using consistent hashing to assign and reassign similarity groups to deduplication object services, ensuring minimal resource reallocation and co-location of metadata, reducing the need for remote procedure calls and optimizing workload distribution across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional resource distribution algorithms are used to redistribute resources when nodes become unavailable or available, then resource allocation can be updated, but significant system resources are consumed and performance bottlenecks occur due to hotspots and reconfiguration events
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing fingerprint indexes for similarity groups before nodes become unavailable. When node failures occur, the system can quickly reassign similarity groups using pre-computed fingerprints without performing expensive real-time comparisons, thus avoiding performance bottlenecks during reconfiguration events
Solution Approach 2:
The patent implements local quality by maintaining local fingerprint indexes at each node for its assigned similarity groups. This allows nodes to independently handle resource redistribution using their own local data without requiring global system-wide resource consumption, reducing hotspots and improving overall system performance during failover events
2Measurement precision
If direct comparison of data file segments is performed to identify duplicates, then accurate duplicate detection can be achieved, but system resources are wasted by comparing thousands of bytes in each segment against millions of stored segments
Solution Approach 1:
The patent applies segmentation by dividing data file segments into fixed-size chunks and computing fingerprints for each chunk. This segmentation allows the system to compare compact fingerprint representations rather than full data segments, dramatically reducing computational resources while maintaining duplicate detection accuracy through the fingerprint matching mechanism
Solution Approach 2:
The patent introduces fingerprints as an intermediary between data file segments and the comparison process. Instead of directly comparing thousands of bytes of actual data, the system compares compact fingerprint representations that serve as proxies for the original segments, reducing system resource consumption while preserving duplicate detection capability
Data Source
AI summary
A system provides affinity-based load-balancing distribution of similarity groups. A system generates service hash values by applying a consistent hash function to identifiers of deduplication object services. The system maps the deduplication object services to locations where the service hash values map on a hash ring. The system generates, for each of multiple similarity groups, a corresponding group hash value by applying the consistent hash function to a corresponding group identifier. The system assigns, each of the multiple similarity groups, to one of the deduplication object services which is mapped to one of the locations on the hash ring, which is a first successor to a location where the corresponding group hash value maps on the hash ring, thereby enabling each deduplication object service to deduplicate a balanced load of data file segments identified by assigned similarity groups.