Consistent Hashing Bucket Mapping for Distributed Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed computing systems face challenges in managing memory footprint and achieving uniform distribution of items across nodes during resizing, as existing consistent hashing methods require large memory and can lead to non-uniform mapping and difficulties in identifying item locations without rehashing.
Innovation Solution
A system that uses consistent hashing to generate a numeric hash value, downscales it into a smaller bucket space, and maps it deterministically to nodes, allowing for uniform distribution and efficient range queries to relocate items during resizing without scanning the entire data corpus or recomputing hash values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If consistent hashing is used to map items to physical resources in a distributed computing system, then uniform distribution of items across nodes is achieved, but large memory footprint is required to maintain the mapping structure
Solution Approach 1:
The patent segments the mapping structure into multiple independent hash tables instead of using a single large mapping structure. Each hash table stores a portion of the item-to-node mappings, allowing the system to maintain uniform distribution while reducing the memory footprint of any single data structure. The system can selectively load and unload hash tables based on current operational needs.
Solution Approach 2:
The patent introduces an additional dimension to the mapping approach by using a hierarchical structure with multiple levels of hashing. Instead of a direct one-to-one mapping, the system uses multi-dimensional hash tables that map items through intermediate buckets before reaching final node assignments. This dimensional transformation allows efficient memory utilization while preserving uniform distribution properties.
2Ease of operation
If traditional consistent hashing is used for item mapping, then item location can be identified, but rehashing is required when system resizing occurs
Solution Approach 1:
The patent performs preliminary organization of items into buckets based on hash values before system resizing occurs. When resizing is needed, the system can efficiently relocate items by working with pre-organized buckets rather than rehashing individual items. This preliminary structuring allows the system to adapt to size changes while minimizing rehashing operations.
Solution Approach 2:
The patent implements a dynamic mapping system where the hash table structure and bucket assignments can be flexibly adjusted during runtime. When the system resizes, the mapping structure dynamically reconfigures by redistributing buckets across the new number of nodes without requiring complete rehashing of all items. This dynamic adaptation reduces the time loss associated with resizing operations.
3Adaptability or versatility
If the number of physical resources increases in a distributed system, then system capacity is improved, but the number of items requiring relocation during resizing increases
Solution Approach 1:
The patent segments the item space into multiple buckets that can be independently managed and relocated. When physical resources increase, only the necessary number of buckets need to be redistributed to new nodes rather than relocating all items. This segmentation allows the system to scale capacity while minimizing the quantity of items requiring relocation during resizing operations.
Solution Approach 2:
The patent changes the parameter of bucket-to-node mapping ratios dynamically based on system capacity. As the number of physical resources increases, the system adjusts the distribution parameters to maintain optimal load balancing. This parameter adaptation allows the system to accommodate increased capacity while reducing the number of items that need to be relocated during resizing events.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A string, identifying an item to be assigned to a physical resource, is hashed to obtain a numeric hash value. The numeric hash value is downscaled to obtain a bucket identifier that identifies a bucket that will hold the numeric hash value. The bucket is then deterministically mapped to a physical resource so that it can be retrieved without accessing a stored data structure representative of the mapping.