Consistent Hashing With Reserved Buckets For Low Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional consistent hash functions for distributing objects or processing tasks among resources face high memory and computational complexity, as well as a high number of keys that need to be reshuffled when the number of buckets changes, such as when servers are added or removed.
Innovation Solution
A method using a consistent hash function that starts with an initial 'anchor' distribution, allowing for efficient addition and removal of buckets by iteratively recalculating the mapping, reducing computational complexity and memory usage, and allowing for arbitrary changes to the bucket set with minimal disruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional consistent hash functions are used to distribute keys among buckets, then key distribution consistency is improved, but memory use and computational complexity increase
Solution Approach 1:
The patent segments the bucket set into two distinct parts: working buckets (currently active resources) and reserved buckets (potential future resources). This segmentation allows the hash function to operate differently on each segment, reducing overall complexity while maintaining consistency. The working bucket set is used for immediate key assignment, while reserved buckets are held in reserve for future expansion without requiring full recomputation.
Solution Approach 2:
The patent performs preliminary action by pre-allocating and reserving bucket slots before they are actually needed. Reserved buckets are set aside in advance during system initialization or expansion planning, so when new working buckets are needed, the system can simply activate pre-reserved slots rather than performing complex reallocation. This eliminates the need for reshuffling keys during normal operations.
2Reliability
If conventional consistent hash functions are used, then key distribution consistency is improved, but the number of keys that must be reshuffled when buckets change increases
Solution Approach 1:
By segmenting buckets into working and reserved categories, the patent ensures that keys only map to working buckets. When buckets are added or removed, only the working bucket set changes, and the hash function is designed to minimize rehashing by leveraging the segmented structure. Reserved buckets act as a buffer that absorbs expansion needs without triggering key remapping.
Solution Approach 2:
Reserved buckets are pre-configured and held in reserve before being activated as working buckets. This preliminary preparation means that when system expansion occurs, new working buckets are already prepared and ready to accept keys immediately, eliminating the need to reshuffle existing keys during the transition.
3Adaptability or versatility
If buckets are dynamically added or removed, then system adaptability is improved, but key distribution consistency deteriorates with conventional hashing
Solution Approach 1:
The patent implements dynamics by allowing the working bucket set to change size and composition while maintaining a stable key-to-bucket mapping mechanism. The hash function dynamically adapts to the current working bucket set without requiring full redistribution of keys. Reserved buckets provide a dynamic buffer that can be converted to working buckets as needed, enabling flexible system scaling.
Solution Approach 2:
Reserved buckets serve as an intermediary between the static hash function and the dynamic working bucket set. When working buckets change, the reserved bucket set acts as a buffer zone that absorbs the changes without propagating disruption to key assignments. This intermediary layer protects key distribution consistency during dynamic transitions.
Data Source
AI summary
Embodiments of the present systems and methods may provide a consistent hash function that provides reduced memory use and complexity, reduced computational complexity, and relatively low numbers of keys that must be reshuffled compared to current techniques. For example, in an embodiment, a computer-implemented method for controlling computing resources may comprise storing a set of labels of potential resources comprising a plurality of labels of working resources allocated to actual resources and a plurality of labels of reserved resources available to be allocated, generating an initial assignment to one of the set of labels of potential resources, when the assignment to one of a set of labels of potential resources is to one of the labels of reserved resources, reassigning the request to another label of a resource selected from a subset of the labels of potential resources, and repeating the reassigning until the request is assigned to a label of a working resource.