Distributed Hash Table Entry Clustering via Consistent Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face inefficiencies in managing and retrieving multiple versions of data objects, particularly in determining the latest version and clustering keymap information, which can be time-consuming and costly in large systems.
Innovation Solution
A distributed storage system that caches keymap information and uses a consistent hashing scheme to cluster related data objects, allowing for efficient retrieval of the latest version by maintaining a latest symbolic key entry and updating it based on PUT operations, and applies the hash function only to a portion of the key to optimize storage and retrieval operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple versions of data objects are stored in different storage blocks and on different computing nodes, then version control capability is improved, but the time and cost to determine the latest version increases
Solution Approach 1:
The patent introduces a keymap as an intermediary data structure that maps data object keys to their latest version locations. Instead of searching across all storage blocks and computing nodes to find the latest version, the system queries the keymap which directly provides the location information, significantly reducing the time and cost of version determination
Solution Approach 2:
The system performs preliminary actions by maintaining the keymap structure that pre-computes and stores the mapping between data object keys and their latest version locations. This preliminary organization of data allows for efficient lookup operations without requiring real-time searches across the distributed storage system
2Productivity
If keymap information is distributed across all computing nodes, then data distribution is improved, but retrieval efficiency deteriorates
Solution Approach 1:
The patent segments the keymap information and distributes it across different computing nodes in a structured manner. Each node maintains a portion of the keymap corresponding to specific key ranges or data types, allowing for efficient localized retrieval while maintaining overall system distribution
Solution Approach 2:
The system implements local quality by optimizing keymap storage and retrieval at each computing node. Each node is equipped with the specific keymap portions it needs to handle local data objects efficiently, reducing the need for cross-node communication during retrieval operations
Data Source
AI summary
A distributed storage system may store data object instances in persistent storage and may store keymap information for those data object instances in a distributed hash table on multiple computing nodes. Each data object instance may include a composite key containing a user key. The keymap information for each data object instance may map the user key to a locator and the locator to the data object instance. A request to store or retrieve keymap information for a data object instance may be routed to a particular computing node based on a consistent hashing scheme in which a hash function is applied to a portion of the composite key of the data object instance. Thus, related entries may be clustered on the same computing nodes. The portion of the key to which the hash function is applied may include a pre-determined number of bits or be identified using a delimiter.