Token Distribution in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems with replication, efficiently reallocating storage responsibility among nodes when some nodes go online or offline while ensuring performance and matching node capabilities is challenging, affecting storage and application node performance.
Innovation Solution
A method involving unique tokens for each partition, where data objects are stored in partitions matching their tokens, and tokens are calculated and assigned for primary and secondary responsibility levels, with adjustments made based on node capabilities and changes in the storage node set to maintain optimal distribution and dispersion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If storage responsibility is reallocated when storage nodes come online or go offline, then the system adapts to changes in storage node availability, but the performance of storage nodes and application nodes may be affected during the reallocation process
Solution Approach 1:
The system pre-calculates the optimal distribution of tokens before reallocation is needed. When a storage node joins or leaves, the pre-established calculation framework allows for rapid determination of new token assignments without performing complex real-time computations, thus minimizing performance impact during transitions
Solution Approach 2:
The patent implements dynamic token assignment where storage nodes can have primary and secondary responsibilities for different tokens. This dynamic structure allows the system to flexibly adjust responsibility distribution based on node availability while maintaining optimal performance by assigning secondary responsibilities to backup nodes that can quickly take over if primary nodes fail
2Productivity
If tokens are assigned to storage nodes for primary and secondary responsibility, then optimal token distribution is maintained, but the complexity of calculating and managing token assignments increases
Solution Approach 1:
The patent segments the token assignment into distinct primary and secondary responsibility levels. Each storage node maintains separate assignments for primary tokens and secondary tokens, allowing the system to manage complexity by treating different responsibility levels as separate manageable entities rather than a single complex assignment problem
Solution Approach 2:
The system introduces an intermediary data location table that stores and manages token-to-storage-node mappings. This intermediary structure simplifies the management complexity by providing a centralized repository that automatically tracks which storage nodes have primary or secondary responsibility for which tokens, eliminating the need for complex real-time coordination between nodes
Data Source
AI summary
A data location table master system generates a master data location table storing associations of tokens with storage nodes for varying responsibility levels. When the master data location table is updated, the data location table master system updates storage nodes affected by the update as well as other storage nodes and application nodes in the system. Then, the storage nodes and the application nodes store a copy of the master data location table. A token migration and synchronization process reallocates data object storage among the storage nodes based on the updated master data location table.