Retention Time Hash Ring for Storage Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed file systems, the deletion of files with the same retention time can lead to unbalanced storage systems, as these files are often assigned to the same node, causing an imbalance in the consistent hash ring.
Innovation Solution
A retention time-based consistent hash ring process is implemented, where files with adjacent expiration dates are assigned to different virtual nodes, which are then distributed across multiple physical nodes, ensuring even distribution by using tagged virtual nodes and a consistent hash ring mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If files with the same retention time are distributed using traditional consistent hashing, then the hash mapping is simple and straightforward, but the storage system becomes unbalanced when files are deleted, as multiple files may be assigned to the same node
Solution Approach 1:
The patent segments the storage system by introducing virtual nodes as an intermediate layer between physical nodes and files. Each physical node is divided into multiple virtual nodes, allowing files with the same retention time to be distributed across different virtual nodes (and thus different physical nodes) while maintaining the simplicity of consistent hashing. This segmentation resolves the contradiction by preventing storage imbalance without complicating the hash mapping mechanism.
Solution Approach 2:
The patent adds a new dimension to the consistent hash ring by introducing virtual nodes as an intermediate mapping layer. Instead of directly mapping files to physical nodes, the system creates a two-dimensional mapping: files → virtual nodes → physical nodes. This dimensional expansion allows the system to maintain balance during file deletions while preserving the simplicity of the original hash function.
2Stability of the object's composition
If virtual nodes are introduced to distribute files evenly across physical nodes, then storage balance is improved, but the system complexity increases
Solution Approach 1:
The virtual nodes automatically self-organize and self-distribute across the hash ring without requiring complex external management. Each virtual node independently computes its position based on its identifier and the consistent hash function, and files automatically route to the appropriate virtual node. This self-service mechanism achieves even distribution while minimizing system complexity.
Solution Approach 2:
The patent changes the parameter being hashed from directly physical node identifiers to virtual node identifiers. By transforming the input parameter of the hash function, the system achieves better distribution characteristics while keeping the hash function itself unchanged. This parameter transformation simplifies the overall system architecture despite introducing virtual nodes.
3Stability of the object's composition
If files are deleted from a single node to maintain balance, then storage distribution improves, but the deletion operation becomes more complex and time-consuming
Solution Approach 1:
The patent performs preliminary distribution actions by pre-distributing files across multiple virtual nodes before deletion occurs. When a file needs to be deleted, the system only needs to remove it from its specific virtual node rather than redistributing multiple files across physical nodes. This preliminary organization significantly reduces the time and complexity of deletion operations while maintaining storage balance.
Solution Approach 2:
The patent extracts the deletion operation from the physical node level and relocates it to the virtual node level. By isolating the deletion operation to specific virtual nodes rather than affecting entire physical nodes, the system minimizes the scope and time of deletion operations. Files are taken out from their specific virtual nodes independently, avoiding complex redistribution across physical nodes.
Data Source
AI summary
A retention-based consistent hash ring process defines each file name in the system to include its expiration date (or time) as a prefix or suffix that is stored and indexed as metadata. The process uses a virtual node to represent adjacent expiration days to create virtual nodes based on individual days of the week. Each physical node contains the same number of labeled virtual nodes, and the consistent hash ring process is used to move files with the same expiration day to different physical nodes by looking for next labeled virtual nodes on the hash ring. This provides a way to locate the virtual node storage location by specifying a file's expiration date as part of the key used in the hash ring process, and distributes files that may otherwise be assigned to the same physical node through a backup policy.