Dynamic Data Object Routing via Timestamped Bucket Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data object storage systems face inefficiencies due to uneven usage of hash buckets, leading to storage space inefficiencies and the need for time-consuming data migrations when reallocating storage nodes.
Innovation Solution
A computer-implemented process that dynamically routes data objects using a unique identifier and timestamp, maintaining a routing table to track bucket associations with storage locations over time, allowing for reallocation without data migration and efficient storage management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If hash buckets are reallocated among storage nodes to balance storage space usage, then storage space efficiency is improved, but data migration time and cost increase
Solution Approach 1:
The system pre-allocates multiple hash buckets to each storage node before reallocation is needed. When a storage node becomes full, the system can immediately reassign hash buckets to other nodes without requiring data migration, because the target storage nodes have pre-established bucket slots ready to receive the data.
Solution Approach 2:
The system creates a mapping table that copies the relationship between hash buckets and storage nodes. When reallocation is needed, the mapping table is updated to reflect new bucket-node assignments, while the actual data remains in place. This virtual copying of the mapping relationship avoids physical data movement.
2Speed
If data objects are stored in hash buckets according to consistent hashing, then data retrieval efficiency is improved, but storage space utilization deteriorates due to uneven bucket usage
Solution Approach 1:
The system dynamically adjusts the mapping between hash buckets and storage nodes based on current storage capacity and usage patterns. Instead of a static consistent hashing mapping, the system continuously monitors storage node status and reassigns buckets to maintain both retrieval efficiency and balanced space utilization across all nodes.
Solution Approach 2:
The system changes the mapping parameter (which storage node a hash bucket corresponds to) based on storage capacity conditions. When storage nodes become full or new nodes are added, the mapping table is updated to reflect new parameter assignments, allowing the system to adapt to changing conditions while maintaining efficient data access.
3Adaptability or versatility
If storage nodes are added or removed from the system, then system flexibility is improved, but data movement and remapping overhead increase
Solution Approach 1:
When new storage nodes are added to the system, the administrator pre-configures hash buckets and assigns them to the new nodes before any data needs to be moved. This preliminary setup allows the system to immediately begin using the new storage capacity without requiring migration of existing data objects.
Solution Approach 2:
The system maintains a mapping table that copies the logical-to-physical relationship between hash buckets and storage nodes. When nodes are added or removed, only this mapping table needs to be updated, not the actual data distribution. This separates the complexity of remapping from data movement, reducing the operational overhead.
Data Source
AI summary
A computer-implemented process is disclosed for dynamically routing data objects to storage locations. The process includes assigning a unique identifier value and a timestamp value to an incoming data object. The process further includes determining a bucket for the data object based on the unique identifier value. The process further includes determining a storage location for the data object from among a plurality of storage locations, wherein the determination is based on the bucket and the timestamp value. The process further includes routing the data object to the storage location for storage therein.