Concurrent Key-Value Store Migration Using Range Progress Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Migrating between key-value stores in object storage systems results in increased system resource usage and latency due to the overhead of querying both stores during concurrent operations, which is inefficient and affects the ability to serve requests without downtime.
Innovation Solution
Implement a multi-threaded migration process that divides the key address space into continuous, non-overlapping ranges, utilizing parallel processes to manage migration status through a navigable map of buckets and progress keys, allowing efficient determination of migration status and reducing the need to query both key-value stores for each request.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a migration is performed between key-value stores, then data can be transferred from the first key-value store to the second key-value store, but system resource usage increases and latency increases due to querying both stores during concurrent operations
Solution Approach 1:
The patent divides the key address space into multiple continuous, non-overlapping ranges and assigns each range to a separate process. This segmentation allows the system to track migration status for different key ranges independently, eliminating the need to query both key-value stores for all keys and reducing redundant operations during concurrent access.
Solution Approach 2:
The patent introduces a navigable map data structure as an intermediary between the two key-value stores. This map stores progress keys that indicate which keys have been migrated, allowing the system to determine whether to access the first or second store without querying both stores for every request, thereby reducing resource usage and latency.
2Reliability
If both key-value stores are queried during concurrent operations, then data consistency can be maintained, but latency increases due to the overhead of querying both stores
Solution Approach 1:
The patent performs preliminary actions by maintaining progress keys in the navigable map that indicate the migration status of different key ranges before requests are processed. When a request arrives, the system can immediately determine which store to access based on pre-computed progress information, eliminating the need for runtime queries to both stores and reducing latency while maintaining consistency.
3Measurement precision
If migration status is tracked for each key, then accurate migration determination can be made, but the complexity of managing migration state increases
Solution Approach 1:
The patent segments the key address space into ranges and maintains one progress key per range in the navigable map, rather than tracking each individual key. This approach maintains accurate migration status information while significantly reducing the complexity of state management compared to per-key tracking.
Data Source
AI summary
The system can divide a key address space of a first key-value store into a group of ranges. The system can initialize a group of processes to perform a migration in parallel, wherein respective processes maintain respective progress keys. The system can maintain a navigable map comprising pairs, wherein the pairs comprise respective starting keys of respective ranges of respective buckets, and respective buckets, and wherein the respective buckets comprise respective indications of the respective ranges and respective progress keys of the progress keys. The system can, in response to receiving a request to access a first value based on a first key, identify a first bucket from the navigable map and the first key, and in response to determining that a progress key of the progress keys of the first bucket is greater than or equal to the first key, access the first value in the second key-value store.