Distributed Key-Value Store Rebalancing via Range Update
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed Key-Value Stores (KVS), the rebalancing process to adjust server loads without stopping the system is challenging, especially when data access switching from an existing partition to a new partition, leading to potential data consistency issues and system instability due to communication time-outs and network partitions.
Innovation Solution
A computer system with a node that processes data and a distribution unit that manages key ranges, transmitting range update information to adjust the data distribution between processes, ensuring seamless data processing and consistency across partitions even during network failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a rebalancing process is performed to adjust server loads in a distributed KVS, then the load balance between servers is improved, but data consistency issues and system instability occur during the transition from existing partition to new partition
Solution Approach 1:
The patent applies preliminary action by creating a new partition before the rebalancing process begins and preparing it in advance. The new partition is created with the same key range as the existing partition, and data copying is initiated beforehand. This allows the system to switch to the new partition smoothly without data consistency issues during the transition.
Solution Approach 2:
The patent uses an intermediary approach by introducing a new partition as a mediator between the old and final state. The new partition serves as a temporary intermediary that holds the data during the rebalancing process, allowing seamless switching without direct modification of the active partition, thus maintaining data consistency.
2Duration of action of stationary object
If the system continues to operate without stopping during rebalancing, then service availability is maintained, but communication time-outs and network partitions cause instability
Solution Approach 1:
The patent applies segmentation by dividing the rebalancing process into distinct phases: creating the new partition, copying data to the new partition, switching to the new partition, and finally deleting the old partition. This segmentation allows the system to maintain service availability in each phase while reducing instability by isolating potential failure points to specific segments rather than the entire process.
3Productivity
If data is copied from existing partition to new partition during rebalancing, then load distribution is improved, but data consistency issues arise during the copying process
Solution Approach 1:
The patent applies the copying principle by creating a complete copy of the existing partition's data to a new partition before switching. The new partition is populated with copied data while the old partition continues to serve requests, ensuring that the copying process does not compromise data consistency. Once copying is complete, the system atomically switches to the new partition.
Data Source
AI summary
A computer system includes a node including a plurality of processes, an instruction unit that designates a key range of data processed by the process, and a distribution unit. When a first key range is processed by a first process and a third key range, which is a portion of the first key range, is processed by a second process that processes a second key range, the instruction unit transmits a first updated key range to the first process, transmits a second updated key range to the second process, and transmits third range update information including first and second new ranges to the distribution unit. When receiving the third range update information, the distribution unit changes the distribution of the data to be processed. The first process performs data processing in the first updated key range and the second process performs data processing in the second updated key range.