Distributed Index Splitting Without Locks for Consistent Replicas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed databases, splitting multiple copies of an index can lead to race conditions and inconsistent splits, which existing methods address by locking the database, causing scalability issues and increased latency.
Innovation Solution
A two-stage index splitting process in distributed databases, where nodes split the index atom into a source and target atom, followed by forwarding messages asynchronously, maintaining consistency and concurrency without locking the database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locks are applied to individual pages or records during index splitting, then data consistency is maintained, but database scalability deteriorates and latency increases
Solution Approach 1:
The index splitting process is divided into two distinct stages: a preparation stage where the chairman node divides the index atom into source and target atoms, and an execution stage where split instructions are propagated to replica nodes. This segmentation allows concurrent data insertion during the preparation phase without requiring locks, while maintaining consistency through coordinated execution.
Solution Approach 2:
The chairman node performs preliminary actions by first dividing the index atom into source and target atoms, then preparing and broadcasting split instructions to all replica nodes before actual splitting occurs. This preliminary coordination enables replica nodes to split their copies in sync without requiring locks during the actual data insertion operations.
2Reliability
If locks are applied during index splitting, then data integrity is ensured, but operation latency increases
Solution Approach 1:
The index splitting process is divided into two distinct stages: a preparation stage where the chairman node divides the index atom into source and target atoms, and an execution stage where split instructions are propagated to replica nodes. This segmentation allows concurrent data insertion during the preparation phase without requiring locks, while maintaining consistency through coordinated execution.
Solution Approach 2:
Data insertion operations can continue uninterrupted during the index splitting process. The chairman node prepares the split and propagates instructions asynchronously, allowing other nodes to continue processing data insertion requests without waiting for the split to complete, thus maintaining continuous useful action.
3Productivity
If multiple copies of the index are split simultaneously, then scalability is improved, but race conditions occur leading to inconsistent splits
Solution Approach 1:
The chairman node receives feedback from replica nodes through acknowledgement messages confirming receipt and execution of split instructions. This feedback mechanism ensures that all replica nodes have successfully split their index copies before the chairman proceeds, maintaining consistency across all copies without requiring locks.
Solution Approach 2:
The chairman node acts as an intermediary that coordinates the splitting process across all replica nodes. It broadcasts split instructions to all replicas and waits for acknowledgements, ensuring that all copies are split consistently without requiring individual locks on each replica node.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
In a distributed database, many nodes can store copies, or instances, of the same record. If the record is split on one node, it should be split on the other nodes to maintain consistency, concurrency, and correctness of the data in the distributed database. In some distributed databases, the records are locked during the update process to ensure data integrity. Unfortunately, locking the records can increase latency, especially for larger databases. But if the records aren't locked and a node fails as a record is being split and updated simultaneously, the split and update may not propagate throughout the distributed database, leading to a loss of data integrity. Exchanging messages about the status of record splitting and forwarding updates internally reduces the likelihood of a loss of data integrity due to a node failure.