Distributed Data Migration to Strong Consistency Without Locking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data migration methods from an eventually consistent system to a strongly consistent system require locking, which prevents data from being read or modified during migration, as transactional dual writes are not possible across these systems.
Innovation Solution
Implement bi-directional replication between the systems, where the first system operates as primary, asynchronously writes data to the second system, and switches the second system to primary once it is caught up, allowing it to serve read and write requests without locking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If locking is used during migration, then data consistency is maintained, but data availability and system functionality are lost
Solution Approach 1:
The patent applies preliminary action by pre-establishing bi-directional replication between the source and target systems before migration begins. This allows the target system to be pre-populated with data from the source system, enabling it to serve read requests during the migration process without requiring locks on the source system.
Solution Approach 2:
The patent uses an intermediary approach by introducing a replication mechanism that acts as a mediator between the source and target systems. This replication layer allows data to be copied asynchronously, enabling the target system to become available for serving requests without directly locking the source system.
2Reliability
If transactional dual writes are implemented, then data consistency during migration is improved, but system complexity increases due to incompatibility between eventually consistent and strongly consistent systems
Solution Approach 1:
The patent applies segmentation by dividing the migration process into distinct phases: initial bi-directional replication setup, asynchronous data copying, consistency verification, and final switchover. This segmentation allows each phase to be handled with appropriate consistency mechanisms rather than requiring complex transactional dual writes across the entire migration process.
Solution Approach 2:
The patent uses parameter changes by transitioning the target system from an eventually consistent state during replication to a strongly consistent state after migration. The system dynamically adjusts its consistency parameters based on the migration phase, allowing compatibility between different consistency models without requiring complex transactional dual writes.
3Ease of operation
If the second system serves as primary during migration, then data availability is improved, but data consistency may be compromised without bi-directional replication
Solution Approach 1:
The patent applies merging by combining bi-directional replication with the migration process. Both source and target systems continuously replicate data to each other, ensuring that when the target system serves as primary, it has up-to-date data from the source system, maintaining consistency while enabling availability.
Solution Approach 2:
The patent implements continuity of useful action by maintaining continuous bi-directional replication throughout the migration process. This ensures that data synchronization is an ongoing process rather than a one-time event, allowing the target system to reliably serve as primary once migration is complete while maintaining consistency with the source system.
Data Source
AI summary
Generally disclosed herein is an approach to migrate data from a first type of distributed system to a second type of distributed system without locking data, where transactional dual writes are not available across the two systems. The approach starts by setting up a bi-directional replication between the first system and the second system. The first system will initially operate as a primary system, where the primary system receives and serves write requests from clients or other devices. For each write to the first system, the second system is updated with an asynchronous write. When the second system is caught up to the first system, such that both the first and second systems reflect approximately the same data, the second system can be switched over to serve as the primary system. The second system can now directly receive and serve all future read and write requests.