Seamless Data Migration via Slicing and Redo
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data migration processes in User Generated Content services, such as microblogging, require disabling writes to the original system, which affects user requests and prolongs the migration time, leading to user awareness and increased maintenance risks.
Innovation Solution
A method and system for seamless data migration that slices data into manageable portions, imports these slices into a new system, and reprocesses pre-recorded write requests in sequence, allowing simultaneous writing to both systems without disabling writes to the original system, using a redo technique to ensure data consistency and reduce migration risks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data migration is performed by disabling writes to the original system, then data consistency can be ensured during migration, but user write requests are affected and service availability deteriorates
Solution Approach 1:
The patent divides the data migration process into distinct phases: initial data import phase and subsequent redo phase. During the initial import, writes are disabled to ensure consistency. After import completes, the system enters a redo phase where pre-recorded write requests are reprocessed in sequence to update the new system without affecting the original system's write operations, thus segmenting the impact on service availability from data consistency requirements
Solution Approach 2:
The patent records write requests in advance before they are executed in the new system. By pre-recording the sequence and content of write requests during the initial data import phase, the system can later replay these recorded requests in the new system without disabling writes to the original system, thereby maintaining service availability while ensuring data consistency through predetermined operation sequences
2Manufacturing precision
If data migration takes more time, then complete data import is achieved, but writes must be disabled for longer periods affecting user requests
Solution Approach 1:
The patent records write requests in advance during the initial data import phase. This preliminary recording allows the system to import data completely without executing writes in real-time, and then replay the recorded writes later in the new system without disabling the original system's write operations, thus achieving complete data import while minimizing write disable duration
Solution Approach 2:
The patent segments the write operation timeline into two independent phases: initial data import with disabled writes for completeness, and subsequent redo phase with enabled writes for service availability. This segmentation allows the system to achieve complete data import in the first phase while restoring write capabilities in the second phase, thereby reducing the total duration writes remain disabled
3Reliability
If conventional data migration is used, then data can be migrated from original system, but operations and maintenances become complex and risks increase
Solution Approach 1:
The patent creates a copy of the original system's data and write request logs in the new system. By copying data in bulk during the initial import phase and then replaying copied write requests in sequence during the redo phase, the system simplifies the migration operation to two straightforward steps rather than complex real-time synchronization mechanisms, reducing operational complexity while maintaining migration capability
Solution Approach 2:
The patent introduces write request logs as an intermediary mechanism between the original system and the new system. These logs mediate the data migration process by capturing write requests in advance and enabling their replay in the new system without requiring complex real-time coordination or disabling writes to the original system, thus simplifying operations and reducing maintenance risks
Data Source
AI summary
A method and a system for data migration are described, the method for data migration including: slicing data in an original system to obtain a data slice; importing the data slice into a new system; and redoing, in the new system, respective data corresponding to at least one pre-recorded write request according to a sequence number order of the at least one pre-recorded write request, wherein the at least one pre-recorded write request above is to simultaneously write to both the original system and the new system. In the method and the system, data in an original system can be seamlessly imported to a new system without forbidding writes to the original system, and the data in the new system can be recovered to the latest by a redo technique. Without any awareness of users, the data migration can be achieved in the background.