Soft Locking Data Migration with Shadow Journals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large data store systems face challenges in migrating data while maintaining access for client applications, especially during decommissioning or expansion, as existing methods often require shutting down access or risk data corruption.
Innovation Solution
A data migration system that uses soft locking mechanisms and shadow journals to migrate data entries from a source memory device to a target device while allowing client access, employing concurrent threads to remap aliases and commit data in an atomic transaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data migration methods are used, then data can be migrated from source to target memory device, but client access must be shut down and data corruption risk increases
Solution Approach 1:
The system creates a shadow journal (preliminary copy structure) in the target memory device before actual data migration begins. This preliminary structure allows the migration process to proceed without interrupting client access to the original data in the source memory device, thereby maintaining availability while enabling safe data transfer.
Solution Approach 2:
The shadow journal acts as an intermediary structure between the source and target memory devices during migration. Client applications continue to access data through the source memory device while the shadow journal receives updated data in the background, eliminating the need to shut down client access and reducing data corruption risk.
2Productivity
If soft locking mechanism is used, then client access can continue during migration, but data corruption risk may increase
Solution Approach 1:
The system implements a feedback mechanism where the migration process continuously monitors client access patterns and adjusts its operation accordingly. When clients access data, the system receives feedback and coordinates updates through the shadow journal to maintain consistency, allowing fast migration while preventing data corruption through real-time coordination.
Solution Approach 2:
The shadow journal provides a dynamic buffer zone that adapts to changing client access patterns. The system can dynamically adjust the migration pace and coordination mechanism based on real-time conditions, enabling fast migration during low-access periods while maintaining data consistency during high-access periods.
3Productivity
If multiple alias redirection threads are launched, then migration can be completed faster, but system complexity increases
Solution Approach 1:
The migration process is segmented into multiple independent alias redirection threads, each handling specific data migration tasks. This segmentation enables parallel execution and higher throughput while the shadow journal provides a simple coordination mechanism that reduces the overhead of managing multiple threads, balancing productivity and complexity.
Data Source
AI summary
A method of migrating data entries stored in a distributed data store from a source memory device of the data store to a target memory device of the data store. The method comprises locking a data entry in the source memory with a soft locking mechanism by an application executing on a computer system, reading the data entry in the source memory by the application, cloning the data entry in a shadow journal in the target memory by the application, launching a plurality of alias redirection threads by the application, where each remaps one of a plurality of aliases of the data entry from referencing the location of the data entry in the source memory to referencing the location of the data entry in the target memory, and committing the data entry in the shadow journal to the target memory by the application.