Record-by-record live migration using lock store
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data migration approaches require outages, making entire data sets inaccessible to clients during the process, which affects availability and performance.
Innovation Solution
Implementing a record-by-record live migration system using a lock store that allows for conditional writes and manages migration through a data migrator and migration router, enabling continuous access during data set migration by flipping authority on a per-record basis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data migration approaches are used, then data migration can be completed, but the entire data set becomes inaccessible during the migration process
Solution Approach 1:
The patent segments the data set into individual records and migrates them separately using a record-by-record approach. Each record is locked, migrated individually, and unlocked, allowing other records to remain accessible. This segmentation resolves the contradiction by enabling continuous data accessibility while maintaining migration productivity through parallel processing of multiple records.
Solution Approach 2:
The patent implements dynamic migration by allowing the migration process to adapt its pace and behavior based on system conditions. The migration router dynamically routes requests between source and destination data stores, and the lock store dynamically manages record availability during migration. This dynamic approach maintains data accessibility while optimizing migration speed based on real-time system state.
2Stability of the object's composition
If the entire data set is locked for migration, then data consistency is maintained, but client applications cannot access the data
Solution Approach 1:
The patent applies segmentation by locking only individual records that are being migrated rather than the entire data set. The lock store manages fine-grained locks at the record level, allowing client applications to access and modify records that are not currently being migrated. This maintains data consistency for migrated records while preserving client access to the majority of the data set.
Solution Approach 2:
The lock store acts as an intermediary between the migration process and client applications. It manages the locking and unlocking of individual records, coordinating between the data migrator and migration router to ensure consistency without blocking client access. This intermediary mechanism resolves the contradiction by mediating between data consistency requirements and client accessibility.
3Reliability
If migration is performed on a per-record basis with locking, then continuous access is enabled, but system complexity increases due to lock management
Solution Approach 1:
The patent merges the lock management functionality into a centralized lock store that handles locking for multiple records and operations. The lock store combines locking, unlocking, and coordination functions in a single component, reducing overall system complexity despite the fine-grained record-by-record approach. This merging enables continuous data availability while managing lock complexity through consolidation.
Solution Approach 2:
The lock store is designed as a universal component that handles multiple functions: acquiring locks on individual records, releasing locks after migration, coordinating between migrators and routers, and managing concurrent access. This multi-functional design reduces the need for separate mechanisms for each operation, thereby reducing overall system complexity while maintaining high data availability.
Data Source
AI summary
Methods, systems, and computer-readable media for record-by-record live migration using a lock store are disclosed. Migration of a data set comprises acquiring a migration lock from a lock store for an individual record being migrated. An instance of the record is then stored in the destination, and the record is deleted from the source. Subsequently, the migration lock is deleted from the lock store or otherwise released. During the migration, a rate at which insert requests are directed to the destination is dialed up. During the process of updating an insert treatment configuration across a distributed set of migration router instances, an insert lock is acquired from the lock store for each record being inserted. Also metadata is stored indicating a determined location for the record being inserted. Any subsequent requests to insert a record with a same key are inserted at the indicated determined location.