Online and Background Data Migration for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Migrating large amounts of data from relational databases to distributed storage systems often requires significant downtime, which is undesirable for systems that need continuous access.
Innovation Solution
Implementing a data migration process that includes an online migration mode for actively used data and a background migration mode for less frequently used data, using a store lookup and partition table to manage data relocation while maintaining client access, and transforming the relational schema to a key-value schema for efficient storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is migrated from relational database to distributed storage system, then storage capacity and scalability are improved, but system downtime increases
Solution Approach 1:
The patent segments the data migration process into two distinct modes: online migration mode for actively used data and background migration mode for less frequently used data. This segmentation allows different migration strategies to be applied to different data subsets, enabling continuous system operation while migrating data from relational database to distributed storage system without requiring complete system downtime.
Solution Approach 2:
The patent implements preliminary actions by setting up dual-write capability before migration begins, where writes are simultaneously performed to both relational database and distributed storage system. This preliminary dual-write setup ensures data consistency is maintained before the actual migration switches over, allowing seamless transition without data loss or system interruption.
2Ease of operation
If online migration mode is used for actively used data, then data accessibility is improved, but migration complexity increases
Solution Approach 1:
The patent introduces an intermediary migration management system that handles the complexity of coordinating online and background migration modes. This intermediary layer manages the store lookup and partition table updates, coordinates dual-write operations, and handles the switching between migration modes, thereby isolating the complexity from the data access operations and maintaining ease of data accessibility.
Solution Approach 2:
The patent implements dynamic migration mode selection where the system automatically switches between online migration mode for actively used data and background migration mode for less frequently used data. This dynamic approach allows the migration process to adapt to changing data access patterns, maintaining high accessibility for hot data while efficiently migrating cold data in the background.
3Productivity
If relational schema is transformed to key-value schema, then storage efficiency is improved, but data transformation complexity increases
Solution Approach 1:
The patent segments the schema transformation process into structured steps: identifying partition keys, defining data model transformations, and validating destination partitions. This segmentation breaks down the complex relational-to-key-value schema transformation into manageable components, making the process more controllable and less error-prone while achieving efficient distributed storage.
Solution Approach 2:
The patent performs preliminary validation of the destination partition using write capture and replay mechanisms before completing the schema transformation. This preliminary action ensures that the key-value schema transformation is correct and that data integrity is maintained, reducing the need for corrective operations later and simplifying the overall transformation process.
Data Source
AI summary
A mechanism for migrating data from a database management system to a remote data store may maintain a row-level mapping of keys to locations, where the location comprises the original location or one of a plurality of destination partitions. Requests to read and write data may be redirected during migration and processed by storing the requested data in one of the destination partitions and updating the row-level mapping. Additional data may be migrated by a background process.