Database Migration Top-Off System for Minimal Downtime
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data migration techniques are slow and often require shutdown of the source database, leading to unacceptable downtime, especially when dealing with large, time-varying data sets in multi-tenant data warehouse systems.
Innovation Solution
The system performs an initial long-duration migration, followed by validation to identify updated data, which is then copied to a temporary container and merged into the target database, with subsequent top-off migrations as needed to ensure complete data transfer while minimizing downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing data migration techniques are used, then data can be migrated between databases, but the migration process is slow and requires shutdown of the source database leading to unacceptable downtime
Solution Approach 1:
The patent segments the data migration process into distinct phases: an initial full migration of the data set, followed by separate top-off migrations for updated data. This segmentation allows the source database to remain operational during migration by handling different data subsets at different times, thereby reducing downtime while maintaining migration productivity.
Solution Approach 2:
The patent performs preliminary actions by completing the initial full migration before the source database needs to be shut down. This preliminary migration establishes a baseline data copy, allowing subsequent top-off migrations to occur during normal operations with minimal disruption, thus reducing overall downtime while maintaining efficient data transfer.
2Reliability
If existing data migration techniques are used, then data can be migrated, but the likelihood of migration failure increases with large data sets requiring restart
Solution Approach 1:
The patent divides large data set migration into smaller manageable segments: the initial migration handles the bulk data, while subsequent top-off migrations handle only the updated portions. This segmentation reduces the complexity and failure risk of each individual migration operation, improving overall reliability while minimizing the time lost to potential restarts.
Solution Approach 2:
The patent performs partial migrations by transferring only the necessary subsets of data (initial full set, then only updated portions) rather than attempting to migrate everything in one operation. This partial action approach reduces the risk of failure for each migration step, improving success rate while reducing total downtime.
3Stability of the object's composition
If source database is shut down during migration, then data consistency can be maintained, but unacceptable downtime results
Solution Approach 1:
The patent performs the main migration operation preliminarily before data consistency becomes critical. By completing the initial full migration while the source database is still operational, the system establishes a complete data copy that can serve as a baseline, allowing the source database to remain online and maintain consistency without unacceptable downtime.
Solution Approach 2:
The patent introduces an intermediary mechanism by implementing top-off migrations that capture and transfer only the updated data portions between the source and target databases. This intermediary approach allows the source database to remain operational and maintain data consistency while still achieving accurate data replication, avoiding the need for unacceptable downtime.
Data Source
AI summary
Systems and methods for the migration of large amounts of time-varying data from a source database to a target database are disclosed. After completion of an initial long-duration migration, a validation is performed to determine if any groups of data were updated in the source database during the initial migration. Updated data is then topped-off by migration to a temporary container before merging into the target database. Subsequent validation can confirm that migration is complete or initiate one or more additional top-off migrations.