Parallel Data Migration Streams for Schema Transformation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data migration from one database to another often involves a linear, sequential process that is time-consuming and requires frequent interruptions to correct errors, disrupting the operation of datacenters and impacting user access.
Innovation Solution
A data migration system utilizing a distributed processing job, such as a Hadoop job, that transforms data from legacy databases to a consolidated database schema, performs validation and correction of data errors, and ensures continuous datacenter operation by routing traffic effectively during the migration process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a linear, sequential data migration process is used, then data can be migrated from one database to another, but the process is time-consuming and requires frequent interruptions to correct errors
Solution Approach 1:
The patent divides the data migration process into multiple parallel migration streams, where different subsets of data are migrated simultaneously through separate processing paths. This segmentation allows the system to process large volumes of data concurrently rather than sequentially, reducing overall migration time while maintaining data accuracy through distributed validation points across each stream
Solution Approach 2:
The patent implements preliminary data validation and error checking before the main migration process begins. By pre-identifying and correcting potential errors in the source data, the system reduces the need for interruptions during migration, allowing the process to continue smoothly and reducing total migration time
2Reliability
If frequent interruptions are made to correct errors during migration, then data accuracy can be maintained, but datacenter operation is disrupted and user access is impacted
Solution Approach 1:
The patent implements continuous feedback mechanisms throughout the parallel migration streams, where data validation and error detection occur automatically during the migration process rather than requiring interruptions. The system monitors migration progress and data integrity in real-time across all streams, providing immediate feedback that allows errors to be corrected without stopping the overall migration process, thus maintaining datacenter operation continuity
Solution Approach 2:
The patent ensures continuous datacenter operation by implementing migration processes that run in parallel with ongoing datacenter activities. The system allows read and write operations to continue on both source and target databases during migration, maintaining productivity and user access while data is being transferred and validated through multiple streams
3Ease of manufacture
If sampling is used to test data migration, then testing can be performed, but it relies on an acceptable level of error and assumptions about repeatability
Solution Approach 1:
The patent combines multiple validation approaches within the parallel migration streams, merging sampling-based testing with comprehensive checksum validation and data equivalence checking. By integrating these different validation methods across all migration streams, the system achieves high measurement precision in verifying migration accuracy while maintaining relative simplicity through automated processes that work consistently across all data subsets
Data Source
AI summary
A machine may be configured to migrate data among databases. For example, the machine determines a control signal value that indicates a migration state of a migration of member profile data associated with one or more members of a Social Networking Service (SNS) from one or more legacy databases to a consolidated database. The machine accesses member profile data at the one or more legacy databases, and causes a replication of the member profile data at the consolidated database. The causing of the replication includes transforming the member profile data from a schema associated with the one or more legacy databases to a schema associated with the consolidated database. The machine performs a data equivalency test. The machine, based on a result of the data equivalency test, modifies the control signal value to indicate a further migration state of the migration of the member profile data.