Simulated Data Migration for Cloud Database Quality Assurance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Migrating data from a legacy system to a non-relational database, such as those used in cloud infrastructure, poses challenges in ensuring data quality due to differences in data structures and the absence of primary keys in target databases.
Innovation Solution
A method is proposed that involves simulating the data migration process, applying migration rules to source data to generate simulated data structured like the target data, merging simulated and actual target data, and using a machine learning algorithm, such as Minhash LSH, to identify discrepancies and ensure data quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is migrated from relational databases to non-relational databases (NoSQL) in cloud infrastructure, then data distribution and availability are improved, but data quality assurance becomes difficult due to absence of primary keys and different data structures
Solution Approach 1:
The patent applies preliminary action by simulating the migration process before actual data migration occurs. A simulation engine creates simulated target data from source data using the same migration rules that will be applied during actual migration. This allows validation and discrepancy detection to be performed in advance, ensuring data quality before the actual migration takes place, thus resolving the contradiction between improved data distribution and maintained data quality assurance.
2Adaptability or versatility
If migration rules are applied to transform source data to target data structure, then data transformation capability is improved, but data accuracy and consistency become harder to validate
Solution Approach 1:
The patent uses copying by creating a simulated copy of the target data through the simulation engine. This simulated target data is generated by applying the same migration rules to source data, creating an identical copy of what the actual migration will produce. By comparing this simulated copy with the actual migrated data, the system can validate accuracy and consistency without compromising the transformation capability, thus resolving the contradiction between adaptability and measurement precision.
3Measurement precision
If machine learning algorithms are used to identify discrepancies in migrated data, then data validation accuracy is improved, but processing time and complexity increase
Solution Approach 1:
The patent applies preliminary action by pre-processing and pre-comparing data structures before actual migration validation. The simulation engine pre-generates expected target data structures and pre-identifies potential discrepancies using machine learning algorithms. This preliminary validation reduces the time required during actual migration, as the heavy computational work of discrepancy detection is performed in advance, thus resolving the contradiction between validation accuracy and processing time.
Data Source
AI summary
Method for controlling data quality assurance after a migration between a source repository, which includes source data associated with a source data structure, and a target repository, which includes target data associated with a target data structure, according to migration rules. The target data is organized as a set of elements. The method includes simulating the migration by applying migration rules to source data so as to output simulated data according to the target data structure, where the simulated data is organized as a set of elements. The method includes merging the simulated data and the target data by excluding identical elements, to form an input vector for a machine learning algorithm. The method includes applying the machine learning algorithm for pairing elements of the input vector according to a similarity score and determining a discrepancies indicator according to the similarity score for each element of the input vector.

