Data Movement System Using Intermediary Files for Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data migration tools, particularly ETL tools, face challenges such as high resource consumption, limitations in handling large data sets, inflexibility, and the need for significant resources and training, which results in high costs and inefficiencies in data center operations.
Innovation Solution
A data movement system utilizing a data-to-file-to-data movement path, where data is moved from a source database to an intermediary file and then to a target database, leveraging a configuration-driven approach with a data movement server that interacts with a configuration database to manage the process, reducing resource requirements and increasing processing speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional ETL tools are used for data migration, then data can be moved from source to target database, but system resource consumption increases significantly
Solution Approach 1:
The patent segments the data migration process into distinct phases: extraction phase where data is retrieved from source database and written to intermediary files, and loading phase where data is read from files and inserted into target database. This segmentation allows database systems to be accessed only when necessary (during extraction and loading), rather than maintaining continuous connections, thereby reducing system resource consumption while maintaining data migration productivity
Solution Approach 2:
The patent introduces an intermediary file system as a mediator between source and target databases. Data flows from source database to intermediary files, then from files to target database. This intermediary approach eliminates the need for continuous database-to-database connections, reducing the burden on database systems and lowering overall system resource consumption while preserving full data migration capability
2Productivity
If traditional ETL tools are used for data migration, then data can be migrated, but processing speed is limited
Solution Approach 1:
The patent performs preliminary actions by extracting data from the source database and writing it to intermediary files before the actual loading phase. This preliminary extraction allows the data to be staged in a format ready for efficient bulk loading into the target database, eliminating the need for repeated database connections and queries during the migration process, thereby increasing processing speed and reducing overall migration time
Solution Approach 2:
The patent maintains continuity of useful action by establishing a streamlined workflow where data flows continuously from extraction to file writing to loading. The intermediary files serve as a persistent staging area that allows the migration process to proceed without interruption or repeated database access, maintaining continuous productive action and significantly reducing the time required for data migration
3Adaptability or versatility
If traditional ETL tools are used for data migration, then data can be moved between databases, but flexibility is reduced
Solution Approach 1:
The patent creates a universal data migration framework that can handle multiple data sources, multiple target databases, and various data formats through a common intermediary file approach. The system can extract data from different source databases, convert it to standardized intermediary files, and load it into various target database systems, providing high flexibility and adaptability without requiring separate complex tools for each migration scenario
4Productivity
If traditional ETL tools are used for data migration, then data can be migrated, but significant training and human resources are required
Solution Approach 1:
The patent implements self-service capabilities through automated configuration and execution of data migration processes. The system can automatically generate extraction and loading scripts, manage intermediary files, and execute migration tasks with minimal human intervention. This automation reduces the need for specialized training and human resources while maintaining robust data migration capability, making the process easier to operate
Data Source
AI summary
A data movement system is provided for moving data using a data-to-file-to-data movement path. The data movement system includes a source database, a target database, a configuration database, and a data movement server. The data movement is in communication with the source database, the target database, and the configuration database. The processor is configured to receive a configuration record including source details and target details. The processor is also configured to define an extraction query based on the source details and to apply the extraction query to the source database to obtain an extraction load. The processor is further configured to generate a load file based on the extraction load, to define a load script based on the target details, and to apply the load script to the load file to obtain a load query. The processor is also configured to update the target database with the load query.