Data Mover Using RAM Buckets for Fast Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data replication across multiple databases, particularly for data science and analytics teams, is complex and requires programming skills, leading to inefficiencies and increased costs due to reliance on IT support, and existing batch read-write processes are slow and inefficient.
Innovation Solution
A web application configured to run on a cluster of virtual machines that allows users to schedule and design ETL workflows without coding, enabling data replication between relational and NoSQL databases with optimized batch read-writes using a direct push method and RAM memory buckets, ensuring maximum speed and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data replication is performed using traditional batch read-write methods, then data transfer can be completed, but the process is slow and inefficient due to database read/write speed restrictions
Solution Approach 1:
The patent introduces a temporary table as an intermediary structure between source and target databases. Data is first loaded into the temporary table from the source database, then transferred to the target database. This intermediary approach allows optimization of the transfer process and avoids direct slow batch operations between the main databases.
Solution Approach 2:
The patent changes the operational parameters by using in-memory processing and optimized SQL operations for the temporary table. By altering how data is stored and processed during the replication process (using temporary tables with optimized schemas), the system achieves faster data transfer speeds while maintaining data integrity.
2Ease of operation
If data replication is performed using traditional methods, then data can be copied between databases, but the process requires programming skills and IT support, increasing complexity and cost
Solution Approach 1:
The patent implements a self-service data replication system where users can initiate and manage data replication tasks through a simplified interface without requiring programming knowledge. The system automatically handles the complex operations of creating temporary tables, executing data transfer, and managing resources, allowing end-users to perform previously IT-dependent tasks independently.
Solution Approach 2:
The patent creates a universal data replication platform that can work with multiple database types and scenarios through a single interface. The system provides multi-functional capabilities including scheduled replication, on-demand replication, and automatic resource management, eliminating the need for different tools or programming expertise for different data replication scenarios.
3Productivity
If temporary files are used to address batch read-write restrictions, then data transfer can proceed, but the process becomes slow and inefficient
Solution Approach 1:
Instead of using slow temporary files on disk, the patent uses temporary tables in the database as an intermediary structure. This allows data to be held and processed in-memory during the replication process, dramatically improving access speed and reducing the time required for batch operations while maintaining the benefits of an intermediary approach.
Solution Approach 2:
The patent replaces the mechanical file I/O system with a database-based in-memory processing system. By substituting disk-based temporary files with database temporary tables that can operate in memory, the system eliminates the bottleneck of mechanical disk operations and achieves significantly faster batch read-write performance.
Data Source
AI summary
Systems and methods for data replication. A data processing service can present a user interface configured to receive a data fusion algorithm from a user, and a request to execute a job, each job including instructions to copy data from one of a plurality of source databases to one of a plurality of target databases, and can present a plurality of virtual machines, each virtual machine comprising: a processing unit configured to attempt to start the job requested by the user, when the job can be started, lock the job from the rest of the plurality of virtual machines, and execute the job according to the data fusion algorithm using a random access memory (RAM) memory bucket.