Parallel Database Copying via Data Streaming Protocol
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing methods for creating secondary databases are manual, time-consuming, and inefficient, especially in cloud-based environments and big data applications, where data volumes are growing rapidly, necessitating a more efficient and automated process for database copying.
Innovation Solution
A computer-implemented method using a data streaming protocol to set up a secondary database identical to a primary database, prioritizing data objects for copying, copying data objects in parallel, restoring data in parallel, and applying changes made during copying, thereby creating a usable secondary database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual methods are used to create secondary databases, then data accuracy and completeness can be ensured, but the process is time-consuming and inefficient
Solution Approach 1:
The patent segments the database copying process into multiple independent data streams that can be copied in parallel. The primary database is divided into multiple data objects or partitions, each handled by separate streaming processes, enabling concurrent copying operations that significantly reduce total copying time while maintaining data integrity through individual stream management.
Solution Approach 2:
The patent performs preliminary actions by setting up the secondary database structure in advance before the actual data copying begins. Schema definitions, table structures, and database configurations are pre-configured on the secondary system, so that when data streaming starts, only data insertion is required, eliminating the need to recreate structures during the copying process and reducing overall time.
2Productivity
If parallel copying is implemented to speed up database creation, then productivity increases, but system complexity and resource requirements increase
Solution Approach 1:
The patent employs a universal data streaming protocol that handles multiple functions within a single framework. The same streaming infrastructure manages data extraction, transformation, loading, error handling, and progress monitoring across all parallel streams, reducing the need for separate complex systems for each copying operation and simplifying overall system architecture.
Solution Approach 2:
The patent implements feedback mechanisms that monitor the status of parallel copying operations in real-time. Progress indicators, error detection, and performance metrics are continuously tracked and fed back to the control system, enabling dynamic adjustment of streaming parameters and automatic error recovery without requiring complex manual intervention or oversight systems.
3Reliability
If data copying is performed online in the background, then operational continuity is maintained, but the copying process may interfere with production operations
Solution Approach 1:
The patent implements dynamic resource allocation and adaptive streaming rates that adjust based on system load and available resources. The copying process dynamically scales its resource consumption based on production system performance, ensuring that database creation operations adapt to changing operational conditions and minimize interference with production workloads while maintaining progress.
Solution Approach 2:
The patent employs periodic batching and interval-based data extraction where data is copied in controlled intervals rather than continuously at full speed. This periodic approach allows the system to pause and resume copying operations, reducing peak resource contention with production systems while ensuring continuous progress toward completing the database replication.
Data Source
AI summary
A method for generating a copy of a database using a data streaming protocol is disclosed. The method comprises setting up a structure of a secondary database, prioritizing data objects to be copied from a primary database to the secondary database, copying data objects from the primary database to the secondary database using a data streaming protocol, restoring data corresponding to the data object in the secondary database in parallel to copying data objects from the primary database to the secondary database, applying changes made to the data corresponding to the data object in the primary database during the time of copying, after the data object has been copied, and making available the secondary database for usage once all data objects have been copied from the primary database to the secondary database so that a usable copy of the primary database is created.