Data Replication Staging for Memory-Constrained Batch Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently replicating large volumes of data across different systems without overwhelming memory resources, leading to system failures and downtime, particularly in scenarios involving shared data management systems and local database systems.
Innovation Solution
Implementing a staging area for data replication that processes data in batches, identifying and prioritizing the most recent records based on unique identifiers, and managing memory consumption by reading and processing only relevant data portions concurrently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is replicated in large volumes across different systems, then data completeness and functionality are improved, but memory resources are overwhelmed leading to system failures
Solution Approach 1:
The patent divides the data replication process into discrete batches, where each batch contains a manageable subset of data records. The system processes and replicates one batch at a time, then discards it, rather than loading all data into memory simultaneously. This segmentation allows large volumes of data to be replicated without overwhelming system memory resources.
Solution Approach 2:
The patent implements preliminary filtering and identification of data batches before replication. The system identifies which data batches need to be replicated, prepares them in advance, and then processes them sequentially. This preliminary action ensures that only necessary data is loaded into memory at any given time, preventing memory overload while maintaining data completeness.
2Quantity of substance
If all data records are processed simultaneously during replication, then data completeness is improved, but processing time and system downtime increase
Solution Approach 1:
The patent implements periodic batch processing where data replication occurs in repeated cycles. Each cycle processes a specific batch of data records, replicates them to the target system, and then moves to the next batch. This periodic action allows the system to maintain operational continuity by processing data in intervals rather than requiring complete system downtime for bulk replication.
Solution Approach 2:
The patent enables continuous data replication operations by processing batches sequentially without interrupting the overall replication workflow. The system continuously moves from one batch to the next, maintaining productive action throughout the replication process. This continuity minimizes total replication time and reduces system downtime compared to stopping and starting the replication process.
3Productivity
If entire data batches are loaded into memory for processing, then processing speed is improved, but memory consumption increases leading to system overload
Solution Approach 1:
The patent applies partial action by loading only the necessary portion of data (one batch at a time) into memory rather than the entire dataset. Each batch contains sufficient data to make meaningful processing progress, but the volume is controlled to prevent memory overload. This partial loading approach maintains adequate processing speed while constraining memory consumption within safe limits.
Data Source
AI summary
Methods, systems, and computer-readable storage media for data replication. Data records associated with business entities are obtained. A plurality of data fields is defined for each record. A first set of data records is determined as associated with a first identifier of a first business entity. Data from a first set of data fields is selected from each data record of the first set of the data records. The first set of data fields are a subset of the plurality of data fields and is defined for evaluation of the first set of data records associated with the first business entity to determine a first data record from the first set of data records to be replicated from the data management system into a database system.