Replication Queue Batching for Data Load Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data replication between systems is resource-intensive, particularly during large data loads, leading to memory and performance issues, and existing methods fail to efficiently manage the replication process, especially when handling a huge volume of data.
Innovation Solution
Implementing a system where data is pre-packaged by assigning a 'replication time' to groups of records in a replication queue, limiting the number of objects replicated together, and dynamically adjusting the package size based on system capabilities, frequency of queries, and historical performance data to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If all changed data objects are replicated together in one batch, then the replication process is simple to implement, but the system experiences memory and performance issues during large data loads
Solution Approach 1:
The patent divides the replication process into multiple batches by assigning different replication times to different groups of data objects. Instead of replicating all changed objects simultaneously, the system segments them into batches of a specified maximum number, with each batch assigned a sequential replication time. This segmentation reduces memory pressure and improves system performance during replication operations.
2Productivity
If a large number of data objects are assigned the same replication time, then replication efficiency is improved, but resource consumption increases significantly
Solution Approach 1:
The system dynamically adjusts the replication process by implementing a queue-based mechanism where data objects are assigned replication times based on current system conditions and queue status. The replication time assignment is not static but adapts to the volume of changed data, system capacity, and historical performance, allowing efficient resource utilization while maintaining good replication throughput.
3Reliability
If the replication queue stores all changed data objects without time assignment, then data completeness is maintained, but memory usage increases during high data volume
Solution Approach 1:
The system performs preliminary action by assigning replication times to data objects as they are added to the replication queue, rather than waiting until replication time. This pre-assignment organizes data into manageable batches in advance, allowing the system to maintain data completeness while controlling memory usage by immediately structuring the queue with time-based groupings.
Data Source
AI summary
A current time associated with a first received notification of a new object in a data set of a first system is accessed and a replication time is set to the current time. For each notification received regarding a new object in the data set of the first system, the new object is added to the replication queue. The new objects added to the queue are assigned the replication time until a specified maximum number of new objects have been added to the replication queue. Afterwards, new objects are added to the queue with no assigned replication time. When the objects in the queue are to be replicated, only the objects with the assigned replication time are copied from the queue to a second system. A new replication time is set to a new current time associated with confirmation of receipt of the copied objects received from the second system.