Data Replication Threads Using Batch Sequence Numbers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication technologies face inefficiencies in maintaining task order across multiple threads and network connections, leading to suboptimal use of network bandwidth and potential inconsistencies in data replication.
Innovation Solution
The solution involves forming batches of tasks for each data replication thread, ensuring that tasks are sent in parallel over multiple connections while maintaining the correct order, using batch sequence numbers to determine the next batch to be executed, and employing a kernel cache to manage and track tasks effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tasks are sent in parallel over multiple network connections, then network bandwidth utilization is improved, but maintaining task order becomes more difficult
Solution Approach 1:
The patent segments tasks into batches and assigns each batch a sequence number. Multiple threads can send different batches in parallel over multiple network connections, while the sequence numbers ensure proper ordering at the target. This segmentation approach allows parallel transmission without losing task order.
Solution Approach 2:
The patent introduces batch sequence numbers as an intermediary mechanism to mediate between parallel task transmission and ordered execution. The sequence numbers act as tags that allow the target to reorder and process tasks correctly even when received out of order due to parallel transmission.
2Speed
If tasks are sent in parallel over multiple connections, then replication speed is improved, but data consistency may be compromised
Solution Approach 1:
The patent performs preliminary actions by assigning batch sequence numbers to tasks before transmission. This pre-tagging ensures that even though tasks are transmitted in parallel at high speed, their execution order can be correctly determined at the target based on the pre-assigned sequence numbers, maintaining data consistency.
Solution Approach 2:
The patent implements feedback mechanisms where the target processes batches in sequence number order and can request retransmission of out-of-order batches. This feedback loop ensures data consistency is maintained while allowing parallel transmission to proceed at high speed.
3Productivity
If multiple threads are used for replication, then throughput is improved, but error handling becomes more complex
Solution Approach 1:
The patent segments the replication process into independent threads, each handling specific batches of tasks. Each thread maintains its own sequence number tracking, which simplifies error handling within each thread while allowing parallel operation. If an error occurs in one thread, it can be handled independently without affecting other threads.
Solution Approach 2:
The patent changes the parameter tracking approach by using batch sequence numbers instead of global task counters. This parameter change allows each thread to independently track and manage its batches, simplifying error handling and recovery while maintaining high throughput through parallel operation.
Data Source
AI summary
A replicated set of data is processed by receiving at a target, from one of a plurality of replication processing threads, a received batch of one or more non-synchronization tasks. It is determined that the received batch comprises a next batch to be performed at the target and the non-synchronization tasks included in the batch are performed in a task order.