Distributed Change Data Capture with Parallel Log Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional change data capture mechanisms in database systems are inadequate for handling high change rates and require short latency, as they rely on sequential processing which is insufficient for objects with high change rates and stringent latency requirements.
Innovation Solution
Implementing parallel processing of logged changes by dividing them into sub-portions and processing each sub-portion independently, with mechanisms to track and manage collisions, commits, and handle failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential processing is used for change data capture, then system complexity is low and ease of operation is maintained, but productivity and latency performance deteriorate for high change rate objects
Solution Approach 1:
The patent divides the logging table entries into multiple sub-portions and assigns each sub-portion to a separate process for parallel compilation. This segmentation enables multiple processes to work simultaneously on different portions of the change data, dramatically improving throughput for high change rate objects while maintaining manageable complexity through structured division of labor
2Loss of time
If sequential processing is used, then processing simplicity is maintained, but latency between source object changes and target object transfer increases
Solution Approach 1:
By segmenting the change data into sub-portions and processing them in parallel across multiple processes, the patent reduces the overall time required to compile and transfer changes. Each process works simultaneously on its assigned sub-portion, eliminating the sequential bottleneck and significantly reducing latency for high change rate objects
Solution Approach 2:
The system performs preliminary actions by pre-dividing the logging table entries into sub-portions and preparing multiple processes in advance. This preliminary segmentation and process preparation enables immediate parallel processing when changes occur, reducing the time from source object change to target object transfer
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods include reception of a request for changed data of an object from a subscriber, determination of a logging table associated with the object and comprising a plurality of logging table entries, determination of a pointer to a last-processed entry of the logging table based on the object and the subscriber, definition of a plurality of sub-portions of logging table entries subsequent to the last-processed entry, and reconstruction and transfer of first data associated with a first one of the plurality of sub-portions to the subscriber using a first process, and reconstruction and transfer, in parallel with the first process, second data associated with a second one of the plurality of sub-portions to the subscriber using a second process.