Database Synchronization Using Cached Log Streams and Block Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing CDC technologies integrating full and incremental data face performance degradation due to pausing log transactions for full data version marking and scanning, leading to intrusion into incremental log streams and reduced processing efficiency.
Innovation Solution
A database synchronization method that reads and caches incremental log streams into memory, sends data in sequence, marks real-time positions, pauses and filters old-version data blocks, and continues iterations without blocking normal processing, ensuring data consistency and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full data version marking and scanning are performed to ensure data consistency, then data consistency is improved, but processing efficiency deteriorates due to pausing log transactions
Solution Approach 1:
The patent performs preliminary actions by reading and caching the incremental log stream into memory before processing, and by pre-dividing the data into data blocks. This allows the system to prepare data in advance without blocking log transactions, thus maintaining data consistency while improving processing efficiency.
Solution Approach 2:
The patent segments the data processing into multiple data blocks and processes them independently. By dividing the full data scan into manageable segments (data blocks), the system can process data in parallel without blocking log transactions, resolving the contradiction between data consistency and processing efficiency.
2Productivity
If incremental log stream is processed in real-time without full data scan, then processing efficiency is improved, but data consistency deteriorates due to missing historical data
Solution Approach 1:
The patent merges the incremental log stream processing with full data scan by combining both approaches. The system processes incremental logs in real-time while also scanning full data blocks, merging the benefits of both methods to achieve both high processing efficiency and data consistency.
Solution Approach 2:
The patent performs preliminary full data scanning and caching before incremental processing, so that historical data is available when needed. This preliminary action ensures data consistency is maintained while allowing efficient incremental processing to proceed without blocking.
3Quantity of substance
If full data scan is performed to obtain all upstream data, then data completeness is improved, but time consumption increases due to sequential processing
Solution Approach 1:
The patent segments the full data scan into multiple data blocks that can be processed in parallel. By dividing the complete data into segments, the system can retrieve and process all data simultaneously rather than sequentially, maintaining data completeness while reducing time consumption.
Solution Approach 2:
The patent transitions from sequential processing to parallel processing by introducing a dimensional change in the processing architecture. Multiple data blocks are processed concurrently across different dimensions (threads or processes), dramatically reducing the time to obtain all upstream data while maintaining completeness.
Data Source
AI summary
An incremental log stream of a first database is read and cached into a memory; in any round of iteration, incremental log data in the incremental log stream is sent to a second database in sequence from a first position in the incremental log stream cached in the memory, and a second position of incremental log data which is latest sent to the second database is marked in real time; any to-be-read data block in the first database is read and sending the incremental log data to the second database is paused; old-version data in the to-be-read data block is filtered out based on the incremental log data between the first position and the second position and a preset filtering rule, and filtered data is sent to the second database; and the first position is moved to the current second position and a next round of iteration is continued.