Database Log Snapshotting for Change Data Capture Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Change data capture (CDC) processes face inefficiencies due to the need to re-read large amounts of database logs after a non-graceful shutdown, especially when dealing with long-running transactions, which affects performance and data integrity.
Innovation Solution
The system addresses this by periodically generating snapshots of database log records for pending transactions and using these snapshots to restore and resume processing, minimizing the need for re-reading by persisting long-running transaction data to disk, thus reducing the overhead and performance penalty.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the CDC process stores all transaction queue data in memory, then data integrity is maintained, but a large amount of log data must be reprocessed after non-graceful shutdown
Solution Approach 1:
The patent segments the transaction queue data into two parts: recently processed transactions remain in memory, while older transactions are periodically flushed to disk. This segmentation allows the system to maintain data integrity for active transactions while reducing the amount of data that needs to be reprocessed after shutdown, directly resolving the contradiction between reliability and time loss.
Solution Approach 2:
The patent implements preliminary action by periodically flushing transaction queue data to disk before shutdown occurs. This pre-positioning of data on disk ensures that after a non-graceful shutdown, the system can resume processing from a known state without needing to reprocess the entire log, thus reducing time loss while maintaining data integrity.
2Productivity
If the CDC process continuously reads and processes database logs, then productivity is maintained, but log data must be re-read after interruption
Solution Approach 1:
The patent performs preliminary action by persisting transaction queue data to disk at periodic intervals. This ensures that after an interruption, the system can resume processing from the last committed position without re-reading previously processed logs, thereby maintaining productivity while reducing energy consumption associated with log re-reading.
Solution Approach 2:
The patent creates a copy of the transaction queue data by flushing it to disk. This copy serves as a recovery point that allows the system to skip re-reading of already processed logs, reducing the overhead of log re-reading while maintaining continuous processing capability and productivity.
3Loss of time
If the CDC process persists all transaction data to disk, then log re-reading is eliminated, but device complexity increases
Solution Approach 1:
The patent applies partial action by persisting only the transaction queue data to disk, not the entire database log. This selective persistence reduces the amount of data that needs to be managed and recovered, thereby limiting the increase in device complexity while still eliminating the need to reprocess most log data after shutdown.
Solution Approach 2:
The patent applies local quality by implementing persistence only for transaction queue data that needs to be preserved, rather than persisting all log data. This targeted approach reduces the complexity of the persistence mechanism by focusing only on the critical data elements that require preservation across interruptions.
4Speed
If the CDC process uses in-memory queues for transactions, then processing speed is fast, but data is lost during non-graceful shutdown
Solution Approach 1:
The patent segments the transaction processing into two states: active transactions remain in memory for fast processing, while completed or paused transactions are flushed to disk for retention. This segmentation allows the system to maintain high processing speed for active transactions while ensuring data retention for transactions that need to be recovered after shutdown.
Solution Approach 2:
The patent performs preliminary action by periodically flushing transaction data to disk before shutdown occurs. This ensures that transaction data is retained on disk even if the process terminates non-gracefully, while the in-memory queues continue to provide fast processing for active transactions during normal operation.
Data Source
AI summary
According to one embodiment of the present invention, a system reduces re-reading of database logs by persisting long running transaction data. The system processes the database log records of the database transaction log; generates a snapshot of the database log records at periodic intervals, wherein each snapshot includes database log records for pending transactions; and in response to an interruption in processing of the database log records, utilizes a snapshot to restore database log records for the pending transactions and resume processing of the database transaction log from a position succeeding the database log records of the selected snapshot.