Database Redo Record Scanning for Heterogeneous Instance Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In heterogeneous database systems, failures in RDBMS instances or compute nodes lead to data inconsistency and loss, disrupting transactional consistency and requiring inefficient data reloading or logging methods for recovery.
Innovation Solution
The proposed solution involves scanning REDO records to identify and transfer only modified data blocks not yet propagated to compute nodes, using a low-watermark value to reduce scanning scope and avoiding per-block logging, thus ensuring transactional consistency without reloading all data blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If log-based methods are used to synchronize data between source and destination, then transactional consistency is maintained, but data loss may occur when RDBMS fails and change records are lost
Solution Approach 1:
The system performs preliminary actions by maintaining a watermark value that tracks the last successfully propagated change record. This watermark is updated during normal operation before failures occur, ensuring that when recovery is needed, only records after the watermark need to be retransmitted, preventing loss of change records while maintaining consistency.
Solution Approach 2:
The system implements feedback mechanisms where the destination node acknowledges receipt of change records and the source node tracks which records have been successfully propagated using watermark values. This feedback loop ensures reliable delivery of change records and enables efficient recovery by knowing exactly what needs to be retransmitted after a failure.
2Reliability
If load-based methods are used to reload data from source to destination, then data availability is improved, but data inconsistency occurs during reloading and performance deteriorates
Solution Approach 1:
The system segments the data synchronization process by transmitting only modified data blocks along with their associated change records, rather than reloading entire tables or datasets. This segmentation approach maintains data availability at the compute node while improving performance by minimizing the amount of data transferred and processed during recovery operations.
Solution Approach 2:
Instead of performing complete data reloading (excessive action), the system applies partial action by selectively transmitting only the modified data blocks that changed since the last successful propagation. This partial approach sufficient for recovery purposes while dramatically improving performance compared to full reloads.
3Reliability
If all data blocks are reloaded during recovery, then data consistency is restored, but time consumption increases significantly
Solution Approach 1:
The system performs preliminary tracking of data block modification status using watermark values that record the last successfully propagated changes. This preliminary action enables the recovery process to start from the correct position in the change log, avoiding the need to review and reload all historical data blocks and significantly reducing recovery time while maintaining consistency.
Solution Approach 2:
The system extracts only the necessary change records and modified data blocks for recovery, separating them from the vast amount of unchanged historical data. By taking out only the essential recovery information (changes after the watermark), the system restores data consistency without the time-consuming operation of processing all historical data blocks.
4Measurement precision
If per-block logging is implemented, then data tracking precision is improved, but device complexity and storage requirements increase
Solution Approach 1:
The system merges the change record tracking function with the existing RDBMS transaction logging infrastructure. Instead of implementing separate per-block logging, the solution leverages the RDBMS's native change record mechanism (watermark values and change logs) to track data modifications. This merging approach maintains precise data tracking while avoiding the complexity of additional logging infrastructure.
Data Source
AI summary
A method and apparatus for data recovery for a RDBMS instance in a heterogeneous database system is provided. A failure of a first RDBMS instance is detected in a plurality of RDBMS instances of a shared-disk database system. A compute cluster is configured to store, in memory, one or more tables stored by the shared-disk database system. The first RDBMS instance is configured to modify the one or more tables stored by the shared-disk database system and transfer modified data to the compute cluster to update the one or more tables at the compute cluster. After detecting the failure of the first RDBMS instance, redo records generated by the first RDBMS instance are scanned, pending modified data that was not transferred to the compute cluster before the failure is identified, and the pending modified data is transferred to the compute cluster.