Database Log End-of-Log Identification via Discard Markers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In database mirroring systems, determining the end-of-log is challenging due to inconsistencies in log processing between the principal and mirror databases, leading to potential data loss and corruption during failover scenarios, especially when the mirror is behind in processing transactions, causing issues in maintaining transactional consistency and identifying the end-of-log for recovery.
Innovation Solution
The method involves creating a logical discard log file, which is invalidated by replacing its contents with an identifiable pattern, allowing the database to discard unnecessary log records and ensuring internal consistency by resetting the storage area information, enabling the system to determine the end-of-log for recovery and start processing new transactions efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the mirror processes log records asynchronously and may lag behind the principal, then the mirror can operate independently and maintain best-effort availability, but the log file grows arbitrarily large with uncommitted transaction records that cannot be discarded
Solution Approach 1:
The system performs preliminary actions by marking log records as discardable before the mirror actually processes them. The principal database identifies and marks log records that are safe to discard based on commit status and mirror processing progress, allowing the mirror to operate asynchronously without blocking the principal from cleaning up old logs.
Solution Approach 2:
An intermediary mechanism is introduced through the discard log marker system that mediates between the principal and mirror. The principal can mark logs for discarding without directly preventing the mirror from reading them, and the mirror can continue processing while the principal prepares for future discarding operations. This intermediary marking system resolves the conflict between log retention and log cleanup.
2Reliability
If the principal waits for the mirror to harden log records before committing transactions, then transactional consistency is maintained, but long-running transactions produce large volumes of log records that cannot be discarded
Solution Approach 1:
The system performs preliminary marking of log records as discardable once they are safely committed and the mirror has processed them, even though the mirror may not have hardened them yet. This preliminary marking allows the principal to identify and prepare for log discarding without waiting for the mirror's hardening process, thus managing log file growth while maintaining consistency.
Solution Approach 2:
The system implements a mechanism to discard log records that have been committed and processed by the mirror, even if the mirror hasn't hardened them yet. The discard markers enable the principal to reclaim log space by identifying records that are safe to discard, while the mirror can later recover or reprocess these records if needed, thus balancing consistency with log management.
3Productivity
If forced failover is initiated when the mirror is significantly behind the principal, then the system can restore service quickly, but data loss may occur and the log contains arbitrary records beyond what is useful for recovery
Solution Approach 1:
Before forced failover occurs, the system performs preliminary identification and marking of discardable log records on the principal. When failover is initiated, the new principal (former mirror) can quickly determine which log records to keep and which to discard based on these pre-marked regions, enabling fast recovery without having to analyze the entire log file, thus achieving quick failover while minimizing data loss.
Solution Approach 2:
The system converts the potential harm of having an arbitrarily large log file with uncommitted records into a benefit by using the discard markers to quickly identify and discard irrelevant portions during failover. The large log file that would normally be a burden becomes manageable because the markers enable rapid identification of the actual end-of-log point, allowing fast failover without processing unnecessary records.
4Productivity
If the principal discards log records to manage file size, then log management efficiency improves, but the ability to recover from crashes is compromised if discard markers are not properly managed
Solution Approach 1:
The discard markers serve as an intermediary mechanism between log retention and log discarding. These markers are written to stable storage and act as a safety layer that ensures recovery capability is maintained. The markers indicate which log records are safe to discard, and their presence in stable storage provides a verification mechanism that recovery can proceed safely, thus enabling efficient log management without compromising reliability.
Solution Approach 2:
The system implements feedback through the discard marker verification process. Before discarding log records, the system verifies the presence and validity of discard markers in stable storage. This feedback mechanism ensures that only logs that are safe to discard are removed, maintaining recovery capability. The feedback loop between marker verification and log discarding decisions ensures both efficiency and reliability.
Data Source
AI summary
A method of invalidating an unused log file in a database includes examining a discard log information storage area and determining if a discard log exists. If the discard log exists, the log is invalidated. The headers of each of the log files that have been fully invalidated are then invalidated. The discard log is then fully invalidated and a new log file may be generated on top of the discarded log. The method may be interrupted by a failure, a changeover, or crash and will be repeated until it completes.