Automatic Consensus Log Recovery Using Rescuee Roles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing consensus protocols do not effectively handle scenarios where a server has a corrupted replication log, leading to manual intervention and prolonged downtime.
Innovation Solution
The implementation of a consensus protocol with automatic recovery of corrupted log entries, where servers store metadata containing the last log index and term, allowing them to recover from corruption by truncating the log and converting to a rescuee role to receive corrected entries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual intervention is used to recover corrupted logs, then recovery accuracy is improved, but recovery time and operational complexity increase significantly
Solution Approach 1:
The system automatically detects corrupted log entries, identifies the corruption point, and recovers by truncating the log at the corruption point and replaying from a known good state. This self-service mechanism eliminates the need for manual expert intervention while maintaining reliable recovery accuracy through automated corruption detection and validation protocols.
Solution Approach 2:
The system maintains a record of the last known good log entry index before corruption occurs. When corruption is detected, this pre-stored information is immediately used to determine the truncation point, eliminating the need for time-consuming manual analysis of log history and enabling instant recovery initiation.
2Reliability
If manual expert intervention is required for log recovery, then recovery precision is improved, but system availability and ease of operation deteriorate
Solution Approach 1:
The automated recovery system performs all recovery operations without human intervention. The system self-detects corruption, self-determines the recovery point using stored metadata, self-truncates the corrupted log, and self-validates the recovery process, thereby maintaining precision while dramatically improving ease of operation.
Solution Approach 2:
The system introduces an automated recovery manager that acts as an intermediary between the corrupted log and the recovery process. This intermediary automatically handles all recovery operations, including corruption detection, truncation decision-making, and log reconstruction, eliminating the need for manual expert operations while ensuring precise recovery.
3Device complexity
If the log is used to determine validity of append entries calls, then protocol simplicity is maintained, but system reliability deteriorates when corruption occurs
Solution Approach 1:
The system segments the log validation process into two independent parts: (1) using the log for normal append entries validation during healthy operation, and (2) using stored metadata (last known good index) for validation when corruption is detected. This segmentation allows the system to maintain protocol simplicity during normal operation while ensuring reliability when corruption occurs by switching to the alternative validation mechanism.
Solution Approach 2:
The system pre-stores the last known good log entry index and term metadata before corruption can occur. This preliminary action creates a fallback validation mechanism that can be immediately activated when corruption is detected, allowing the system to maintain reliability without complicating the normal protocol flow, as the fallback is only activated when needed.
Data Source
AI summary
The illustrative embodiments provide a consensus protocol with recovery of corrupted log entries within a log stored at a server within a cluster. Each server stores metadata comprising a last log index and a last log term of the copy of the log at the server. In response to a server having one or more corrupted log entries, the server makes a backup of the log and truncates log entries. The server then converts to a rescuee role. When an append entries call is received, the server uses the last log index and last log term from the metadata to determine whether the append entries call is received from a valid leader and can be appended to the truncated log. The server then applies one or more append entries calls to recover the corrupted log entries. Once the corrupted log entries are recovered, the server converts to the follower role.