Recovery Log Checksum Validation for Backup Data Integrity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file systems face data loss and corruption issues due to inconsistent recovery logs, particularly in systems using NVRAM or NVMe-based distributed logs, leading to loss of data during crash recovery and excessive disk pressure.
Innovation Solution
Implementing synchronous and asynchronous validation of data units in recovery logs using checksums before flushing data to storage, ensuring data integrity and preventing loss or corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous validation with checksum verification is implemented for each data unit in recovery logs, then data reliability and integrity are improved, but processing time and computational overhead increase
Solution Approach 1:
The patent applies preliminary action by computing and storing checksums for data units when they are first written to the recovery log. This pre-computed validation data is then reused during backup generation, avoiding the need to re-verify every data unit and thus reducing processing time while maintaining data integrity verification.
Solution Approach 2:
The patent uses copying by creating and storing checksum copies of data units separately from the original data. These checksum copies serve as validation tokens that can be quickly compared against original data during backup operations, providing efficient verification without re-processing the entire data units.
2Reliability
If checksum verification is performed for every data unit during backup generation, then data corruption is detected and prevented, but backup generation speed decreases
Solution Approach 1:
The system performs preliminary checksum computation and stores validation tokens alongside data units in the recovery log. During backup generation, it retrieves and compares these pre-stored checksums rather than re-computing them, enabling fast verification that maintains corruption detection while improving backup generation speed.
Solution Approach 2:
The recovery log structure includes self-verification capabilities where each data unit carries its own checksum token. The system uses these self-contained validation markers to verify data integrity without requiring external validation processes, thus maintaining reliability while reducing processing overhead.
3Reliability
If validation is performed synchronously for each data unit, then data integrity is ensured before backup completion, but system throughput and performance decrease
Solution Approach 1:
The patent implements preliminary validation by computing checksums and storing validation tokens when data units are initially logged. This upfront validation work allows the backup process to proceed rapidly by simply comparing against pre-computed checksums rather than performing synchronous verification during backup, thus maintaining integrity assurance while improving throughput.
Solution Approach 2:
The system uses periodic validation where checksums are computed and verified at specific intervals (when data units are logged to the recovery log) rather than continuously during every backup operation. This periodic approach ensures data integrity while minimizing the performance impact on system throughput.
Data Source
AI summary
Techniques described herein relate to a method for managing recovery logs. The method includes obtaining, by a validation manager of a backup storage, a first snapshot comprising a first plurality of data units; making a first determination that the snapshot is associated with synchronous validation; in response to the first determination: writing a first data unit of the plurality of data units to a recovery log; computing a new checksum associated with the first data unit; comparing the new checksum with a previously generated checksum associated with the first data unit; making a second determination that the new checksum and the previously generated checksum match; and in response to the second determination: generating a backup using the plurality of data units.