End-to-End Integrity Storage System Using Frozen Buffer Copies
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face challenges in maintaining end-to-end data integrity due to bugs, human error, and hardware issues, leading to frequent data corruption and errors, especially with growing data volumes, where traditional methods like host kernel CRC calculations can cause overhead and mismatch issues.
Innovation Solution
Implementing an optimized end-to-end integrity system that writes requested application data and only performs integrity checks upon failure, using cyclic redundancy checks (CRC) with frozen application data, and marking buffers as read-only or copy-on-write to prevent data corruption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If host kernel calculates CRCs for data integrity, then data integrity is improved, but system overhead increases and buffer modification issues occur
Solution Approach 1:
The patent creates a copy of the buffer (frozen buffer) containing the data before the buffer is modified. The CRC is calculated from this frozen copy, ensuring integrity checking is based on the original data state. This copying approach allows the buffer to be modified afterward without affecting the integrity calculation, resolving the conflict between continuous data modification and integrity verification.
Solution Approach 2:
The patent performs the CRC calculation and buffer freezing action before the buffer can be modified by subsequent operations. By establishing the integrity baseline in advance, the system ensures that any later modifications to the buffer do not compromise the integrity check, thereby maintaining data integrity without requiring continuous overhead calculations.
2Reliability
If buffers are marked read-only to prevent modification, then data integrity is improved, but memory management complexity increases
Solution Approach 1:
Instead of making the original buffer read-only, the patent creates a separate frozen copy of the buffer data. The original buffer remains writable for normal operations, while the frozen copy is used for integrity calculations. This approach maintains data integrity through the frozen copy while avoiding the memory management complexities of read-only marking.
Solution Approach 2:
The frozen buffer acts as an intermediary between the original writable buffer and the integrity verification process. Rather than restricting the original buffer, the frozen copy serves as a mediator that preserves the data state needed for CRC calculation while allowing the original buffer to continue its normal writable operations.
3Reliability
If CRC calculations are performed frequently, then data integrity verification is improved, but processing time increases
Solution Approach 1:
The patent performs the CRC calculation in advance, before the buffer data is modified. By establishing the integrity baseline beforehand, the system avoids the need for repeated CRC calculations during subsequent buffer modifications, thereby reducing processing time while maintaining verification reliability.
Solution Approach 2:
By creating a frozen copy of the buffer data, the patent enables the CRC to be calculated once from the original data state without requiring repeated calculations when the buffer is modified. This single calculation on the frozen copy provides ongoing integrity verification without the time cost of frequent recalculation.
Data Source
AI summary
A method of optimized end-to-end integrity comprises receiving a request to write application data stored in a first kernel buffer to a storage device. The method further comprises determining, by a processing device, that a first cyclic redundancy check (CRC) of the application data failed. The method further comprises copying the application data to a second kernel buffer. The method further comprises performing, by the processing device, a second CRC on the application data stored in the second kernel buffer.