Storage Engine Check Code Protection Across Logical Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Performing data integrity checks on large objects stored in high-speed storage devices consumes significant bandwidth and processor resources due to the need to retrieve and recalculate check codes for the entire data, especially when the object spans multiple logical blocks.
Innovation Solution
Implement a storage system that utilizes a network interface to generate a check code for the entire object and a storage engine to perform integrity checks using check codes generated at the storage device level, reducing the need to retrieve and recalculate codes for the entire data, thereby conserving bandwidth and processor resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data object is retrieved from storage device to verify integrity, then data integrity can be confirmed, but bandwidth consumption and memory usage increase significantly
Solution Approach 1:
The patent extracts only the check code from the stored data object rather than retrieving the entire object. The storage device returns only the check code (e.g., CRC code) associated with the data, which is then verified against a newly calculated check code. This extraction principle reduces bandwidth consumption from transferring entire large objects to transferring only small verification codes.
Solution Approach 2:
Instead of performing a complete data retrieval and full verification process, the patent applies partial action by only retrieving and verifying the check code portion. This partial verification approach is sufficient to confirm data integrity without the excessive resource consumption of full data transfer, directly addressing the bandwidth efficiency problem.
2Reliability
If the entire data object is retrieved to calculate check code, then data integrity can be verified, but processor occupancy and computation resources increase
Solution Approach 1:
The patent extracts only the check code from storage and performs verification on this small portion rather than processing the entire data object. This extraction reduces processor workload from calculating check codes over gigabytes of data to calculating check codes over kilobytes of verification data only.
Solution Approach 2:
The patent applies partial action by performing verification only on the check code portion rather than the complete data object. This partial verification maintains reliability while significantly reducing computation resources and processor occupancy, as the check code is a small fraction of the original data size.
3Reliability
If check code verification is performed on entire data objects, then comprehensive integrity check is achieved, but time consumption increases for large objects
Solution Approach 1:
The patent extracts and verifies only the check code component rather than processing entire large data objects. This extraction reduces verification time from potentially seconds or minutes for large objects to milliseconds for small check codes, while maintaining comprehensive integrity verification through the mathematical properties of check code algorithms.
Solution Approach 2:
The patent performs partial verification on the check code portion, which is sufficient to guarantee complete data integrity due to the mathematical relationship between the data and its check code. This partial approach dramatically reduces verification time while achieving the same reliability as full-object verification.
Data Source
AI summary
A client device may transmit a data stream including an object (such as a data file or record) to a storage system for storing the object in a storage device. In response to receiving the data stream, the storage system may store or write data of the object into a plurality of logical blocks of the storage device with an end-to-end data protection based at least in part on a comparison of a combination of check codes of a header of the object, the object and padding data with a combination of a plurality of check codes that are generated separately for metadata associated with the object and corresponding parts of the object stored in the plurality of logical blocks. The storage system may further provide an end-to-end data protection for reading data of an object stored in multiple logical blocks of a storage device.