Unit-Level Checksum Verification for Cloud Range Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In managed cloud services, existing systems face challenges in efficiently verifying data integrity at a granular level, leading to potential corruption and resource-intensive complete object reads during caching, where corruption is only detectable at the end of the read operation.
Innovation Solution
The system calculates and stores checksums at a unit level (e.g., 1 MB) within the application layer, allowing for intermediate and final checksums to be generated and stored, enabling precise verification of data integrity during range reads, thus identifying corruption at a smaller granularity and reducing resource expenditure.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If checksum verification is performed on entire objects during range reads, then data integrity can be confirmed, but resource consumption increases and corruption detection is delayed until the end of the operation
Solution Approach 1:
The patent divides the object into multiple units, each with its own checksum stored in metadata. During range reads, only the checksums of the specific units being accessed need to be verified, rather than reading and verifying the entire object. This segmentation enables efficient partial verification while maintaining data integrity checks.
2Reliability
If checksum verification is performed on entire objects during range reads, then data integrity can be confirmed, but the verification process takes longer to detect corruption
Solution Approach 1:
By storing checksums at the unit level in metadata, the system can immediately verify only the specific units involved in a range read operation. Corruption is detected as soon as the relevant unit checksums are checked, rather than waiting to process the entire object, thus reducing verification time while maintaining detection accuracy.
3Productivity
If checksums are stored at finer granularity (unit level), then verification efficiency during range reads improves, but metadata complexity increases
Solution Approach 1:
The patent organizes metadata to include checksums for each unit within an object. This structured segmentation allows the system to efficiently retrieve and verify only the necessary checksums for partial reads, improving verification efficiency while maintaining an organized metadata structure that manages the complexity through systematic unit-level tracking.
Data Source
AI summary
The technology disclosed herein pertains to a system for data integrity verification that is optimized at unit level. One or more implementations of such system include a method including identifying a first unit boundary in data block received from a client at an application layer, the unit boundary indicating end of a first unit of data, generating an intermediate checksum for the first unit of data at the application layer, receiving a second unit of data of the data block at the application layer, and generating a final checksum with an intermediate checksum as a basis and the second unit of data.