Cloud Data Verification via Checksum Translation Map
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud storage systems face challenges in verifying data integrity due to high latency and cost associated with read-verify operations, and existing checksum methods fail to detect errors during data writing to disk, leading to potential corruption that may not be discovered until later, making corrective actions difficult.
Innovation Solution
A verification framework that utilizes a local verifier engine and a cloud verifier engine to create a translation map, allowing for efficient verification of data written to the cloud tier without incurring substantial costs, by comparing checksums and reconstructing corrupted objects from the local tier, ensuring data integrity and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If read-verify operations are performed on cloud tier data, then data integrity is ensured, but latency and cost increase significantly
Solution Approach 1:
The patent applies preliminary action by generating checksums for cloud tier objects before the actual write operation completes. The checksum is stored in the object metadata ahead of time, so when verification is needed, the system can immediately compare the stored checksum with a newly generated one without waiting for the write to finish or performing expensive read operations. This pre-computation approach ensures data integrity while minimizing verification latency.
2Reliability
If read-verify operations are performed on cloud tier data, then data integrity is ensured, but verification cost becomes prohibitive
Solution Approach 1:
The patent replaces the mechanical read-verify operation (which requires physically reading data from cloud storage) with a computational checksum comparison. Instead of performing expensive cloud read operations to verify data, the system generates a checksum from the uploaded data, stores it in metadata, and later verifies by comparing checksums. This substitution eliminates the need for costly data egress and cloud read operations, dramatically reducing verification costs while maintaining integrity checks.
3Productivity
If checksum is generated before writing to disk, then write speed is maintained, but errors during disk writing are not detected
Solution Approach 1:
The patent implements feedback by continuously monitoring the data integrity state through checksum comparison. When an object is uploaded and a checksum is generated, the system stores this checksum in metadata. Later, when the object is read or verified, the system generates a new checksum and compares it with the stored one. This feedback mechanism detects any corruption that occurred during disk writing or storage, allowing the system to identify and handle corrupted data while maintaining high write speeds through the use of checksums rather than slow read-verify operations.
4Productivity
If local tier data is reclaimed through garbage collection, then storage efficiency is improved, but data availability for correction is lost
Solution Approach 1:
The patent introduces an intermediary verification mechanism using checksums stored in object metadata. Instead of relying on local tier data availability to detect and correct errors, the system uses the stored checksum as an intermediary reference that can verify data integrity independently. When data is corrupted in the cloud tier, the system can detect the corruption by comparing the stored checksum with a newly generated one, and the metadata checksum serves as the intermediary evidence needed for correction without requiring the original local data to still be present.
Data Source
AI summary
Data protection operations including verification operations are disclosed. Objects written to a cloud tier are verified without reading the objects out of the cloud. A translation map is used that allows a cloud verifier engine to compare a checksum of an object generated at an local tier with a checksum of the object as stored in the cloud tier. Mismatches are identified and corrective actions can be taken by reconstructing and rewriting the object to the cloud tier. Garbage collection may be prevented from reclaiming data associated with objects that have not been verified.