Cloud Data Verification Using Hash Manifests for Instant Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data protection systems face challenges in minimizing downtime during recovery and keeping pace with high data transaction rates at production sites, leading to potential shutdowns due to backlog of un-logged transactions at backup sites.
Innovation Solution
A method involving cloud-based object storage that breaks down logical units into objects of varying sizes, uses hashes for data integrity verification, and enables bi-directional snapshots to efficiently manage and restore data, allowing instant access and recovery without requiring full data retrieval from the cloud.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If journaling is used to enable continuous data protection and rollback to any point in time, then data protection capability is improved, but the backup site cannot keep pace with high data transaction rates at the production site, causing backlog and potential shutdowns
Solution Approach 1:
The patent segments the LUN into multiple chunks and uses a manifest file to track them. Instead of journaling every data transaction, the system only needs to track changes at the chunk level, significantly reducing the overhead. The manifest file stores metadata about chunks including their hashes, allowing the system to verify data integrity without processing every individual transaction through journaling.
Solution Approach 2:
The patent creates a copy of the manifest file and stores it in the cloud. This cloud-based copy serves as a backup that can be used for verification and recovery. The cloud manifest acts as a reference that allows the production site to continue operating at full speed while the backup site processes verification asynchronously, eliminating the bottleneck caused by real-time journaling.
2Reliability
If the backup site processes every data transaction to maintain data integrity, then data integrity is improved, but the production site must slow down, increasing downtime
Solution Approach 1:
The patent performs partial verification by comparing hashes of chunks against the manifest file. Instead of verifying every single data byte in real-time, the system uses hash comparison which is computationally efficient. This partial verification approach maintains data integrity while allowing the production site to operate at full speed, as the verification can be performed asynchronously on the backup site.
Solution Approach 2:
The patent replaces the mechanical journaling process with a hash-based verification system. Instead of processing and storing detailed journal entries for every transaction, the system uses cryptographic hashes to verify data integrity. This substitution dramatically reduces the processing overhead and allows the backup site to keep pace with high transaction rates without forcing the production site to slow down.
3Quantity of substance
If cloud-based object storage with hashes is used for verification, then storage overhead is reduced and recovery speed is improved, but data verification complexity increases
Solution Approach 1:
The patent changes the verification parameter from detailed transaction logging to hash comparison. By storing only the hash values in the manifest file and comparing these compact representations, the system achieves verification with minimal storage overhead. The hash values are much smaller than full transaction logs, and the comparison operation is computationally efficient despite the added cryptographic step.
Data Source
AI summary
A computer implemented method, system, and computer program product for data replication and restoration. The method includes downloading a plurality of hashes from a cloud corresponding to a set of backed up hashes for a set of chunks, wherein each chunk corresponds to a portion of a LUN at a production site at a point in time, determining a second plurality of hashes for the LUN at the production site, and comparing each of the second plurality of hashes for the LUN at the production site with the plurality of hashes downloaded from the cloud. The method further includes downloading data corresponding to one or more non-matching hashes from the cloud, and restoring the data that was downloaded to the LUN on the production site. Once the LUN has been validated, interception of IOs by a splitter may be stopped.