Remote Data Integrity Verification via Track Metadata Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication systems face inefficiencies in verifying data integrity between primary and backup storage arrays, as they require costly CPU and disk bandwidth to compare hashes, leading to potential data mismatches and resource consumption.
Innovation Solution
Implementing in-line compression and deduplication on the primary storage array, which generates fingerprints and spatial/temporal metadata, allowing for remote data integrity verification by comparing these metadata without reading the actual data, thereby reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional hash comparison methods are used to verify data integrity between primary and backup storage arrays, then data consistency can be verified, but CPU resources and disk bandwidth are consumed excessively
Solution Approach 1:
The system performs preliminary actions by creating fingerprints and spatial/temporal metadata during the normal data write operation on the primary storage array, before replication occurs. These fingerprints and metadata are attached to the data and transmitted to the backup storage array, eliminating the need for subsequent hash comparison operations that would consume excessive CPU resources.
Solution Approach 2:
Instead of copying the entire data for verification, the system creates and transmits only the fingerprint and metadata representations of the data. The backup storage array stores these fingerprints and metadata, and verification is performed by comparing these compact representations rather than the actual data, significantly reducing resource consumption.
2Reliability
If traditional hash comparison methods are used to verify data integrity, then data consistency can be verified, but disk bandwidth is consumed excessively
Solution Approach 1:
The system extracts only the essential verification information (fingerprints and spatial/temporal metadata) from the data during the write operation. This extracted information is what gets replicated and verified, rather than replicating and verifying the entire data content, thereby reducing disk bandwidth consumption during verification operations.
Solution Approach 2:
By performing the extraction of fingerprints and metadata as a preliminary action during the initial data write, the system avoids the need for subsequent bandwidth-intensive hash comparison operations. The verification process only requires comparing the pre-extracted fingerprints and metadata, not the actual data blocks.
3Productivity
If in-line compression and deduplication are implemented on the primary storage array, then data efficiency is improved, but device complexity increases
Solution Approach 1:
The system merges multiple functions into a single integrated process: compression, deduplication, fingerprint generation, and metadata creation all occur as part of the normal data write operation on the primary storage array. This consolidation means that while the device complexity increases, the operational complexity remains low as these functions execute automatically during standard data operations.
Data Source
AI summary
A remote data replication facility includes a primary storage array and a backup storage array, on which tracks of data are replicated from the primary storage array to the backup storage array as they are received by the primary storage array. Remote data verification is implemented on the remote data replication facility by comparing track fingerprints, track temporal write metadata, and track spatial write metadata, for a given track on the primary storage array, with corresponding track fingerprints, track temporal write metadata, and track spatial write metadata, for the given track on the backup storage array. If any difference is determined in the combination of track fingerprints, track temporal write metadata, track spatial write metadata, for a given track, the integrity of the data at the backup storage array is not verified for the track.