Backup Data Pipeline Verification with Hash Trees and Bitmaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for verifying backup data across a data pipeline are inefficient and time-consuming, as they require object-by-object and field-by-field comparisons, which can lead to data corruption and inefficiencies.
Innovation Solution
A method involving the generation of hash trees and bitmaps at both ends of the data pipeline to efficiently verify the integrity of backup data, using hash trees to compare the source and destination data sets for corruption, and preventing access to the destination storage until verification is complete.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If object-by-object and field-by-field comparison is used to verify backup data, then data integrity can be ensured, but verification time and computational resources increase significantly
Solution Approach 1:
The patent segments the data verification process by dividing the data pipeline into multiple stages and introducing verification checkpoints at each stage. Instead of performing a single exhaustive comparison at the end, the system verifies data integrity incrementally at intermediate points, reducing the total verification time while maintaining comprehensive coverage of data integrity
Solution Approach 2:
The patent applies preliminary action by performing verification operations at intermediate checkpoints during the data pipeline processing, rather than waiting until the end. This allows early detection of corruption and reduces the amount of data that needs to be verified in subsequent stages, thereby reducing overall verification time while ensuring data integrity
2Reliability
If comprehensive verification of all data objects is performed, then data corruption can be detected, but processing efficiency decreases
Solution Approach 1:
The patent implements partial verification by focusing verification efforts on critical data objects and high-risk pipeline stages rather than uniformly verifying all data. The system identifies and prioritizes verification of data objects that are more susceptible to corruption or more critical to business operations, thereby maintaining high corruption detection capability while improving processing efficiency
Solution Approach 2:
The patent introduces checksums and hash values as intermediary verification mechanisms. Instead of directly comparing all data objects, the system uses these computational intermediaries to efficiently detect corruption. The checksums serve as lightweight proxies that can be computed and compared quickly, providing reliable corruption detection without the overhead of exhaustive data comparison
3Reliability
If data verification is performed at multiple pipeline stages, then data integrity is better ensured, but system complexity increases
Solution Approach 1:
The patent implements a universal verification framework that uses the same core verification mechanisms (checksums, hash functions, comparison algorithms) across all pipeline stages. This multi-functional approach allows the system to maintain consistent data integrity verification throughout the pipeline without requiring stage-specific verification logic, thereby reducing overall system complexity while ensuring comprehensive data integrity
Data Source
AI summary
Systems, devices, and techniques are disclosed for verification of backup data across a data pipeline. Records from a first storage may be received at a first end of a data pipeline. The records may be hashed to generate first hashes. A first hash tree may be generated from the first hashes. The records may be received at a second end of the data pipeline. Bits of bitmaps that correspond to the records may be set. The records may be hashed to generate second hashes. The records may be stored in a second storage. A second hash tree may be generated form the second hashes. Using the bitmaps, whether all of the records or any duplicate records were received may be determined. The first hash tree and the second hash tree may be compared to determine if any of the records stored in the second storage are corrupt.