Physical File Verification Using Perfect Hash Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication file systems face inefficiencies in verifying data integrity due to complex algorithms and large file systems, leading to prolonged verification times, which can result in missing data going undetected until it's too late for recovery.
Innovation Solution
Implementing a physical file verification system that uses a perfect hash vector to efficiently track live references and detect missing data segments, reducing the need for repeated disk seeks and improving verification speed by traversing segment trees in a breadth-first manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a file-by-file verification approach is used, then verification reliability is improved, but verification time increases excessively
Solution Approach 1:
The patent merges the verification of multiple files into a single operation by using a segment tree structure. When multiple files share common data segments, the verification process traverses the segment tree once and verifies all shared segments simultaneously, rather than verifying each file independently. This combining approach maintains verification reliability while dramatically reducing the total verification time for systems with many files sharing data.
Solution Approach 2:
The patent segments the file system into a hierarchical segment tree structure, where data is organized into segments that can be shared across multiple files. The verification process operates on this segmented structure, allowing it to verify common segments once and propagate the verification result to all files that reference those segments, thereby reducing redundant verification operations.
2Reliability
If verification is performed on each file independently, then verification thoroughness is improved, but the number of disk seeks increases
Solution Approach 1:
The patent combines disk access operations by organizing verification around segment trees. When multiple files reference the same data segments, the system performs a single disk seek to retrieve the shared segment and verifies it once, then logically applies the verification result to all files that reference that segment. This merging of disk access operations maintains thorough verification while dramatically reducing the total number of physical disk seeks required.
3Reliability
If traditional verification algorithms are used, then data integrity checking is performed, but verification speed decreases for large file systems
Solution Approach 1:
The patent segments the verification process to operate on a hierarchical segment tree structure rather than on individual files. This segmentation allows the system to identify and verify shared data segments once, then propagate the verification result through the tree structure to cover multiple files simultaneously. This segmented approach maintains comprehensive data integrity checking while significantly improving verification speed for large file systems with many shared segments.
Data Source
AI summary
A perfect hash vector (PHVEC) is created to track segments in a deduplication file system. Files are represented by segment trees having hierarchical segment levels. Containers store the segments and fingerprints of segments. Upper-level segments are traversed to identify a first set of fingerprints of each level. These fingerprints correspond to segments that should be present. The first set of fingerprints are hashed and bits are set in the PHVEC corresponding to positions from the hashing. The containers are read to identify a second set of fingerprints. These fingerprints correspond to segments that are present. The second set of fingerprints are hashed and bits are cleared in the PHVEC corresponding to positions from the hashing. If a bit was set and not cleared, a determination is that there is at least one segment missing. If all bits set were also cleared, a determination is that no segments are missing.