Bloom Filter File Verification for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplicated data storage systems face inefficiencies in file verification due to hardware or software inconsistencies, leading to potential data loss and slow verification processes, especially when dealing with large datasets where not all live segment references and segments fit into memory, causing verification to lag behind.
Innovation Solution
The implementation of bloom filters to track live references and fingerprints in a deduplicated storage system, allowing for a probabilistic verification method that compares live reference and index vectors, enabling fast identification of mismatches without requiring full memory access, and performing verification in a breadth-first manner to reduce redundant processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the file system performs logical file verification by traversing the segment tree in depth first manner and comparing checksums, then data consistency can be verified, but the verification process becomes extremely slow due to random disk input and output operations
Solution Approach 1:
The patent divides the verification process into two distinct phases: a fast probabilistic filter phase using bloom filters to identify potential mismatches, and a slower verification phase that only processes segments flagged by the filter. This segmentation eliminates redundant random disk I/O operations by pre-filtering candidates before actual verification.
Solution Approach 2:
The patent performs preliminary action by constructing bloom filters for live references and index fingerprints before the actual verification process. This preliminary filtering step identifies potential mismatches in advance, so that subsequent verification only needs to check a small subset of segments rather than performing random disk I/O for all segments.
2Reliability
If the file system enumerates all live segment references and segments to verify consistency, then complete verification coverage is achieved, but memory requirements exceed available memory capacity
Solution Approach 1:
The patent creates simplified copies (bloom filters) of the live reference sets and index fingerprint sets that occupy minimal memory space. These bloom filter copies preserve the essential verification capability by enabling membership queries, while using only a fraction of the memory required to store the complete segment reference data.
Solution Approach 2:
The patent changes the parameter representation from storing complete segment reference data to using bloom filter data structures with configurable false positive rates. By adjusting the bloom filter parameters (size, number of hash functions), the system achieves a balance between memory usage and verification reliability, fitting large datasets into available memory.
3Reliability
If the file system uses traditional checksum comparison verification, then data integrity can be confirmed, but the process lags behind by weeks or months due to insufficient memory and slow disk operations
Solution Approach 1:
The patent introduces bloom filters as intermediary data structures that mediate between the complete segment reference data and the verification process. These intermediaries enable fast in-memory comparisons to identify potential mismatches, reducing the verification delay from weeks/months to a fraction of that time, while traditional checksum verification is performed only on the small subset of candidates identified by the bloom filters.
Data Source
AI summary
According to one embodiment, fingerprints of segment trees are scanned, each segment tree representing one of the files in a filesystem namespace. For each of the fingerprints representing a segment, setting a corresponding bit in a live reference vector (LRV) to indicate that the segment has been referenced by a file in the filesystem namespace. A file index mapping fingerprints to storage locations of segments is scanned, including, for each fingerprint found in the file index, setting a corresponding bit in a live index vector (LIV) to indicate that the fingerprint exists in the file index. The LR vector and the LI vector are compared to determine whether there is any mismatch. A garbage collection operation is performed in response to determining that the LR vector and the LI vector are matched.