Deduplicated Filesystem Verification Using Fingerprint Index Passes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data backup systems struggle to identify and recover files impacted by data loss due to the difficulty in identifying and listing missing segments, making it challenging to restore data from backup copies.
Innovation Solution
A method and apparatus using a fingerprint hashtable to verify metadata and data segments, employing a breadth-first scan to identify missing segments and impacted files in a deduplicated filesystem, allowing for efficient recovery of impacted files from backup systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data backup systems are used to identify missing segments, then data protection is provided, but it becomes difficult or impossible to identify and list the missing segments and impacted files
Solution Approach 1:
The patent introduces a fingerprint index table as an intermediary data structure that maps file identifiers to data segment fingerprints. This intermediary enables efficient identification of missing segments by providing a direct lookup mechanism, resolving the contradiction between maintaining data protection and enabling easy detection of missing segments.
Solution Approach 2:
The patent replaces traditional mechanical scanning and verification methods with a computational approach using fingerprint hashing and index-based lookup. Instead of physically examining each data segment, the system uses cryptographic fingerprints and index tables to computationally identify missing segments, dramatically improving detection capability.
2Reliability
If all segments are verified in a large-scale deduplicated filesystem, then data integrity is ensured, but the verification process becomes extremely time-consuming and resource-intensive
Solution Approach 1:
The patent segments the verification process by organizing data into hierarchical structures (files divided into segments, segments represented by fingerprints). This segmentation allows the system to verify data integrity at the fingerprint level rather than examining every byte of every segment, significantly reducing verification time while maintaining integrity assurance.
Solution Approach 2:
The patent changes the verification parameter from examining raw data segments to comparing cryptographic fingerprints. By transforming the verification task from a byte-by-byte comparison to a fingerprint matching operation, the system maintains data integrity verification while dramatically reducing the time and computational resources required.
3Measurement precision
If redundant segment verification is performed, then thorough checking is achieved, but system efficiency decreases and redundant processing occurs
Solution Approach 1:
The patent creates a fingerprint copy of each data segment and stores it in the fingerprint index table. Instead of verifying the actual data segments repeatedly, the system verifies the fingerprint copies, which contain the essential verification information. This copying approach maintains thorough checking while eliminating redundant processing of the original data segments.
Data Source
AI summary
One example method includes calculating a size of a fingerprint hashtable, calculating a number of verification passes that are needed, partitioning a fingerprint index table into index ranges corresponding to the number of verification passes needed, and during each verification pass for each index range: (1) finding all Ln segments in a namespace, and adding the Ln segments to the fingerprint hashtable and to a walk-vector data structure, (2) scanning a container set and locating any Ln segments in the container set; (3) reading all Ln-1 segments pointed to by the Ln segments and adding the Ln-1 segments to the fingerprint hashtable and to the walk-vector data structure; (4) repeating 1-3 for all levels of the tree structure that include Ln segments; and (5) comparing the fingerprint hashtable with the portion of the fingerprint index table corresponding to the index range to identify any missing Ln segments.