Duplicate File Detection via Metadata Hash Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating storage reports in large file systems is time-consuming and resource-intensive due to the need for scanning extensive storage volumes, which is inefficient and burdensome on input/output and processor resources.
Innovation Solution
A method and system that efficiently detect duplicate files by performing a series of fast file system metadata scans to eliminate non-duplicates and identify actual duplicates, using hash values to determine wasted space and filter out non-duplicate files, thereby optimizing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a full scan of storage volumes is performed to generate storage reports, then complete file information is obtained, but the process consumes significant time and processor resources
Solution Approach 1:
The patent segments the storage scanning process into multiple passes: a first pass that scans only file system metadata to compute hash values and identify potential duplicates, and a second pass that scans only the subset of files identified as potential duplicates. This segmentation reduces the overall scanning workload and time required while maintaining accurate duplicate detection.
Solution Approach 2:
The patent performs preliminary action by conducting a first metadata scan to compute hash values and identify potential duplicate files before performing the actual content verification. This preliminary filtering step eliminates the need to scan and compare all files in the storage volume, significantly reducing the time and resources required for the complete duplicate detection process.
2Measurement precision
If a full scan of storage volumes is performed to generate storage reports, then complete file information is obtained, but processor resources are heavily consumed
Solution Approach 1:
The patent segments the processing workload by dividing duplicate detection into two distinct phases: metadata processing (hash computation) and content verification. By separating these tasks and executing them in sequence on different file subsets, the patent reduces peak processor resource consumption compared to a single-pass full scan approach.
Solution Approach 2:
The patent applies partial action by performing metadata scanning on all files to identify potential duplicates, but then limiting the resource-intensive content verification step to only the subset of files identified as potential duplicates. This partial verification approach maintains detection accuracy for actual duplicates while avoiding unnecessary processing of non-duplicate files.
3Measurement precision
If a full scan of storage volumes is performed to generate storage reports, then complete file information is obtained, but input/output resources are heavily consumed
Solution Approach 1:
The patent segments I/O operations into two distinct phases: a first phase that performs metadata-only reads from storage volumes to compute hash values, and a second phase that performs content reads only on the much smaller subset of potential duplicate files. This segmentation dramatically reduces total I/O resource consumption while maintaining complete audit capability.
Solution Approach 2:
The patent performs preliminary metadata scanning to identify potential duplicate files before initiating content verification. This preliminary action filters out the majority of non-duplicate files, preventing unnecessary I/O operations on those files and significantly reducing overall I/O resource consumption during the complete duplicate detection process.
Data Source
AI summary
Described is a storage reports duplicate file detector that operates by receiving file records during a first scan of file system metadata. The detector computes a hash based on attributes in the record, and maintains the hash value in association with information that indicates whether a hash value corresponds to more than one file. In one implementation, the information corresponds to the amount of space wasted by duplication. The information is used to determine which hash values correspond to groups of potentially duplicate files, and eliminate non-duplicates. A second scan locates file information for each of the potentially duplicate files, and the file information is then used to determine which groups of potentially duplicate files are actually duplicate files.