File Storage System Single Instantiation via Size-First Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing file storage systems face performance deterioration due to the need to read and calculate hash values for all files to detect duplication, leading to high overhead and slow detection of duplicated files, especially with large files.
Innovation Solution
The system performs file size comparison first to identify potential duplicates, followed by binary comparison for files with matching sizes, eliminating the need for hash value calculation across all files and reducing processing overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hash value comparison is performed for all files to detect duplication, then detection accuracy is improved, but processing time and system overhead increase significantly
Solution Approach 1:
The patent segments the file comparison process into two distinct stages: first comparing file sizes (a coarse filter), and only for files with matching sizes performing binary comparison (fine verification). This segmentation allows the system to quickly eliminate non-duplicate files based on size alone, reserving computationally intensive binary comparison only for potential duplicates, thus significantly reducing overall processing time while maintaining detection accuracy.
Solution Approach 2:
The patent performs preliminary file size comparison before conducting binary comparison. This preliminary action serves as a filtering mechanism that identifies candidate files for further inspection. By performing this simpler, faster size comparison first, the system avoids the time-consuming binary comparison of all files, instead only comparing binary data for files that pass the size filter, thereby reducing total processing time while ensuring accurate duplicate detection.
2Measurement precision
If hash value calculation is performed for all files, then duplication detection is accurate, but system performance deteriorates due to high overhead
Solution Approach 1:
The patent segments the detection process into two phases: a fast file size comparison phase that filters out most files, and a slower binary comparison phase that is applied only to files with matching sizes. This segmentation dramatically reduces the number of files requiring intensive processing, thereby maintaining high detection accuracy while significantly improving system performance and reducing overhead.
Solution Approach 2:
The patent applies partial action by performing binary comparison only on a subset of files (those with matching sizes) rather than all files. This partial application of the more resource-intensive binary comparison operation to only necessary candidates maintains sufficient detection accuracy while substantially reducing the overall computational load and improving system performance.
3Measurement precision
If all file data is read for hash value comparison, then complete file comparison is achieved, but storage I/O overhead increases
Solution Approach 1:
The patent segments the data access process into two stages: first accessing only file size metadata (a small amount of data), and only for files with matching sizes reading the actual file data for binary comparison. This segmentation minimizes the total amount of data that needs to be read from storage, reducing I/O overhead while maintaining complete comparison capability for potential duplicates.
Solution Approach 2:
The patent performs preliminary retrieval of file size information before reading actual file data. This preliminary action filters out files that cannot be duplicates based on size alone, thereby reducing the amount of subsequent data reading required for binary comparison. This approach maintains comparison completeness for candidate files while significantly reducing overall storage I/O overhead.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In file de-duplication using hash value comparison, hash values of all target files must be calculated and actual data of all files must be read for hash value calculation, so that the processing time was long. The present invention provides a file storage system comprising a controller and a volume storing a plurality of files, the volume including a first directory storing a first file and a second file and a second directory storing a third file being created, wherein the controller migrates actual data of the second file to the third file, sets up a management information of the second file so that the third file is referred to when the second file is read, and if the sizes of actual data of the first file and the actual data of the third file are identical and the binaries of the actual data of the first file and the actual data of the third file are identical, sets up a management information of the first file to refer to the third file when reading the first file.