Storage System Backup De-duplication Using Segmentation and Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current storage system de-duplication techniques are inefficient in identifying and removing redundant data, particularly at the sub-file level, and fail to consider image data, leading to increased backup times and storage space usage.
Innovation Solution
A hybrid approach combining file-level and sub-file-level de-duplication using natural language processing (NLP) to convert files into bag-of-words representations, clustering, and minhash techniques to identify and store unique data segments, reducing the dimensionality of data and efficiently eliminating duplicates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sub-file level de-duplication is performed to improve duplicate detection accuracy, then de-duplication performance improves, but backup time increases significantly
Solution Approach 1:
The patent divides files into fixed-size data segments (e.g., 4KB chunks) and applies de-duplication at the segment level rather than file level. This segmentation enables identification of duplicate portions within files while maintaining efficient processing through independent segment comparison.
Solution Approach 2:
The patent introduces data segment hashes as an intermediary representation to compare file contents. Instead of directly comparing large file data, the system computes hashes of data segments and compares these compact representations, significantly reducing computation time while maintaining detection accuracy.
2Quantity of substance
If traditional de-duplication methods are used to reduce storage space, then storage efficiency improves, but computing resource consumption increases
Solution Approach 1:
The patent performs preliminary hashing of data segments during the backup process itself, rather than requiring separate de-duplication passes. This preliminary action integrates duplicate detection into the normal backup workflow, avoiding additional computing overhead.
Solution Approach 2:
The patent uses disposable hash values as temporary representations of data segments. These hash objects are computed quickly, used for comparison, and then discarded, replacing the need for expensive and persistent full data segment comparisons.
3Productivity
If file level de-duplication is performed to reduce backup time, then processing speed improves, but duplicate detection accuracy decreases
Solution Approach 1:
By segmenting files into fixed-size chunks, the system maintains fast processing through independent segment handling while improving detection accuracy by identifying duplicates at the granular segment level rather than requiring complete file matches.
Solution Approach 2:
The patent applies de-duplication to more granular portions (data segments) than traditional file-level approaches, performing partial de-duplication within files. This excessive action on smaller units achieves both speed and accuracy by allowing selective duplication removal.
Data Source
AI summary
Techniques are provided for storage system backup and de-duplication. In an example, a file system stores a plurality of files. These files can be grouped into smaller sets through a series of iterations, where a set of files is similar to each other. Then, files of a set of files can be directly compared for duplicate information relative to each other, one copy of this duplicate information can be stored, and other copies can be discarded.