Pseudo Fingerprint Chunk Boundary Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication methods fail to optimize chunk boundaries in compound storage objects, leading to incomplete elimination of redundant data, as they do not recognize artificial chunk boundaries within compound files.
Innovation Solution
The method involves determining chunk boundaries by calculating hash values, creating pseudo fingerprints for artificial chunk boundaries, and storing them in a pseudo fingerprint list to recognize and eliminate redundant data efficiently during deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional fingerprint-based deduplication is used without pseudo fingerprints, then the deduplication process is simpler, but redundant data within compound files cannot be fully eliminated
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing pseudo fingerprints for artificial chunk boundaries during the initial deduplication process. This allows the system to quickly recognize and handle compound file structures in subsequent deduplication operations without re-processing the entire file hierarchy, thereby improving efficiency while managing complexity through advance preparation
Solution Approach 2:
The patent introduces pseudo fingerprints as an intermediary mechanism that bridges traditional fingerprint-based deduplication and compound file handling. These pseudo fingerprints act as mediators that enable the deduplication engine to recognize artificial chunk boundaries and properly process nested file structures without requiring fundamental changes to the core deduplication algorithm
2Quantity of substance
If artificial chunk boundaries are recognized in compound files, then storage utilization is optimized, but the deduplication process becomes more complex
Solution Approach 1:
The patent applies segmentation by dividing compound files into distinct chunks based on artificial chunk boundaries identified through pseudo fingerprint matching. This segmentation allows the deduplication system to process and store individual file components separately, optimizing storage capacity by eliminating redundancy at the chunk level while maintaining manageable processing complexity through structured division
3Productivity
If pseudo fingerprints are stored for every artificial chunk boundary, then redundant data elimination is maximized, but memory usage increases
Solution Approach 1:
The patent applies local quality by selectively storing pseudo fingerprints only at artificial chunk boundaries within compound files, rather than uniformly across all data structures. This targeted approach maximizes redundant data elimination where it is most needed (at file boundaries within compounds) while minimizing memory consumption by avoiding unnecessary pseudo fingerprint storage in other areas
Data Source
AI summary
Described are a method, system, and computer program product for deduplicating a storage object. A hash of a window of data of a storage object is determined and a determination is made as to whether the window of data of the storage object corresponds to a chunk boundary. A determination is made as to whether the hash of the object matches one pseudo fingerprints in a list of at least one pseudo fingerprint. A storage object chunk boundary based on the window of data is stored in response to the window of data corresponding to the chunk boundary or in response to determining that the hash of the object matches one of the pseudo fingerprints. A determination is made of a new window of data in the storage object following the window of data when the window of data is not an end of data of the storage object.