Realigned Data Page Fingerprinting for Leading-Zero Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication storage systems struggle to efficiently store data pages with varying leading zero pads, leading to increased storage requirements due to non-matching fingerprints.
Innovation Solution
A deduplication method that involves realigning data pages by removing the leading zero pad and appending it to the end, generating a fingerprint for comparison, and storing the realigned page if unique, allowing deduplication of data pages with different leading zero pad lengths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data pages with varying leading zero pads are stored as-is in a deduplication storage system, then the system maintains simple storage operations, but storage space requirements increase due to non-matching fingerprints
Solution Approach 1:
The system performs preliminary realignment of data pages by removing leading zero pads and appending them to the end before generating fingerprints. This preliminary action ensures that data pages with varying leading zero pads are normalized to a common format, allowing their fingerprints to match and enabling deduplication, thereby reducing storage space requirements
Solution Approach 2:
The system changes the parameter of data page alignment by repositioning the leading zero pads from the beginning to the end of the data page. This parameter change transforms the data page structure, enabling fingerprints generated from realigned pages to match between different data pages that originally had different leading zero pad lengths
2Productivity
If data pages are realigned by removing leading zero pads and appending to end, then deduplication efficiency improves, but processing time increases
Solution Approach 1:
The realignment operation is performed as a preliminary action before fingerprint generation and comparison. By normalizing the data page structure upfront, the system enables efficient fingerprint matching without requiring complex comparisons during the deduplication process, thus improving overall deduplication efficiency despite the additional processing step
Data Source
AI summary
Example implementations relate to deduplication operations in a storage system. An example implementation includes receiving a data page to be stored in a persistent storage of a deduplication storage system, and, in response to a determination that the data page includes the leading zero pad, performing a circular shift of the plurality of data blocks to generate a realigned data page that begins at a non-zero value portion. The example implementation further includes generating a fingerprint for the realigned data page, and comparing the generated fingerprint to a plurality of fingerprints included in metadata of the deduplication storage system. The example implementation further includes, in response to a determination that no matches are found between the generated fingerprint and the plurality of fingerprints included in the metadata, storing the realigned data page and the generated fingerprint in the persistent storage.