Segmented Fingerprint Generation for Deduplication Collision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As the amount of data managed by deduplication storage systems increases, the likelihood of identical fingerprints being generated for different data segments increases, leading to potential fingerprint collisions, and computing checksums to mitigate this issue is computationally expensive and resource-intensive.
Innovation Solution
The system generates an effective fingerprint by processing each unit of a data chunk using a fingerprint generator, saving the internal state before processing the final unit, reloading it to compute a base fingerprint, and then generating a derivative fingerprint from the transformed final unit, combining both to create a stronger and less computationally intensive fingerprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If checksum computation is used to identify data segments, then fingerprint collision risk is reduced, but computational resources and processing time increase significantly
Solution Approach 1:
The patent segments the fingerprint generation process into multiple independent components: base fingerprint computation from processed data units, and derivative fingerprint computation from transformed final units. This segmentation allows parallel processing and reduces the computational burden compared to traditional checksum methods while maintaining collision resistance through the combination of multiple fingerprint elements.
Solution Approach 2:
The patent performs preliminary processing of data units through the fingerprint generation algorithm before final fingerprint computation. Internal state is saved and reused to generate both base and derivative fingerprints, eliminating the need for redundant full-data processing. This preliminary action reduces computational overhead while ensuring fingerprint uniqueness.
2Productivity
If traditional fingerprint generation is used, then processing speed is maintained, but fingerprint collision risk increases with large data volumes
Solution Approach 1:
The patent merges base fingerprints and derivative fingerprints into a combined effective fingerprint. This combination maintains the processing efficiency of traditional fingerprint generation while significantly reducing collision probability, as the effective fingerprint space becomes the product of individual fingerprint spaces rather than a single limited space.
Solution Approach 2:
The patent introduces an intermediary transformation process that operates on the final data units to generate derivative fingerprints. This intermediary step acts as a mediator between the base fingerprint generation and the final effective fingerprint, adding uniqueness assurance without requiring complete reprocessing of all data units, thus maintaining processing speed.
Data Source
AI summary
One embodiment provides a computer implemented method of data identification within a deduplication storage system, the method comprising processing multiple units of a segment of data within the deduplication storage system using a fingerprint generation algorithm; storing the internal state generated while processing the multiple units of the segment of data; generating a first fingerprint for the segment of data based on the internal state; reloading the internal state after generating the first fingerprint for the segment of data; and generating a second fingerprint for the segment of data based on a transformed unit of the segment of data.