MinHash File Union Estimation for Deduplication at Low Compute Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating the union size of data files in deduplication systems are resource-intensive, with high time and memory complexities, especially when dealing with large numbers of data segments, making them inefficient and costly.
Innovation Solution
The MinHash technique is employed to estimate the Jaccard similarity coefficient between data files, allowing for a more efficient calculation of the union size by using a smaller number of hash functions and reducing the need for extensive memory and processing resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If brute force algorithm is used to estimate union size by sorting and comparing all fingerprints, then measurement precision is improved, but time complexity increases to O(N^2) or O(N log N) and memory consumption increases
Solution Approach 1:
The patent extracts only the essential information needed for union size estimation by using MinHash to compute a compact signature (k minimum hash values) for each data file, rather than processing all N fingerprints. This extraction reduces the problem from comparing N fingerprints to comparing k signature values, achieving O(k) time complexity while maintaining acceptable estimation accuracy.
Solution Approach 2:
The patent changes the parameter representation from storing all N fingerprint values to storing only k MinHash signature values (where k << N). This parameter transformation reduces both time complexity from O(N^2) to O(k) and space complexity, while the Jaccard similarity estimation remains valid through the MinHash mathematical framework.
2Quantity of substance
If physical capacity reporting algorithm is used with bloom filters, then memory usage is reduced, but measurement precision deteriorates due to false positives
Solution Approach 1:
The patent uses MinHash signatures as a compact copy or representation of the full fingerprint sets. Instead of storing all N fingerprints or using large bloom filters, it creates a condensed k-value signature that captures the essential similarity information, achieving both space efficiency and accuracy without bloom filter false positives.
3Productivity
If data files are divided into many small segments for deduplication, then deduplication efficiency is improved, but the number of fingerprints increases making union size computation more complex
Solution Approach 1:
The patent applies segmentation at two levels: first, data files are segmented into chunks for deduplication (existing practice), and second, the MinHash signature itself is segmented into k independent hash values. This dual segmentation allows efficient processing of fine-grained segments while the aggregated MinHash signature provides a coarse-level summary that simplifies union size computation across all segments.
Data Source
AI summary
Identify first file's minimum value from at least value from applying function to first file's segment and value from applying function to first file's second segment. Identify first file's second minimum value from at least value from applying second function to first file's segment and value from applying second function to first file's second segment. Identify second file's minimum value from at least value from applying function to second file's segment and value from applying function to second file's second segment. Identify second file's second minimum value from at least value from applying second function to second file's segment and value from applying second function to second file's second segment. Estimate first and second files' union size based on whether first file's minimum value equals second file's minimum value, whether first file's second minimum value equals second file's second minimum value, first file's size, and second file's size.