MinHash Error Bounds for Deduplicated File Size Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication systems face challenges in accurately estimating the size of deduplicated files due to inherent errors in determining file similarity using MinHash, which affects chargeback calculations in data storage systems.
Innovation Solution
The system employs MinHash to estimate the Jaccard similarity coefficient by averaging multiple hash functions, reducing variance and providing an unbiased estimator for file union size calculations, thereby improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If MinHash is used to estimate file similarity in deduplication systems, then computation efficiency is improved, but measurement precision deteriorates due to inherent error factors
Solution Approach 1:
The patent segments the file similarity estimation process into multiple independent hash function computations. Instead of using a single MinHash value, the system divides the estimation into k separate hash function evaluations, each providing an independent sample. This segmentation allows the system to maintain computational efficiency while reducing estimation error through aggregation of multiple samples.
Solution Approach 2:
The patent changes the parameter k (number of hash functions) to control the trade-off between computation efficiency and measurement precision. By adjusting k, the system can optimize performance for different scenarios: smaller k for faster computation with acceptable accuracy, and larger k for higher precision when needed. This parameter adjustment directly addresses the contradiction by providing a controllable balance point.
2Measurement precision
If multiple hash functions are used to reduce estimation error, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent merges multiple independent hash function results into a single aggregated estimate. Instead of maintaining separate complex estimation systems for each hash function, the system combines their results through simple averaging or aggregation operations. This merging approach reduces overall system complexity while achieving improved measurement precision through the combined information from multiple hash functions.
3Measurement precision
If accurate file size calculation is performed in deduplication systems, then chargeback accuracy is improved, but computation time increases
Solution Approach 1:
The patent applies partial action by using a limited number of hash functions (k) rather than exhaustively analyzing all possible file similarities. This partial approach provides sufficient accuracy for chargeback purposes without requiring the excessive computation time of complete analysis. The system performs just enough computation to achieve acceptable chargeback accuracy, balancing precision and time requirements.
Data Source
AI summary
A system and method for an estimation of error bounds for file size calculations using MinHash in deduplication systems. The system includes one or more processors to determine a similarity score between the first file and the second file. The one or more processors are further to determine a size estimation of a combination of the first and second files based on the similarity score. Finally, the one or more processors are to determine a maximum error for the size estimation of the combination of the first and second files, wherein the first and second file are to be combined via deduplication and have at least one shared data segment.