Sampling-Based Deduplication Estimation Using Histogram Minimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Estimating deduplication ratios in large datasets is challenging due to the need for extensive processing, which requires large memory and computing resources, and sampling methods struggle to accurately identify global repetitions across vast data ranges.
Innovation Solution
A method that partitions a dataset into logical units, selects a random sample based on a sampling ratio, calculates hash values, computes histograms for duplication counts and frequencies, derives a target function to minimize distance between sampling and observed histograms, and determines a deduplication ratio using an optimal histogram.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire dataset is processed to accurately estimate deduplication ratio, then measurement precision is improved, but device complexity and processing resources are increased
Solution Approach 1:
The patent divides the large dataset into multiple smaller chunks or blocks, processing each segment independently through hashing and frequency counting. This segmentation allows the system to estimate deduplication ratio across the entire dataset by aggregating results from smaller, manageable portions, thereby reducing immediate processing resource requirements while maintaining estimation accuracy.
Solution Approach 2:
Instead of processing the entire dataset, the patent applies sampling techniques where a representative subset of data is analyzed to infer characteristics of the whole. By carefully selecting and analyzing a partial sample, the system achieves sufficient measurement precision for deduplication ratio estimation without the full computational burden of processing every data element.
2Device complexity
If sampling method is used to reduce processing resources, then device complexity is reduced, but measurement precision deteriorates
Solution Approach 1:
The patent implements feedback mechanisms where the results from sampled data are used to adjust and refine the estimation process. By continuously comparing sampled results against expected patterns and using this feedback to optimize sampling strategies, the system improves measurement precision while maintaining reduced processing complexity.
Solution Approach 2:
The patent dynamically adjusts sampling parameters such as sample size, chunk size, and sampling frequency based on initial analysis results and dataset characteristics. By optimizing these parameters, the system achieves sufficient measurement precision with minimal sampling, thereby reducing processing resource requirements while maintaining accuracy.
3Productivity
If random sampling is applied to estimate deduplication potential, then productivity is improved, but measurement precision may deteriorate due to sampling variability
Solution Approach 1:
The patent performs preliminary analysis on the sampled data to establish baseline characteristics and distribution patterns before final estimation. This preliminary action allows the system to optimize subsequent sampling and processing steps, improving both productivity and measurement precision by avoiding redundant analysis and focusing computational resources on the most informative data elements.
Data Source
AI summary
A method, including partitioning a dataset into a first number of data units, and selecting, based on a sampling ratio, a second number of the data units. A hash value is calculated for each of the selected data units, and a first histogram is computed indicating a first duplication count for each of the calculated hash values. Based on respective frequencies of the calculated hash values, a second histogram is computed indicating an observed frequency for each of the first duplication counts in the first histogram, and based on the sampling ratio and the second histogram, a target function is derived. A third histogram that minimizes the target function is derived, the third histogram including, for the first number of the storage units, second duplication counts and a respective predicted frequency for each of the second duplication counts. Finally, a deduplication ratio is determined based on the third histogram.