Duplication-Count Grouping for Fast Deduplication Estimates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating deduplication rates in large data sets are time-consuming, requiring significant time to perform actual deduplication, such as over 100 days for 1 PB of data.
Innovation Solution
An information processing device that classifies chunks into groups based on duplication counts and uses different estimation methods for each group, employing a first calculation unit for minor chunks and a second calculation unit for major chunks to estimate deduplication rates accurately in a short time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If actual deduplication is performed to estimate the deduplication rate, then the estimation accuracy is improved, but the time required increases significantly
Solution Approach 1:
The patent divides the data set into multiple chunks and processes them in parallel. The deduplication estimation is performed by segmenting the large data set into smaller manageable units (chunks), calculating deduplication rates for each chunk independently, and then aggregating the results. This segmentation enables faster processing while maintaining overall estimation accuracy.
Solution Approach 2:
The patent calculates deduplication rates for only a portion of the data (specific chunks) rather than performing complete deduplication on the entire data set. By selecting representative chunks and calculating their deduplication rates, the system obtains a sufficiently accurate estimation without the time cost of processing all data, thus applying partial action to resolve the time-accuracy tradeoff.
2Measurement precision
If complete deduplication is performed on large data sets, then the deduplication rate is accurately determined, but the calculation cost and processing time increase
Solution Approach 1:
The patent segments the large data set into multiple chunks and processes them in parallel to improve processing speed. By dividing the work into smaller units that can be handled concurrently, the system maintains accuracy while significantly reducing overall processing time and calculation costs.
Solution Approach 2:
The patent performs deduplication calculations on a partial subset of chunks rather than completing full deduplication on the entire data set. This partial action approach provides sufficiently accurate results for estimation purposes while dramatically reducing the computational resources and time required.
Data Source
Figure 1A~1B
Figure 2
Figure 3
AI summary
An information processing device includes a management unit, a classification unit, a first calculation unit, a second calculation unit, and a synthesis unit. The management unit extracts part of data from estimation target data as a plurality of pieces of sample data and manage a number of duplications of the extracted plurality of pieces of sample data. The classification unit classifies the plurality of pieces of sample data into a first group and a second group based on the number of duplications. The first calculation unit calculates a first deduplication rate for sample data classified into the first group. The second calculation unit calculates a second deduplication rate for sample data classified into the second group. The synthesis unit calculates a deduplication rate of the estimation target data based on the first deduplication rate and the second deduplication rate.