Fingerprint Index Sets for Faster Big Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems waste resources due to inefficient data reduction methods, particularly in large data block environments, as they require extensive fingerprint comparisons for deduplication and duplication, which takes significant time and resources.
Innovation Solution
A data reduction method using an LSM tree and/or K-V tree to form index sets based on fingerprint information to identify and group similar or identical data blocks, thereby improving efficiency by forming an index set, and performing data reduction on a data block to which the index set, thereby reducing the storage space required for data storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data reduction is performed by comparing fingerprints of all data blocks, then data deduplication accuracy is improved, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent segments the large set of all data block fingerprints into multiple groups or subsets. Instead of comparing a candidate fingerprint against all fingerprints sequentially, the system divides the fingerprint set into manageable segments, allowing parallel or staged comparison processes that reduce overall processing time while maintaining deduplication accuracy.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and organizing fingerprints into structured formats (such as hash tables, trees, or sorted lists) before the actual deduplication comparison. This preliminary organization enables faster lookup and comparison operations, reducing the time required during the actual data reduction process while preserving accuracy.
2Measurement precision
If data reduction is performed by comparing fingerprints of all data blocks, then data deduplication completeness is improved, but resource consumption increases
Solution Approach 1:
The patent segments the comprehensive fingerprint comparison task into smaller sub-tasks that can be executed with lower resource intensity. By dividing the large dataset into manageable portions, the system reduces peak memory usage, CPU load, and energy consumption while still achieving complete deduplication across all data blocks through systematic processing of segments.
Solution Approach 2:
The patent employs partial action by performing fingerprint comparisons in stages or batches rather than all at once. This allows the system to process data reduction incrementally, consuming resources at sustainable levels while ultimately achieving complete deduplication coverage across the entire dataset.
3Measurement precision
If traditional fingerprint comparison methods are used for data reduction, then deduplication accuracy is maintained, but processing speed decreases
Solution Approach 1:
The patent segments the fingerprint comparison process to enable parallel processing. By dividing the dataset into multiple segments that can be processed simultaneously across different computing units or threads, the system maintains accurate fingerprint matching while significantly improving overall data reduction throughput and efficiency.
Solution Approach 2:
The patent creates optimized copies or representations of fingerprint data in efficient data structures (such as hash tables or inverted indexes) that enable rapid lookup. These copied organizational structures allow the system to maintain deduplication accuracy while dramatically speeding up the comparison process through constant-time or logarithmic-time lookups instead of linear scanning.
Data Source
Figure 1~3
Figure 4
Figure 5
AI summary
A data reduction method, apparatus, and computing device and a storage medium are provided. A technical field to which the data reduction method, apparatus, and computing device and the storage medium belong is the field of big data technologies. The method includes: when reduction is to be performed on a to-be-reduced data block, obtaining a fingerprint of the to-be-reduced data block (201); forming an index set based on the fingerprint of the to-be-reduced data block by using index information of data blocks with identical fingerprints (202); and performing, in the to-be-reduced data block based on the fingerprint of the to-be-reduced data block, data reduction processing on a data block to which index information in a same index set belongs (203). When this method is used, data reduction efficiency can be improved.