Storage Data Compression Using Similar Fingerprint Block Grouping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication technologies exhibit a poor data reduction effect due to the low probability of identical data blocks, leading to inefficient storage utilization and high costs.
Innovation Solution
A method that combines deduplication with similar compression by determining whether deduplication can be performed on a target data block, obtaining its similar fingerprint when deduplication is not possible, and grouping it with reference blocks for differential data compression, thereby improving the reduction rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If deduplication technology is used to improve storage utilization, then storage space efficiency is improved, but data reduction effect deteriorates due to low probability of identical data blocks
Solution Approach 1:
The patent changes the parameter from exact fingerprint matching to similar fingerprint matching based on similarity thresholds. Instead of requiring identical data blocks, the system calculates similarity between data blocks using hash values and compares them against a threshold, allowing flexible matching of similar content while maintaining storage efficiency.
Solution Approach 2:
The patent applies different processing strategies to different data blocks based on their similarity characteristics. Data blocks are divided into segments, and each segment is independently hashed and compared. This allows the system to identify similar regions within data blocks and apply compression selectively to those regions rather than treating entire blocks uniformly.
2Productivity
If similar compression is applied to data blocks that cannot be deduplicated, then data reduction rate is improved, but processing complexity increases
Solution Approach 1:
The patent segments data blocks into smaller units for independent processing. Each data block is divided into multiple segments, and similarity detection is performed on each segment separately. This segmentation reduces the computational complexity of comparing large data blocks while maintaining the ability to identify similar content across the entire dataset.
Solution Approach 2:
The patent introduces hash values as an intermediary mechanism to facilitate similarity detection. Instead of directly comparing large data blocks, the system first computes hash values for each block and compares these compact representations. This intermediary step significantly reduces processing complexity while maintaining accurate similarity detection capability.
3Measurement precision
If data blocks are divided into segments for similarity detection, then similarity detection accuracy is improved, but processing time increases
Solution Approach 1:
The patent divides data blocks into segments and computes hash values for each segment independently. This segmentation enables parallel processing of multiple segments simultaneously, improving detection accuracy by examining finer-grained portions of data while reducing overall processing time through parallel computation rather than sequential analysis of entire blocks.
Data Source
Figure 1~3
Figure 4~6
Figure 7~8
AI summary
This application discloses a method and an apparatus for compressing data of a storage system, a device, and a readable storage medium. The method relates to the field of storage technologies. The method includes: when a target data block is stored, determining whether deduplication can be performed on the target data block; when deduplication cannot be performed on the target data block, obtaining a similar fingerprint of the target data block; determining, based on the similar fingerprint, a combined data block group to which the target data block belongs; and performing similar compression on the target data block based on a reference block in the combined data block group. This can improve a reduction rate.