Dedupe Ingest Compressibility Hints to Cut Wasted CPU Cycles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplication systems incur unnecessary CPU cycles due to compressing and decompressing data that is already compressed or has a low compression ratio, resulting in minimal storage space savings and increased processing overhead.
Innovation Solution
Generating compressibility hints for unique data segments to determine whether compression is beneficial, allowing the system to store segments as either compressed or uncompressed, thereby reducing CPU cycles and metadata overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the dedupe system compresses all unique segments to save storage space, then storage space is reduced, but CPU cycles are unnecessarily consumed for segments that are already compressed or have low compressibility
Solution Approach 1:
The system performs preliminary compressibility assessment on unique segments before committing to compression. By evaluating compressibility hints in advance, the system avoids unnecessary CPU cycles spent compressing segments that are already compressed or have low compressibility, while still compressing segments that would benefit from compression.
Solution Approach 2:
The system applies different compression strategies to different segments based on their individual compressibility characteristics. Instead of uniformly compressing all segments, the system selectively compresses only those segments that are determined to be compressible, leaving already-compressed or low-compressibility segments in their original form.
2Quantity of substance
If the dedupe system compresses segments to reduce data transmission, then network I/O is reduced, but CPU cycles are wasted on compressing non-compressible data
Solution Approach 1:
The system performs preliminary compressibility assessment before data transmission compression. By evaluating compressibility hints in advance, the system identifies which segments would actually benefit from compression and only compresses those segments, avoiding wasted CPU processing time on non-compressible data while still reducing network I/O for compressible segments.
3Quantity of substance
If the dedupe system compresses segments for storage efficiency, then storage capacity is optimized, but CPU cycles are consumed for segments with low compression ratio
Solution Approach 1:
The system applies different storage strategies to different segments based on their compressibility. Compressible segments are compressed to optimize storage capacity, while non-compressible or already-compressed segments are stored without additional compression, preserving CPU processing efficiency.
Solution Approach 2:
The system performs preliminary compressibility assessment to identify which segments are suitable for compression before storage. This advance evaluation ensures that CPU resources are only spent compressing segments that will actually achieve compression ratios that justify the processing effort.
Data Source
AI summary
A method, system and computer-readable storage medium for transferring data segments from one computer system to a second computing system. Prior to transfer of the data segments, the first system calculates compressibility ratio of each segment and compares the compressibility ratio to a preset threshold. Based on the comparison, the first system assigns a compressibility hint to each segment. The first system transfers the segments to the second system, together with the corresponding compressibility hint. The second system stores each segment in a compressible region or in a non-compressible region based on the hint. Then the second system compresses the compressible region and stores the compressed region in a container, and stores the non-compressible region uncompressed in the container.