Hybrid Data Reduction Using Ordered Chunk Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data deduplication techniques are inefficient in maximizing data reduction due to the high CPU cost of comparing similar but non-identical data blocks, leading to increased latency and degraded storage system performance.
Innovation Solution
A two-tier method combining data deduplication and compression, where deduplication is performed first to identify unique data chunks, followed by using weighted graphs to determine an optimal compression order for these chunks, maximizing the data compression ratio through graph theory techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If data deduplication is performed by comparing fixed-size data chunks using brute force byte-by-byte comparison, then data reduction is achieved, but CPU time consumption increases and system performance degrades
Solution Approach 1:
The patent applies preliminary action by performing deduplication operations before compression. The system first identifies and eliminates duplicate data chunks, then compresses the remaining unique data. This sequencing optimizes overall data reduction while managing computational resources efficiently, as deduplication reduces the data volume that subsequently requires compression processing.
Solution Approach 2:
The patent segments data into fixed-size chunks for processing. By dividing the data stream into manageable blocks, the system can apply deduplication and compression algorithms to individual segments independently. This segmentation enables efficient memory management and allows parallel processing of multiple chunks, improving overall system throughput while maintaining data reduction effectiveness.
2Loss of substance
If every data block is compared with other data blocks in a brute force manner, then complete deduplication is achieved, but latency of host I/O data increases
Solution Approach 1:
The system performs deduplication as a preliminary step before compression and storage operations. By identifying and removing duplicates early in the data processing pipeline, the system reduces the amount of data that requires subsequent I/O operations, thereby decreasing overall I/O latency despite the computational overhead of comparison.
Solution Approach 2:
Data is divided into fixed-size chunks that can be processed independently. This segmentation allows the system to compare and process smaller units of data rather than entire data blocks, reducing the time required for each comparison operation and enabling more efficient parallel processing to minimize overall latency.
3Ease of manufacture
If data compression is performed without optimizing the order of data chunks, then compression is simpler to implement, but compression ratio is suboptimal
Solution Approach 1:
The system performs deduplication as a preliminary action before compression. By removing duplicate chunks first, the remaining unique data exhibits better compressibility characteristics. This preliminary processing step enhances the effectiveness of subsequent compression algorithms, achieving higher compression ratios without significantly complicating the overall implementation.
Solution Approach 2:
The data is segmented into fixed-size chunks that are processed individually through deduplication and then compressed. This segmentation approach maintains implementation simplicity by allowing each chunk to be handled independently, while still achieving improved compression ratios through the systematic processing of segmented data units.
Data Source
AI summary
An information handling system may include at least one processor and a memory coupled to the at least one processor. The information handling system may be configured to receive data comprising a plurality of data chunks; perform deduplication on the plurality of data chunks to produce a plurality of unique data chunks; determine a compression ratio for respective pairs of the unique data chunks; determine a desired compression order for the plurality of unique data chunks based on the compression ratios; combine the plurality of unique data chunks in the desired compression order; and perform data compression on the combined plurality of unique data chunks.