Storage Compression Using Similarity Sketches Across Data Chunks
Solution Overview
Problem
Conventional data compression techniques in storage systems are limited by their reliance on small back-reference windows, which restrict the identification and elimination of redundant data, leading to suboptimal compression efficiency and increased computational overhead.
Innovation Solution
The proposed solution involves partitioning data into chunks, generating similarity sketches, and reorganizing these chunks to group similar data regions together, allowing for more effective compression by standard compressors, while preserving metadata for reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional compression algorithms use small back-reference windows, then computation speed is maintained, but compression efficiency deteriorates due to inability to identify distant redundancies
Solution Approach 1:
The patent segments the data stream into fixed-size blocks and creates separate hash indexes for each block. This allows the compression system to efficiently search for redundancies across block boundaries without requiring a single large computational window, thereby improving compression efficiency while maintaining manageable computation levels.
Solution Approach 2:
The patent performs preliminary hashing and index creation for each data block before the actual compression process. By pre-computing hash values and organizing them in indexes, the system prepares redundancy information in advance, enabling faster compression execution without excessive real-time computation overhead.
2Loss of substance
If larger redundancy windows are used to find more compression, then data reduction improves, but computation time and memory requirements increase
Solution Approach 1:
The patent divides the data into sequential blocks and creates compact hash indexes for each block. This segmentation allows the system to achieve effective data reduction by searching within and across blocks without requiring a single large memory window, thus reducing overall memory overhead while maintaining compression effectiveness.
Solution Approach 2:
The patent uses hash values as compact copies or representations of the actual data blocks. Instead of storing entire block contents in memory for comparison, the system stores only hash signatures, dramatically reducing memory requirements while enabling efficient redundancy detection across large data ranges.
3Quantity of substance
If delta compression is applied to similar data chunks, then storage requirements decrease, but system complexity increases due to sketch generation and matching overhead
Solution Approach 1:
The patent uses hash sketches as compact copies or fingerprints of data blocks. Instead of storing complete block data for comparison, the system stores only hash representations, which dramatically reduces storage requirements while enabling efficient similarity detection through simple hash matching operations.
Solution Approach 2:
The patent transforms the complex problem of data similarity comparison into a simpler parameter-based matching problem by using hash values. This parameter transformation changes the comparison operation from examining entire data blocks to comparing compact hash signatures, reducing system complexity while maintaining storage efficiency.
Data Source
AI summary
Techniques for improving data compression of a storage system in an online manner are described herein. According to one embodiment, in response to a sequence of data to be stored, the sequence of data is partitioned into a plurality of data chunks according to a predetermined chunking algorithm. A sketch for each of the data chunks is generated based on one or more features extracted from the data chunk. Each of the data chunks of the sequence of data is associated with one of a plurality of groups based on the sketch, wherein each group is represented by a sketch. The data chunks of each group are compressed and stored in a compression region of the storage systems, such that similar data chunks are compressed and stored in the same compression region.



