Storage Compression Using Similarity Sketches Across Data Chunks

Solution Overview

Problem

Conventional data compression techniques in storage systems are limited by their reliance on small back-reference windows, which restrict the identification and elimination of redundant data, leading to suboptimal compression efficiency and increased computational overhead.

Innovation Solution

The proposed solution involves partitioning data into chunks, generating similarity sketches, and reorganizing these chunks to group similar data regions together, allowing for more effective compression by standard compressors, while preserving metadata for reconstruction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional compression algorithms use small back-reference windows, then computation speed is maintained, but compression efficiency deteriorates due to inability to identify distant redundancies

Engineering Contradiction:
Core Design Contradiction:
ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent segments the data stream into fixed-size blocks and creates separate hash indexes for each block. This allows the compression system to efficiently search for redundancies across block boundaries without requiring a single large computational window, thereby improving compression efficiency while maintaining manageable computation levels.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary hashing and index creation for each data block before the actual compression process. By pre-computing hash values and organizing them in indexes, the system prepares redundancy information in advance, enabling faster compression execution without excessive real-time computation overhead.

Inventive Principle:
Principle #10Preliminary action

2Loss of substance

If larger redundancy windows are used to find more compression, then data reduction improves, but computation time and memory requirements increase

Engineering Contradiction:
Core Design Contradiction:
Loss of substanceVSDevice complexity

Solution Approach 1:

The patent divides the data into sequential blocks and creates compact hash indexes for each block. This segmentation allows the system to achieve effective data reduction by searching within and across blocks without requiring a single large memory window, thus reducing overall memory overhead while maintaining compression effectiveness.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent uses hash values as compact copies or representations of the actual data blocks. Instead of storing entire block contents in memory for comparison, the system stores only hash signatures, dramatically reducing memory requirements while enabling efficient redundancy detection across large data ranges.

Inventive Principle:
Principle #26Copying

3Quantity of substance

If delta compression is applied to similar data chunks, then storage requirements decrease, but system complexity increases due to sketch generation and matching overhead

Engineering Contradiction:
Core Design Contradiction:
Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent uses hash sketches as compact copies or fingerprints of data blocks. Instead of storing complete block data for comparison, the system stores only hash representations, which dramatically reduces storage requirements while enabling efficient similarity detection through simple hash matching operations.

Inventive Principle:
Principle #26Copying

Solution Approach 2:

The patent transforms the complex problem of data similarity comparison into a simpler parameter-based matching problem by using hash values. This parameter transformation changes the comparison operation from examining entire data blocks to comparing compact hash signatures, reducing system complexity while maintaining storage efficiency.

Inventive Principle:
Principle #35Parameter changes

Data Source

PatentUS9767154B1System and method for improving data compression of a storage system in an online manner
Publication Date: 2017.09.19
  • US9767154B1 patent drawing
  • US9767154B1 patent drawing
  • US9767154B1 patent drawing

AI summary

Techniques for improving data compression of a storage system in an online manner are described herein. According to one embodiment, in response to a sequence of data to be stored, the sequence of data is partitioned into a plurality of data chunks according to a predetermined chunking algorithm. A sketch for each of the data chunks is generated based on one or more features extracted from the data chunk. Each of the data chunks of the sequence of data is associated with one of a plurality of groups based on the sketch, wherein each group is represented by a sketch. The data chunks of each group are compressed and stored in a compression region of the storage systems, such that similar data chunks are compressed and stored in the same compression region.