Hash-Based Hardware Compression for Low-Latency Data Deduplication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing lossless data compression techniques are not well-suited for high-speed, low-latency applications and are inefficient in handling data fragmentation caused by thin provisioning and sparse mapping, leading to performance penalties in storage systems.

Innovation Solution

A hardware-accelerated lossless data compression system using hash memories and control logic to identify and compress duplicate data blocks, combined with deduplication and compression methods that group data into chunks and use Bloom filters for efficient storage and retrieval.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of substance

If software-based lossless compression techniques are used, then compression efficiency is improved, but processing speed and latency are worsened

Engineering Contradiction:

Improvedata sizeVSAvoidprocessing speed

Core Design Contradiction:

Loss of substanceVSSpeed

Solution Approach 1:

The patent replaces software-based compression algorithms with a hardware-based compression engine that uses dedicated circuits for hashing, comparison, and compression operations. This mechanical/electrical substitution enables parallel processing of multiple data blocks simultaneously, achieving both high compression efficiency and low latency by performing compression operations in hardware rather than through sequential software execution.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The compression engine is divided into multiple functional units including hash computation units, comparison units, and compression units that can operate in parallel. Data is processed in discrete blocks that can be handled independently by different hardware units simultaneously, increasing overall processing throughput while maintaining compression efficiency.

Inventive Principle:

Principle #1Segmentation

2Device complexity

If hardware-based compression processes one byte at a time, then hardware implementation simplicity is improved, but processing throughput is worsened

Engineering Contradiction:

Improvehardware complexityVSAvoiddata throughput

Core Design Contradiction:

Device complexityVSProductivity

Solution Approach 1:

The patent segments data into fixed-size blocks (e.g., 512 bytes or larger) that are processed independently and in parallel by multiple hardware units. Each block is hashed and compared separately, allowing the system to process multiple blocks simultaneously through different pipeline stages, thereby achieving high throughput without requiring overly complex hardware design.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary hashing of data blocks before comparison operations. By pre-computing hash values and organizing them in hash tables, the system prepares data for rapid comparison and matching, enabling high-speed processing of large data blocks without increasing hardware complexity during the actual compression phase.

Inventive Principle:

Principle #10Preliminary action

3Speed

If other hardware implementations process multiple bytes at one time, then processing speed is improved, but compression efficiency is worsened

Engineering Contradiction:

Improveprocessing speedVSAvoidcompression ratio

Core Design Contradiction:

SpeedVSLoss of substance

Solution Approach 1:

The patent processes data in optimized block sizes that may be larger or smaller than traditional byte boundaries, depending on the specific application requirements and data characteristics. This partial processing approach allows the system to achieve high throughput by processing multiple bytes in parallel while maintaining good compression ratios by adjusting block sizes to match actual data patterns and redundancy levels.

Inventive Principle:

Principle #16Partial or excessive action

4Speed

If caching is used for frequently accessed data, then access speed is improved, but performance deteriorates with fragmented data distribution

Engineering Contradiction:

Improvedata access speedVSAvoidcache efficiency

Core Design Contradiction:

SpeedVSProductivity

Solution Approach 1:

The patent introduces a deduplication layer with hash-based indexing as an intermediary between the storage system and the caching mechanism. This intermediary identifies and eliminates duplicate data blocks before they are cached, ensuring that only unique data is stored in cache memory. This approach maintains high cache efficiency even with fragmented data distribution by preventing redundant cache entries and reducing cache pollution from duplicate fragments.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS8694703B2Hardware-accelerated lossless data compression

Publication Date: 2014.04.08 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LTD

AI summary

Systems for hardware-accelerated lossless data compression are described. At least some embodiments include data compression apparatus that includes a plurality of hash memories each associated with a different lane of a plurality of lanes (each lane including data bytes of a data unit being received by the compression apparatus), an array including array elements each including a plurality of validity bits (each validity bit within an array element corresponding to a different lane of the plurality of lanes), control logic that initiates a read of a hash memory entry if a corresponding validity bit indicates that said entry is valid, and an encoder that compresses at least the data bytes for the lane associated with the hash memory comprising the valid entry if said valid entry comprises data that matches the lane data bytes.