Hardware-Accelerated Data Deduplication for Storage Efficiency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing data backup solutions face inefficiencies in maximizing disk storage due to redundant data, leading to increased costs for storage, power, cooling, and management, as traditional compression and deduplication methods are processor-intensive and do not effectively utilize disk space.

Innovation Solution

A system utilizing a dedicated hardware card to generate hashes for data blocks, discard duplicates, and compress unique blocks for storage, leveraging cryptographic hash functions and compression algorithms to optimize storage capacity, with the option to perform deduplication at both file and block levels.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data compression is used to maximize storage usage, then storage capacity is improved, but processing speed decreases and processor intensity increases

Engineering Contradiction:

Improvestorage capacityVSAvoidprocessing speed

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent replaces software-based compression with hardware-accelerated compression using dedicated compression cards or processors. This substitution of the processing mechanism enables compression to achieve nominal ratios while maintaining acceptable processing speeds, resolving the contradiction between storage capacity improvement and processing speed degradation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Quantity of substance

If post-processing deduplication is used, then storage capacity is reduced, but storage benefits are defeated due to temporary storage requirements

Engineering Contradiction:

Improvestorage capacityVSAvoidtemporary storage time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The patent implements in-line deduplication where hash computation and duplicate detection occur during the data write process itself, before data reaches permanent storage. This preliminary action eliminates the need for temporary storage of full backups, as duplicates are identified and rejected in real-time during the write operation.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces a hash table as an intermediary data structure that maps data hashes to their first occurrence locations. This intermediary enables efficient duplicate detection during in-line processing without requiring temporary storage of complete backup data, resolving the contradiction between storage capacity reduction and temporary storage requirements.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Quantity of substance

If in-line deduplication is used, then storage benefits are maximized, but processor intensity increases

Engineering Contradiction:

Improvestorage efficiencyVSAvoidprocessor intensity

Core Design Contradiction:

Quantity of substanceVSUse of energy by moving object

Solution Approach 1:

The patent replaces general-purpose processor-based hashing with dedicated hardware hash computation units. This substitution reduces the processing burden on system processors while maintaining in-line deduplication capabilities, thereby achieving storage efficiency without excessive processor intensity.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS8205065B2System and method for data deduplication

Publication Date: 2012.06.19 EXAR CORP

AI summary

A system for deduplicating data comprises a card operable to receive at least one data block and a processor on the card that generates a hash for each data block. The system further comprises a first module that determines a processing status for the hash and a second module that discards duplicate hashes and their data blocks and writes unique hashes and their data blocks to a computer readable medium. In one embodiment, the processor also compresses each data block using a compression algorithm.