In-Storage Hashing Engine for Data Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale computing systems, data deduplication processes are hindered by the time and resource-consuming signature index look-up operations, which degrade the overall speed performance due to the need for frequent hash function calculations and Bloom filter checks.
Innovation Solution
Implementing a hashing engine within a data storage device controller that performs rolling hash functions, chunk signature calculations, and Bloom filter operations, allowing for in-storage data deduplication processes that reduce host computational workload, memory capacity, and data traffic, while enabling concurrent data chunking, signature calculation, and delta compression for similarity-based deduplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If signature index look-up operations are performed frequently to ensure accurate data deduplication, then data deduplication accuracy is improved, but system speed performance deteriorates due to time and resource consumption
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing signatures of data chunks in a signature index before actual deduplication operations. This allows the system to quickly retrieve and compare signatures during deduplication without performing complex hash calculations on-the-fly, thus maintaining accuracy while improving speed.
Solution Approach 2:
The patent introduces a signature index as an intermediary data structure that mediates between the data chunks and the deduplication logic. Instead of directly comparing raw data chunks, the system uses pre-computed signatures as intermediaries, which significantly reduces the computational overhead of look-up operations while preserving deduplication accuracy.
2Reliability
If hash function calculations are performed extensively to identify duplicate chunks, then data deduplication completeness is improved, but computational resource consumption increases
Solution Approach 1:
The patent extracts the computationally intensive hash calculation function from the main processing flow and implements it as a dedicated hashing engine within the storage device controller. This separation allows the host system to offload computational tasks, reducing overall resource consumption while maintaining complete deduplication coverage.
Solution Approach 2:
The storage device controller's hashing engine performs hash calculations autonomously without requiring host system intervention. The controller services its own computational needs by implementing the hash function locally, enabling the system to maintain complete deduplication while reducing the host's computational burden.
3Productivity
If Bloom filter checks are performed for every data chunk to eliminate unnecessary look-ups, then data transfer efficiency is improved, but processing time increases due to multiple hash function applications
Solution Approach 1:
The patent merges the Bloom filter check functionality with the signature index look-up operation. Instead of performing separate Bloom filter checks and then full signature index look-ups, the system combines these operations into a unified process that leverages the hashing engine to serve both purposes simultaneously, reducing overall processing time while maintaining transfer efficiency.
Data Source
AI summary
A data storage system having a hashing engine for facilitating data deduplication. A device is provided that includes: a storage media; and a controller, wherein the controller includes a hashing engine for implementing a data deduplication process on data stored in the storage medium, wherein the hashing engine: inputs parameters from a host that specifies a sliding widow size and a boundary condition; implements a rolling hash function; and outputs a data chunk boundary.