Hardware-Hash Storage Device Deferred Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data deduplication schemes in storage systems face performance issues due to CPU-intensive hashing operations and excessive data traffic, leading to reduced throughput and increased workload on servers.
Innovation Solution
A storage system with a hardware-hash-enabled storage device that offloads CPU-intensive hashing operations and buffers data, allowing the deduplication process to be deferred until the server is available, reducing traffic and enhancing performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data deduplication is performed using software-based hashing on the server, then deduplication functionality is achieved, but CPU-intensive operations reduce system performance and increase server workload
Solution Approach 1:
The patent extracts the hashing function from the server's CPU and implements it as a dedicated hardware module within the storage device. This separation removes the computationally intensive hashing operations from the server's processing path, allowing the server to focus on data management tasks while the hardware module handles fingerprint generation independently and in parallel.
Solution Approach 2:
The patent introduces a hardware-based intermediary module that sits between the data write path and the server's deduplication logic. This hardware module acts as a mediator by pre-computing hashes and providing fingerprints to the server, thereby reducing the server's computational burden and improving overall system throughput.
2Reliability
If all incoming data is written to storage before deduplication processing, then data availability is maintained, but excessive data traffic increases network bandwidth consumption
Solution Approach 1:
The patent performs preliminary hashing operations at the hardware level within the storage device, generating fingerprints before data is fully written to the storage medium. This preliminary action allows the system to identify and eliminate duplicates early in the data ingestion process, reducing the volume of data that needs to be transmitted over the network and stored in the first place.
3Measurement precision
If duplicate data is stored temporarily before deduplication, then deduplication accuracy is improved, but storage capacity is wasted on redundant copies
Solution Approach 1:
The patent implements a self-service mechanism where the hardware module within the storage device autonomously computes hashes and identifies duplicates without requiring the server to retrieve and re-process the data. This self-service approach maintains deduplication accuracy by performing comparisons at the point of data ingress, while simultaneously optimizing storage capacity by eliminating duplicates before they consume permanent storage resources.
Data Source
AI summary
A storage system may include at least one storage device and a server. The storage device may store an incoming data, calculate a hash value for the incoming data, and store the hash value as meta data. The server may provide the incoming data to the storage device, read the meta data from the storage device, determine whether the meta data is duplicate in the table, and perform a deduplication process regarding the incoming data based on the determination result.