Host Side Deduplication for Network Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Server-side deduplication in network storage environments leads to significant network bandwidth consumption and processing burdens due to the transfer of redundant data across the network, as storage servers handle deduplication for multiple host computing devices.

Innovation Solution

Implementing host-side deduplication, where the host computing device performs deduplication before sending data to the storage server, using techniques such as hash functions to identify redundant data and sending deduplication notifications instead of redundant data, thereby offloading processing from the storage server and reducing unnecessary network traffic.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If storage server performs deduplication for multiple host computing devices, then storage capacity utilization is improved, but network bandwidth consumption and processing load increase significantly

Engineering Contradiction:

Improvestorage capacity utilizationVSAvoidnetwork bandwidth consumption

Core Design Contradiction:

Quantity of substanceVSLoss of energy

Solution Approach 1:

The host computing device performs deduplication locally before transmitting data to the storage server. By computing hashes of data blocks and comparing them with previously stored hashes locally, the host identifies redundant data in advance and sends only deduplication notifications for data that needs to be stored, rather than transmitting all data including duplicates.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts the deduplication processing function from the storage server and relocates it to the host computing device. This separation allows the storage server to receive only essential information (deduplication notifications) rather than full data streams, significantly reducing network bandwidth consumption while maintaining storage capacity utilization.

Inventive Principle:

Principle #2Taking out (Extraction)

2Quantity of substance

If storage server performs deduplication processing, then redundant data storage is reduced, but processing load and time consumption increase

Engineering Contradiction:

Improveredundant data storageVSAvoidprocessing time

Core Design Contradiction:

Quantity of substanceVSLoss of time

Solution Approach 1:

The host computing device performs deduplication processing locally before data transmission to the storage server. By computing hashes of data blocks and comparing with previously stored hashes in advance, the host identifies redundant data beforehand, eliminating the need for the storage server to perform time-consuming deduplication processing on all incoming data.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts the computationally intensive deduplication processing from the storage server and relocates it to the host computing device. This distribution of processing tasks reduces the storage server's processing load and time consumption while achieving the same goal of reducing redundant data storage.

Inventive Principle:

Principle #2Taking out (Extraction)

3Extent of automation

If all data is transmitted to storage server for deduplication, then storage server can manage deduplication centrally, but network traffic and processing burden increase

Engineering Contradiction:

Improvecentralized deduplication managementVSAvoidnetwork traffic and processing burden

Core Design Contradiction:

Extent of automationVSDevice complexity

Solution Approach 1:

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The host computing device autonomously performs deduplication processing locally using stored hash information. The host computes hashes of incoming data blocks, compares them with previously stored hashes, and automatically determines whether data is redundant without requiring continuous centralized management or processing from the storage server.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10459649B2Host side deduplication

Publication Date: 2019.10.29 NETAPP INC

AI summary

One or more techniques and/or systems are provided for performing host side deduplication. Host side deduplication may be performed upon writeable data within a write request received at a host computing device configured to access data stored by a storage server. The host side deduplication may be performed at the host computing device to determine whether the writeable data is already stored by the storage server based upon querying a host side cache comprising data stored by a storage server and/or a data structure comprising unique signatures of data stored by the storage server. If the writeable data is stored by the storage server, then a deduplication notification excluding the writeable data may be sent to the storage server, otherwise a write command comprising the writeable data may be sent. Accordingly, unnecessary network traffic of redundant data already stored by the storage server may be reduced.