DPU Deduplication Engine for Duplicate Write Prevention

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional networking and computing systems face issues with duplicative data writes, leading to increased network bandwidth utilization due to the need to read duplicate data from backend storage during deduplication processes.

Innovation Solution

A data processing unit (DPU) with a deduplication engine that identifies and prevents duplicate data writes by comparing data identifiers without accessing the backend storage, using secure hash algorithms to transmit only unique data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional deduplication processes read duplicate data from backend storage to identify duplicates, then duplicate detection can be performed, but network bandwidth utilization increases due to the need to transmit duplicate data identifiers

Engineering Contradiction:

Improveduplicate detection accuracyVSAvoidnetwork bandwidth consumption

Core Design Contradiction:

ReliabilityVSLoss of energy

Solution Approach 1:

The system performs preliminary deduplication checking by comparing data identifiers (such as hashes) against the data repository before actual data transmission occurs. This preliminary action identifies duplicate data blocks in advance, allowing the system to prevent transmission of duplicate data identifiers over the network, thereby reducing network bandwidth consumption while maintaining accurate duplicate detection

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The invention extracts only the essential identifying information (data identifiers such as hashes) from the data blocks and stores them in a separate data repository. This extraction allows the system to perform deduplication checks by comparing identifiers rather than transmitting actual data blocks, significantly reducing network bandwidth usage while preserving the ability to accurately detect duplicates

Inventive Principle:

Principle #2Taking out (Extraction)

2Speed

If data identifiers are stored locally in the DPU, then access speed improves, but device complexity increases

Engineering Contradiction:

Improvedata identifier access speedVSAvoidDPU storage structure complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The system separates the data identifier storage from the data block storage by introducing a distinct data repository dimension. This dimensional separation allows data identifiers to be stored locally in the DPU for fast access while maintaining a structured relationship with the backend storage system, achieving both speed improvement and manageable complexity through organized data architecture

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS20260003836A1Methods, systems, and devices for preventing duplicative data writes

Publication Date: 2026.01.01 MELLANOX TECHNOLOGIES LTD(IL)

AI summary

Methods, systems, devices, and computer program products for preventing duplicative data writes in networking applications are provided. An example data processing unit (DPU) including a data deduplication engine coupled to a non-transitory storage device including at least a processor receives, from an initiating device, a request for a data write operation. The data write operation is associated with a destination write location and includes data identifiers indicative of data entries for writing to the destination write location. The data deduplication engine then accesses one or more data identifiers indicative of data entries stored by the destination write location and precludes writing of duplicate data entries to the destination write location based on a comparison between the one or more data identifiers of the data write operation and the one or more data identifiers indicative of data entries stored by the destination write location.