DPU Data Identifier Deduplication for Backend-Free Write Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional networking and computing systems face issues with duplicative data writes, leading to increased network bandwidth utilization due to the need to read duplicate data from backend storage during deduplication processes.
Innovation Solution
A data processing unit (DPU) with a deduplication engine that identifies and prevents duplicate data writes by comparing data identifiers without accessing the backend storage, using secure hashing algorithms and deduplication parameters to determine unique data for transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional deduplication processes read data from backend storage to identify duplicates, then duplicate detection can be performed, but network bandwidth utilization increases
Solution Approach 1:
The patent extracts only the essential identifying characteristics of data (hash values, metadata, fingerprints) rather than transferring the actual data content. The deduplication engine compares these extracted identifiers against stored records to detect duplicates, eliminating the need to read or transmit full data blocks over the network while maintaining accurate duplicate detection capability.
Solution Approach 2:
The system performs preliminary hashing and identifier generation at the source before data transmission. By pre-computing data identifiers and comparing them against the deduplication database beforehand, the system identifies duplicates in advance, preventing unnecessary data transmission and reducing network bandwidth consumption while ensuring reliable duplicate detection.
2Speed
If data identifiers are stored in a separate data repository, then access speed improves, but system complexity increases
Solution Approach 1:
The patent segments the storage system into two distinct components: a data repository for storing actual data blocks and a separate deduplication database for storing data identifiers. This segmentation allows independent optimization of each component, enabling fast identifier access through specialized data structures in the deduplication database while maintaining the integrity and capacity of the data repository, with the trade-off of increased architectural complexity.
Data Source
AI summary
Methods, systems, devices, and computer program products for preventing duplicative data writes in networking applications are provided. An example data processing unit (DPU) including a data deduplication engine coupled to a non-transitory storage device including at least a processor receives, from an initiating device, a request for a data write operation that includes data identifiers indicative of data entries associated with the data write operation. The data deduplication engine determines a destination write location for the data write operation that is associated with one or more deduplication parameters. The data deduplication engine then accesses one or more data identifiers indicative of data entries stored by the destination write location and precludes writing of duplicate data entries to the destination write location based on a comparison between the data identifiers of the data write operation and the data identifiers indicative of data entries stored by the destination write location.