NIC-Based Data Deduplication for Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed data storage systems, existing technologies face challenges in efficiently managing data deduplication, leading to increased network bandwidth consumption and latency due to the need for redundant data transmission and fingerprint generation across network storage entities.
Innovation Solution
Implementing a network interface controller (NIC) that segments data into chunks, generates fingerprints, and performs deduplication by comparing these fingerprints with existing stored data, thereby reducing the need for transmitting non-unique data chunks and shifting fingerprint identification and storage responsibilities to network storage, thus conserving network bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is performed using existing network storage entities, then data storage is achieved, but network bandwidth consumption increases due to redundant data transmission
Solution Approach 1:
The NIC performs preliminary fingerprint generation and deduplication checking before data is transmitted over the network. By segmenting incoming data into chunks, generating fingerprints, and checking for duplicates at the source, the system avoids transmitting redundant data chunks, thus reducing network bandwidth consumption while maintaining storage capacity
Solution Approach 2:
The invention extracts the fingerprint generation and deduplication functionality from the network storage entity and relocates it to the NIC. This extraction allows the storage system to receive only unique data chunks, eliminating the waste of network bandwidth on redundant transmissions while preserving the ability to store and retrieve data
2Measurement precision
If fingerprint generation is performed across network storage entities, then data uniqueness is verified, but latency increases due to distributed processing
Solution Approach 1:
The invention merges the fingerprint generation function into the NIC, combining data reception, segmentation, and fingerprint generation into a single location. This eliminates the need for data to traverse the network for fingerprint generation, reducing latency while maintaining precise uniqueness verification through the fingerprint matching process
Solution Approach 2:
Fingerprints are generated preliminarily at the NIC before data transmission, and the storage system performs preliminary checking against stored fingerprints. This preliminary action at the source eliminates the need for iterative distributed verification, reducing latency while ensuring accurate uniqueness determination
3Reliability
If all data chunks are transmitted to storage for deduplication checking, then complete deduplication is achieved, but network bandwidth is wasted transmitting non-unique chunks
Solution Approach 1:
The invention extracts only the necessary information (fingerprints and metadata) for deduplication checking and transmits it to the storage system, rather than transmitting all raw data chunks. The storage system uses these fingerprints to identify unique chunks and requests only those from the NIC, ensuring complete deduplication while minimizing network bandwidth usage
Solution Approach 2:
The NIC acts as an intermediary that generates fingerprints from incoming data and communicates only these fingerprints and metadata to the storage system. This intermediary role allows the system to achieve complete deduplication verification while transmitting minimal data over the network, as the fingerprint serves as a compact representation of the original data
Data Source
AI summary
Methods and systems for managing storage of data in a distributed system is disclosed. To manage storage of data in a distributed system, a data processing system may include a network interface controller (NIC). The NIC may present emulated storages that may be used for data storage. The emulated storage devices may utilize storage resources of storage devices. The NIC and the storage devices may implement a distributed deduplication process. The NIC may segment data into chunks and obtain fingerprints of the chunks. The fingerprints may be provided to the storage which may check the fingerprints against fingerprints of already stored chunks. The storage may request the chunks corresponding to the fingerprints that did not match any fingerprints of the already stored chunks. The NIC may provide only those requested chunks to the storage before discarding all of the chunks.