FPGA NIC Data Replication via Inline De-duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods in storage devices, such as the DATA DOMAIN product, require CPU-intensive operations like de-duplication, which consume significant resources and result in lower performance per unit power, and dedicated hardware solutions lack flexibility due to hardware changes difficulties.
Innovation Solution
Implementing a field programmable gate array (FPGA)-based network interface card that segments data, determines fingerprints using the SHA1 algorithm, and compares these with existing fingerprints to de-duplicate data, reducing CPU load and optimizing cost and power performance by performing operations like data segmentation and compression in the network interface card.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CPU-based de-duplication operations are used, then data replication can be performed, but CPU resource utilization increases and performance per unit power decreases
Solution Approach 1:
The patent extracts de-duplication operations from the CPU and implements them in hardware within the network interface card. Specifically, the NIC includes a de-duplication module that performs fingerprint calculation and comparison operations independently of the CPU, thereby reducing CPU resource utilization and power consumption while maintaining data replication capability.
Solution Approach 2:
The patent introduces a fingerprint as an intermediary representation of data segments. Instead of comparing entire data segments directly, the system calculates fingerprints (hash values) of data segments and compares these compact representations. This intermediary approach significantly reduces the computational burden on the CPU and enables efficient de-duplication operations.
2Use of energy by moving object
If dedicated hardware (ASIC) is used for data replication operations, then performance per unit power improves, but design flexibility and adaptability are dramatically restrained
Solution Approach 1:
The patent implements de-duplication operations in the network interface card using programmable logic rather than fixed ASIC hardware. This allows the de-duplication algorithm and parameters to be dynamically adjusted and updated through software configuration, maintaining hardware-level performance while preserving design flexibility and adaptability.
Solution Approach 2:
The network interface card is designed with multi-functionality, serving both as a standard network interface and as a de-duplication processing unit. The NIC's de-duplication module can handle various data replication scenarios and workloads, providing universal functionality that combines the benefits of dedicated hardware performance with the flexibility of a general-purpose device.
3Device complexity
If all de-duplication operations are performed by CPU, then implementation is simple, but operation speed and processing efficiency are limited
Solution Approach 1:
The patent segments the data processing pipeline by separating de-duplication operations from the main CPU processing flow. The network interface card independently handles data segment reception, fingerprint calculation, and de-duplication comparison, while the CPU focuses on higher-level replication management. This segmentation enables parallel processing and significantly improves overall processing speed.
Data Source
AI summary
There is provided a method and device for data replication. The method comprises: obtaining, in a network interface card, data segments by segmenting input first data; determining, in the network interface card, fingerprints corresponding to the data segments; and comparing, in a central processing unit, the fingerprints of the data segments with existing fingerprints corresponding to processed data segments, and determining, based on a result of the comparing, whether to de-duplicate the data segments corresponding to the fingerprints, to perform the data replication.