Smart NIC Erasure Coding for Faster DFS Data Rebuilds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed file systems (DFS) face limitations in improving rebuilding performance despite reduced storage capacity overhead, as erasure coding and parity data management are typically handled by the host processor, leading to increased overhead and latency.
Innovation Solution
A storage node equipped with a smart Network Interface Card (NIC) performs erasure coding and forward error correction decoding independently of the host processor, directly communicating with the storage device via PCIe switch, allowing for accelerated data recovery and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If erasure coding and parity data management are handled by the host processor, then data protection and storage efficiency are improved, but system overhead and latency increase
Solution Approach 1:
The patent extracts the erasure coding and parity data management functions from the host processor and relocates them to a dedicated smart NIC. This separation allows the host processor to focus on higher-level operations while the smart NIC handles the computationally intensive coding/decoding tasks, thereby reducing system overhead and latency while maintaining data protection capabilities.
Solution Approach 2:
The smart NIC acts as an intermediary between the host system and storage devices, performing erasure coding operations locally. This intermediary device reduces the communication overhead and processing latency by handling parity data generation and reconstruction operations closer to the storage layer, rather than requiring constant host processor intervention.
2Reliability
If erasure coding operations are performed by the host processor, then data recovery capability is improved, but rebuilding performance is limited
Solution Approach 1:
The patent extracts the erasure coding/decoding functionality from the host processor and implements it in dedicated hardware on the smart NIC. This hardware acceleration enables parallel processing of multiple coding/decoding operations simultaneously, dramatically improving rebuilding performance while preserving full data recovery capability through the same erasure coding mechanisms.
3Quantity of substance
If storage capacity overhead is reduced through erasure coding, then storage efficiency is improved, but rebuilding performance improvement is limited
Solution Approach 1:
The patent replaces the software-based erasure coding implementation with hardware-based processing on the smart NIC. This substitution of mechanical/software processing with dedicated hardware acceleration enables faster parity data generation and reconstruction operations, thereby improving rebuilding performance while maintaining the storage capacity overhead reduction benefits of erasure coding.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are a storage node and method. The storage node includes a storage device, and a smart NIC, where the smart NIC is configured to receive a write request including original data to be stored in the DFS, and responsive to the receiving, generate parity data by performing erasure coding on the original data, and store the parity data and the original data directly from the smart NIC to the storage device, and receive, from a client device or another client device, a read request for data stored in the DFS, where the smart NIC is further configured to, responsive to receiving the read request, fetch a portion of the original data and the parity data directly from the storage device, and recover another portion of the original data by applying forward error correction (FEC) decoding to the fetched portion of the original data using the fetched parity data.