Network Storage Reliability Coding With DPU Erasure Offload
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud-based data centers, general-purpose processors are inefficient in handling high-capacity network and storage workloads, leading to poor performance in packet stream processing, and storage systems often become unavailable due to hardware or software errors, requiring effective data durability solutions.
Innovation Solution
A programmable data processing unit with specialized hardware accelerators performs data durability coding by storing data in fragments across multiple fault domains, enabling efficient recovery using erasure coding schemes, and offloading data durability operations to reduce server workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If general-purpose processors are used for data durability operations, then device complexity is reduced, but processing speed and productivity deteriorate
Solution Approach 1:
The system segments processing functions by separating general-purpose processors from specialized hardware accelerators. The hardware accelerators handle specific data durability operations (erasure coding, parity generation) while general-purpose processors handle other tasks, resolving the contradiction by dedicating specialized resources to high-performance operations without increasing overall system complexity.
Solution Approach 2:
A data processing unit acts as an intermediary component between storage systems and general-purpose processors. This intermediary contains specialized hardware accelerators that perform data durability operations, allowing the system to achieve high processing throughput without requiring complex modifications to general-purpose processor architecture.
2Reliability
If data is stored in fragments across multiple fault domains, then data reliability is improved, but storage overhead and device complexity increase
Solution Approach 1:
Data is segmented into fragments and distributed across multiple fault domains (different storage devices, racks, or data centers). This segmentation improves reliability by ensuring that data can be recovered even if some domains fail, while the automated coding schemes manage the complexity of distribution and retrieval.
Solution Approach 2:
The system uses erasure coding schemes that transform data into encoded fragments with specific mathematical relationships. By changing the representation of data through encoding parameters, the system achieves improved reliability without proportionally increasing storage overhead, as redundant information is generated efficiently through mathematical transformations.
3Quantity of substance
If erasure coding schemes are implemented, then storage efficiency is improved, but processing complexity and time increase
Solution Approach 1:
The patent replaces software-based erasure coding implementations with hardware-based accelerators that perform coding operations. This substitution reduces processing complexity and time by utilizing dedicated hardware circuits optimized for mathematical operations required by erasure coding schemes, while maintaining storage efficiency benefits.
Solution Approach 2:
The system generates parity fragments as copies of encoded data that can be stored separately. These parity copies enable data recovery without requiring complex real-time computation during failure scenarios, as the pre-computed parity information can be directly applied to reconstruct lost data fragments.
4Device complexity
If hosts perform data durability operations, then device complexity is reduced, but server productivity and availability deteriorate
Solution Approach 1:
Data durability operations are extracted from host servers and delegated to specialized data processing units. This extraction allows servers to focus on their primary computational tasks while the dedicated units handle encoding, decoding, and recovery operations, thereby improving server productivity without reducing storage system functionality.
Solution Approach 2:
Data processing units serve as intermediary components between hosts and storage systems, handling data durability operations transparently. This intermediary layer offloads processing requirements from servers, improving their availability and productivity while maintaining the complexity benefits of centralized storage management.
Data Source
AI summary
This disclosure describes a programmable device, referred to generally as a data processing unit, having multiple processing units for processing streams of information, such as network packets or storage packets. This disclosure also describes techniques that include enabling data durability coding on a network. In some examples, such techniques may involve storing data in fragments across multiple fault domains in a manner that enables efficient recovery of the data using only a subset of the data. Further, this disclosure describes techniques that include applying a unified approach to implementing a variety of durability coding schemes. In some examples, such techniques may involve implementing each of a plurality of durability coding and/or erasure coding schemes using a common matrix approach, and storing, for each durability and/or erasure coding scheme, an appropriate set of matrix coefficients.