Inline Durability Coding Across Fault Domains for Network Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud-based data centers, general-purpose processors are inefficient in handling high-capacity network and storage workloads, leading to poor performance in packet stream processing, and storage systems often become unavailable due to hardware or software errors, requiring effective data durability solutions.
Innovation Solution
A programmable data processing unit with specialized hardware accelerators is used to implement data durability coding across multiple fault domains, enabling efficient recovery of data with low latency and reduced storage requirements through techniques like erasure coding and matrix-based approaches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose processors are used to process network and storage data, then devices can handle multiple functions, but processing performance for high-capacity workloads deteriorates
Solution Approach 1:
The system segments processing functions by introducing specialized data processing units (DPUs) that handle network and storage workloads separately from general-purpose servers. This segmentation allows each component to be optimized for its specific function, with DPUs dedicated to packet stream processing and servers focused on application logic.
Solution Approach 2:
Data processing units act as intermediaries between storage systems and the network fabric. These DPUs offload durability operations, packet processing, and data movement tasks from general-purpose processors, enabling servers to maintain versatility while DPUs provide specialized high-performance processing.
2Reliability
If data is replicated across multiple fault domains for durability, then data reliability improves, but storage overhead increases
Solution Approach 1:
The system changes the parameter of data representation by using erasure coding instead of simple replication. Data is encoded into multiple fragments with mathematical relationships, allowing reconstruction from any sufficient subset. This parameter change enables achieving the same reliability with reduced storage overhead compared to full replication.
Solution Approach 2:
Instead of creating full copies of data for redundancy, the system creates encoded fragments that collectively represent the original data. Any sufficient number of fragments can be used to reconstruct the original, providing durability without requiring complete duplicate copies of the entire dataset.
3Reliability
If data fragments are stored across multiple fault domains, then resilience to failures improves, but data recovery complexity increases
Solution Approach 1:
The system implements self-service through hardware-accelerated erasure coding operations in DPUs. The specialized processing units automatically perform the complex mathematical operations required for encoding and decoding erasure codes, making the complexity transparent to users and simplifying recovery operations.
Solution Approach 2:
The system replaces software-based erasure coding implementations with hardware-accelerated operations in specialized processing units. This substitution of mechanical/software complexity with dedicated hardware circuitry maintains failure resilience while significantly reducing the operational complexity and latency of encoding and decoding operations.
4Quantity of substance
If erasure coding is implemented for data durability, then storage efficiency improves, but processing latency increases
Solution Approach 1:
The system replaces software-based erasure coding with hardware-accelerated operations in data processing units. These specialized hardware circuits perform the mathematically intensive encoding and decoding operations much faster than general-purpose processors, reducing the latency penalty associated with erasure coding while maintaining storage efficiency.
Solution Approach 2:
Data is pre-encoded into fragments and distributed across fault domains before any potential failure occurs. This preliminary encoding action, performed by hardware accelerators, ensures that when recovery is needed, the complex decoding operations can proceed quickly using pre-positioned fragments rather than requiring time-consuming real-time encoding.
Data Source
AI summary
This disclosure describes a programmable device, referred to generally as a data processing unit, having multiple processing units for processing streams of information, such as network packets or storage packets. This disclosure also describes techniques that include enabling data durability coding on a network. In some examples, such techniques may involve storing data in fragments across multiple fault domains in a manner that enables efficient recovery of the data using only a subset of the data. Further, this disclosure describes techniques that include applying a unified approach to implementing a variety of durability coding schemes. In some examples, such techniques may involve implementing each of a plurality of durability coding and/or erasure coding schemes using a common matrix approach, and storing, for each durability and/or erasure coding scheme, an appropriate set of matrix coefficients.