DPU Erasure Coding Across Fault Domains for Storage Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale cloud-based data centers, general-purpose processors are inefficient for high-capacity network and storage workloads, leading to poor performance in packet stream processing, and storage systems often become unavailable due to hardware or software errors, requiring effective data durability solutions.
Innovation Solution
A programmable data processing unit with specialized hardware accelerators is used to implement data durability coding across multiple fault domains, enabling efficient recovery of data with low latency and reduced storage requirements through techniques like erasure coding and matrix-based approaches, offloading operations from servers to free them for other tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated across multiple storage systems for durability, then data reliability is improved, but storage overhead and cost increase significantly
Solution Approach 1:
The patent segments data into multiple fragments using erasure coding algorithms (e.g., breaking data into 65-byte chunks and distributing them across multiple storage systems). This allows the original data to be reconstructed from any subset of fragments, reducing the need for full replication while maintaining durability across multiple failure domains.
Solution Approach 2:
The patent changes the storage parameter from full replication (100% overhead) to erasure coding ratios (e.g., 50% or less overhead by storing n data fragments and m parity fragments). This parameter change optimizes the balance between reliability and storage efficiency, allowing data recovery with significantly reduced storage requirements.
2Productivity
If general-purpose processors are used for packet stream processing, then system simplicity is maintained, but processing performance and efficiency deteriorate
Solution Approach 1:
The patent introduces data processing units (DPUs) as intermediary devices between storage systems and servers. These DPUs specialize in network packet processing and durability operations, offloading these functions from general-purpose server processors. This intermediary layer improves overall system performance while allowing servers to focus on application workloads.
Solution Approach 2:
The patent replaces general-purpose software-based processing with specialized hardware accelerators within DPUs. These hardware components provide dedicated circuitry for packet processing, encryption, compression, and erasure coding operations, significantly improving processing throughput and efficiency compared to software implementations on general-purpose CPUs.
3Reliability
If data fragments are stored across multiple fault domains, then data availability is improved, but network complexity and coordination overhead increase
Solution Approach 1:
The patent creates a universal durability layer implemented in DPUs that handles erasure coding, fragment distribution, and recovery operations across multiple fault domains. This multi-functional durability layer provides a standardized interface for storing and recovering data fragments across diverse storage systems and network configurations, simplifying the coordination complexity while maintaining high availability across fault domains.
Data Source
AI summary
This disclosure describes a programmable device, referred to generally as a data processing unit, having multiple processing units for processing streams of information, such as network packets or storage packets. This disclosure also describes techniques that include enabling data durability coding on a network. In some examples, such techniques may involve storing data in fragments across multiple fault domains in a manner that enables efficient recovery of the data using only a subset of the data. Further, this disclosure describes techniques that include applying a unified approach to implementing a variety of durability coding schemes. In some examples, such techniques may involve implementing each of a plurality of durability coding and/or erasure coding schemes using a common matrix approach, and storing, for each durability and/or erasure coding scheme, an appropriate set of matrix coefficients.