Erasure Coding Parallelism for Storage Availability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional erasure coding systems face challenges in scalability and efficiency, particularly in large-scale distributed storage systems, where computing complexity increases linearly, making it difficult to achieve high data availability with reduced cost and efficient data recovery.
Innovation Solution
The implementation of optimized software erasure coding and decoding calculations using SIMD, cache-friendly improvements, instruction-level parallelism, and minimum spanning tree optimizations, along with parallel data recovery mechanisms, enhances the efficiency of Cauchy Reed-Solomon erasure coding, enabling high-performance data storage and recovery at unprecedented scales like '80+20' coding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional erasure coding (e.g., 8+2) is implemented in enterprise storage systems, then data redundancy and fault tolerance are provided, but the system cannot guarantee 100% data availability when multiple servers fail and the computing complexity remains manageable
Solution Approach 1:
The patent segments the erasure coding process into independent parallel operations across multiple storage nodes. Each node performs localized encoding/decoding operations on its own data blocks rather than requiring centralized computation, dividing the overall computing complexity into manageable distributed units that can operate independently
Solution Approach 2:
The patent transitions from traditional small-scale erasure coding (8+2) to large-scale distributed erasure coding (80+20) by adding a dimensional aspect to the system architecture. This involves organizing storage across hundreds of nodes with hierarchical data distribution, where data is first divided into chunks and then further distributed across the network, creating a multi-dimensional storage topology that enables both high availability and manageable complexity
2Reliability
If erasure coding scale is increased from 8+2 to 80+20 to guarantee 100% data availability, then data availability is improved, but computing complexity increases linearly
Solution Approach 1:
The patent implements self-service mechanisms where each storage node autonomously performs encoding and decoding operations on its local data blocks without requiring centralized coordination. Nodes independently calculate parity blocks and reconstruct lost data using only local computations and peer-to-peer communication, eliminating the need for complex centralized processing that would scale linearly with system size
Solution Approach 2:
The patent performs preliminary encoding operations during data write operations, distributing encoded data blocks across storage nodes before any potential failures occur. This advance preparation ensures that parity information is already distributed and accessible, eliminating the need for complex real-time computation during failure recovery scenarios and improving both availability and computational efficiency
3Reliability
If three-replica approach is used to store data copies, then data availability is provided, but the cost triples and data may become unavailable when more than two servers fail in a 100-node system
Solution Approach 1:
The patent fundamentally changes the redundancy parameter from fixed three-replica copying to flexible erasure coding ratios (e.g., 80+20). This allows the system to achieve the same or better data availability guarantees with significantly reduced storage overhead by mathematically distributing redundancy across hundreds of nodes rather than duplicating entire data sets across three nodes
Solution Approach 2:
The patent implements a universal erasure coding framework that can operate at any scale (from small clusters to hundreds of nodes) with a single unified approach. The same encoding algorithms and data distribution mechanisms work whether the system has 10 nodes or 1000 nodes, eliminating the need for different replication strategies at different scales and enabling consistent cost-effective redundancy across the entire distributed storage infrastructure
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A server configured to form part of a data protection group, wherein the server comprises a memory, a transceiver configured to receive a request to store a data file from another device, and a processor coupled to the memory and the transceiver and configured to acquire the request to store a data file, write the data file to a container in the memory, and encode the container contents when the container becomes full to generate a plurality of encoded blocks, wherein the transceiver is further configured to distribute the plurality of encoded blocks over peer servers in the data protection group.