Erasure Coding Layout for Distributed Storage Rebuild Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure coding techniques in distributed storage systems require an exponential number of blocks per storage node, making them impractical for real-world applications, as they need to optimize bandwidth usage.
Innovation Solution
The proposed solution involves a mirroring scheme that distributes data chunks among non-volatile memory nodes, using a systematic and parity node configuration where each data chunk has a role based on its position, and includes summation data chunks that are functions of both systematic and extra data chunks from other rows, reducing the number of storage nodes required for optimal erasure coding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional erasure coding techniques are used in distributed storage systems, then data redundancy and error correction are achieved, but the number of storage nodes required increases exponentially
Solution Approach 1:
The patent segments data into chunks and organizes them in a hierarchical structure with systematic nodes and parity nodes. Each node stores specific segments according to defined roles, allowing the system to achieve redundancy without requiring exponential growth in total nodes. The segmentation principle enables scalable erasure coding by breaking down the storage problem into manageable, organized units.
Solution Approach 2:
The patent introduces a hierarchical dimension to traditional erasure coding by organizing nodes into systematic nodes and parity nodes with different roles. This dimensional organization allows the system to optimize bandwidth usage and reduce the total number of nodes required while maintaining the same level of data protection and recovery capability.
2Productivity
If more storage nodes are added to optimize bandwidth usage, then data distribution is improved, but the rebuild ratio increases
Solution Approach 1:
The patent applies local quality by assigning different roles to different nodes based on their position and function in the hierarchy. Systematic nodes store original data chunks while parity nodes store computed parity information. This localized specialization allows efficient bandwidth utilization during normal operations while keeping the rebuild ratio manageable, as repairs can be performed using only the necessary parity nodes rather than all nodes in the system.
3Quantity of substance
If a large number of storage nodes are used, then data capacity increases, but the number of rows or stripes required increases exponentially
Solution Approach 1:
The patent creates a universal structure where parity nodes serve multiple functions: they store parity information for their associated systematic nodes, can participate in repair operations, and maintain hierarchical relationships that scale efficiently. This multi-functionality allows the system to increase data capacity by adding nodes to the hierarchical structure without requiring a proportional increase in rows or stripes, as the hierarchical organization reuses existing structures efficiently.
Data Source
AI summary
A system and method for erasure coding. The method includes distributing a plurality of data chunks according to a mirroring scheme, wherein the plurality of data chunks is distributed as a plurality of rows among a plurality of non-volatile memory (NVM) nodes, wherein the mirroring scheme defines a plurality of groups, each group including a subset of the plurality of data chunks, wherein each data chunk in a group has a role corresponding to a relative position of the data chunk within the group, wherein data chunks included in the plurality of groups having the same relative positions within their respective groups have the same role, wherein each row of the plurality of rows includes at least one summation data chunk that is a function of at least one data chunk included in the row and of at least one extra data chunk included in at least one other row.