Erasure Coding Matrix Grouping to Cut Storage Encoding Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure coding implementations in distributed storage systems require a large number of arithmetic operations, making them inefficient for large data sets and unsuitable for scalable data protection.
Innovation Solution
The method involves splitting data into fragments, using a binary coding matrix to identify and sum groups of fragments, and updating the matrix to reduce the number of arithmetic operations required for encoding and decoding, thereby accelerating the erasure coding process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional erasure coding is used to protect data in distributed storage systems, then data reliability is improved, but processing time and computational resources increase significantly
Solution Approach 1:
The patent divides the data into multiple fragments and processes them in parallel groups rather than sequentially. The coding matrix is partitioned into multiple sections, allowing simultaneous processing of different data fragments through multiple computing nodes, thereby reducing overall processing time while maintaining data protection reliability.
Solution Approach 2:
The patent performs preliminary organization of data fragments and pre-computation of coding matrix sections before the actual encoding process. By preparing data structures and computing resources in advance, the system reduces the time required during the critical encoding phase while ensuring data reliability through pre-validated coding schemes.
2Reliability
If traditional erasure coding is used to protect data in distributed storage systems, then data reliability is improved, but computational resources and processing overhead increase
Solution Approach 1:
The computing workload is segmented across multiple nodes, with each node responsible for specific fragments and matrix sections. This parallel processing approach distributes computational resources efficiently, improving overall processing efficiency while maintaining the reliability guarantees of erasure coding through coordinated computation across segments.
Solution Approach 2:
The patent combines multiple coding operations into unified matrix multiplications and merges computation results from different fragments. By consolidating redundant operations and utilizing shared computational resources, the system improves processing efficiency while maintaining comprehensive data protection coverage.
3Reliability
If erasure coding is applied to large data sets, then data protection coverage is improved, but the number of arithmetic operations increases making it unsuitable for scalable systems
Solution Approach 1:
The large coding matrix is divided into smaller manageable sections that can be processed independently. This segmentation reduces the complexity of individual arithmetic operations while maintaining comprehensive data protection coverage through the coordinated application of multiple smaller coding operations across data fragments.
Solution Approach 2:
The patent transforms the traditional single-dimension encoding approach into a multi-dimensional structure by organizing data fragments and coding matrix sections in parallel dimensions. This allows simultaneous processing across multiple dimensions, reducing the total number of sequential arithmetic operations required while maintaining full data protection coverage for large data sets.
Data Source
AI summary
A method for generating coded fragments comprises receiving data to be encoded, splitting the data into a plurality of data fragments, identifying a first group of data fragments from among the plurality of data fragments using a coding matrix, summing the data fragments within the first group of data fragments to generate a first group sum, and using the first group sum to calculate at least a portion of two or more coded fragments.