Structured Erasure Coding for Reliable Low-Overhead Data Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage technologies face challenges in ensuring data storage reliability while minimizing computing resource overhead, with multi-copy technology leading to high storage costs and erasure coding being computationally complex.
Innovation Solution
A data processing method using a first generator matrix to encode source data blocks, reducing encoding computation by employing a structured matrix that allows for efficient data recovery through XOR operations, thereby maintaining reliability and reducing resource overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multi-copy technology is used to ensure data reliability, then data storage reliability is improved, but storage space utilization deteriorates and storage costs increase
Solution Approach 1:
The patent changes the mathematical parameters of the encoding system by using a specially structured generator matrix with elements from a finite field GF(q), where q is a prime number or prime power. This parameter change enables the system to achieve the same reliability as multi-copy technology while using less storage space through more efficient encoding.
Solution Approach 2:
The patent applies local quality by making different parts of the encoding system have different properties. The generator matrix has a specific structure where certain elements are from finite fields while others follow different rules, allowing the system to optimize both reliability and storage efficiency in different parts of the encoding process.
2Quantity of substance
If erasure coding technology is used to improve storage space utilization, then storage efficiency is improved, but encoding and decoding complexity increases and computing resource overhead increases
Solution Approach 1:
The patent simplifies the computational complexity by changing the algebraic parameters from general finite field arithmetic to a specific structure where the generator matrix elements are constrained to GF(q). This parameter constraint enables the use of simpler XOR-based operations instead of complex multiplication operations in finite fields.
Solution Approach 2:
The patent replaces complex computational operations with simpler, cheaper operations. By using a structured generator matrix, the system can use simple XOR operations instead of expensive finite field multiplication operations, effectively using 'cheap' computational primitives to achieve the same encoding and decoding functionality.
3Reliability
If traditional erasure coding is used to ensure data reliability, then data storage reliability is improved, but computing resource overhead increases due to complex multiplication operations in finite fields
Solution Approach 1:
The patent substitutes complex mechanical-like computational operations (finite field multiplication) with simpler operations (XOR-based arithmetic). This substitution replaces the 'mechanical' process of complex multiplication with a more efficient algebraic process using properties of finite fields and structured matrices, reducing computing resource overhead.
Solution Approach 2:
The patent changes the computational parameters by constraining the generator matrix elements to a specific finite field GF(q) structure. This parameter change enables the use of more efficient arithmetic operations that require fewer computational resources while maintaining the same reliability guarantees.
Data Source
Figure 1
Figure 2~4
Figure 5~7
AI summary
The embodiments of the present application relate to the technical field of communications. Provided are a data processing method, an apparatus and a storage medium, which are used for ensuring the reliability of data storage and reducing the computing resource overheads. The method comprises: acquiring k source data blocks, wherein k is a positive integer; and encoding the k source data blocks according to a first generation matrix to obtain n first encoded data blocks, wherein the value range of elements in the first generation matrix is [0, q-1], and q is a prime number or the power of the prime number.