Numerical Erasure Coding for Fast Fault-Tolerant Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure coding systems, relying on Galois field arithmetic, are computationally complex and slow, limiting their deployment to archival data rather than live data streams due to higher computational requirements.
Innovation Solution
Implementing numerical operation-based erasure coding that determines the bit-width of the computing system, converts data elements into numerical representations, and generates code words using a generator matrix, enabling faster encoding and decoding through simple numerical operations like XOR and matrix multiplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Galois field arithmetic is used for erasure coding, then data redundancy and fault tolerance are achieved, but computational complexity increases and processing speed decreases
Solution Approach 1:
The patent changes the mathematical domain parameter from Galois field arithmetic to numerical operations. By representing data elements as numerical values and using matrix multiplication with a generator matrix, the system maintains erasure coding functionality while operating in a more computationally efficient numerical domain rather than the complex Galois field domain.
Solution Approach 2:
The patent substitutes complex Galois field arithmetic operations with simpler numerical operations. Specifically, it replaces finite field multiplication and addition with standard numerical multiplication and addition, allowing the use of optimized numerical linear algebra libraries and hardware accelerators that are more efficient for these operations.
2Reliability
If Galois field arithmetic is used for erasure coding, then fault tolerance is provided, but processing speed and throughput are reduced
Solution Approach 1:
The patent changes the computational parameter from Galois field operations to numerical operations, enabling the use of highly optimized numerical linear algebra implementations. This parameter change allows leveraging hardware accelerators and software libraries that provide significantly higher throughput for matrix operations compared to traditional Galois field arithmetic.
Solution Approach 2:
The patent segments the erasure coding process into matrix multiplication operations that can be parallelized. By formulating erasure coding as matrix multiplication with a generator matrix, the system can divide the computation into independent operations that execute in parallel, dramatically improving processing speed and throughput.
3Reliability
If traditional replication is used to ensure high availability, then data safety is achieved, but storage efficiency decreases
Solution Approach 1:
The patent changes the mathematical foundation from simple replication to numerical-based erasure coding. By using matrix multiplication with a generator matrix to create coded fragments, the system achieves the same fault tolerance as replication but with fewer storage units required, improving storage efficiency while maintaining data safety and high availability.
Data Source
AI summary
One embodiment provides a system that facilitates numerical operation-based erasure coding. During operation, the system determines the bit-width of processing capability of the computing system. The system then determines, based on the bit-width, a number of bits for representing a respective data element from a data stream and a number of data elements based on a number of a dimension of a generator matrix of erasure encoding. The system then obtains the number of data elements from the data stream and converts a respective obtained data element to a corresponding numerical representation. Here, a respective obtained data element is represented by the determined number of bits. The system then generates a code word, which comprises a plurality of coded fragments, of erasure encoding from the numerical representations based on the generator matrix of the erasure encoding.