Dual-Mode Erasure Coding for Stable Distributed Data Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current erasure codes, such as Reed-Solomon, are not well-suited for distributed data storage due to unpredictable performance during erasure operations, particularly in non-systematic modes, leading to inefficiencies in data reconstruction and increased computational demands.
Innovation Solution
The development of a high-performance erasure code, OPTFEC, which employs a dual-mode systematic and non-systematic Mojette transform for encoding and decoding, dividing the decoding process into preamble and stable phases, utilizing arithmetic operations for efficient data reconstruction and error correction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If Reed-Solomon systematic code is used for non-erasure operations, then optimal performance is achieved, but performance suffers severely when erasure is present
Solution Approach 1:
The patent implements a dynamic code switching mechanism that adapts the encoding/decoding approach based on the presence or absence of erasures. When no erasures are detected, the system uses systematic Reed-Solomon code for optimal performance. When erasures are detected, it switches to non-systematic Mojette transform code to maintain reliable decoding. This dynamic adaptation resolves the contradiction by allowing the system to optimize for speed when possible while ensuring reliability when necessary.
2Reliability
If Mojette transform non-systematic code is used, then erasure reconstruction is improved, but CPU cycles increase due to larger parity chunk size
Solution Approach 1:
The patent applies different code characteristics to different operational contexts: systematic codes with smaller parity chunks are used when erasures are absent, while non-systematic codes with larger parity chunks are used when erasures are present. This local optimization allows the system to use the appropriate code properties only where needed, reducing unnecessary CPU cycle consumption while maintaining erasure reconstruction capability when required.
3Reliability
If data is replicated over several DC sites, then data availability is improved, but data multiplication and maintenance work increase
Solution Approach 1:
The patent changes the fundamental parameter of data redundancy from replication (creating multiple identical copies) to erasure coding (creating encoded fragments where any k of n fragments can reconstruct the original data). This parameter change allows the system to achieve the same data availability goal with reduced data volume, as erasure-coded data requires only k fragments out of n total fragments for reconstruction, whereas replication would require storing all n copies.
Data Source
AI summary
A method of redundantly encoding data includes formatting the data into columns and rows, and generating first and second sets of projections of the data using an encoding transform. For each set of projections generated, an encoding parameter of the encoding transform is set to a different value. The first and second sets of projections are stored as the encoded data. A decoding method reads settings including an indication of a number of data fragments. The number of data fragments is compared to a number of projections in a first set of projections of the encoded data in order to determine whether to use a first or a second decoding mode. The encoded data is then decoded according to the selected decoding mode and the result is outputted.