Mojette Erasure Coding for Low-Overhead Distributed Repair
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current erasure codes, such as Reed-Solomon, are not well-suited for distributed data storage systems as they incur high repair and update efforts due to their inefficient handling of erasures and redundancy, particularly in non-perfect networks, and do not optimize CPU cycles effectively.
Innovation Solution
The Mojette Transform-based erasure code design introduces sub-block encoding and decoding mechanisms to reduce computational and network efforts, utilizing arithmetic operations for correction and incorporating dual-mode operation for systematic and non-systematic codes, enabling efficient repair and update of data chunks with reduced redundancy projections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Reed-Solomon FEC code is used for distributed data storage, then data redundancy and error correction capability are improved, but repair effort and network overhead increase significantly
Solution Approach 1:
The data block is divided into multiple sub-blocks, and the encoding is performed separately on each sub-block. This segmentation allows repair operations to be performed on smaller units, reducing the network overhead and computational effort required for repair compared to operating on the entire data block at once.
Solution Approach 2:
The patent changes the encoding parameter k (number of data sub-blocks) to optimize the balance between redundancy overhead and repair efficiency. By adjusting k, the system can adapt to different failure scenarios and network conditions, achieving lower repair overhead while maintaining adequate error correction capability.
2Ease of manufacture
If systematic erasure code is used, then encoding simplicity and data availability are improved, but performance during erasure operations deteriorates
Solution Approach 1:
By segmenting the systematic code into sub-blocks with independent encoding, the patent maintains the simplicity of systematic encoding while improving erasure operation performance. Each sub-block can be independently encoded and decoded, allowing faster recovery from erasures without requiring complex global operations.
Solution Approach 2:
The patent introduces dynamic selection of encoding parameters and sub-block configurations based on the erasure scenario. This allows the system to adapt its encoding strategy to match the specific failure pattern, optimizing performance for erasure operations while maintaining encoding simplicity.
3Productivity
If non-systematic Mojette transform is used, then repair overhead is reduced, but compatibility with legacy frameworks and optimal performance in non-erasure modes is lost
Solution Approach 1:
The patent designs the sub-block encoding system to serve multiple functions: it provides efficient repair overhead reduction like non-systematic codes, while also maintaining compatibility with legacy systematic code frameworks. The dual-mode capability allows the same infrastructure to handle both erasure and non-erasure scenarios optimally.
Solution Approach 2:
The patent introduces an intermediary layer of sub-block encoding that bridges between the Mojette transform and legacy frameworks. This intermediary structure allows the system to leverage the repair efficiency of non-systematic codes while maintaining interface compatibility with existing systematic code libraries and frameworks.
4Reliability
If full block encoding is performed, then complete data redundancy is achieved, but computational effort and network bandwidth consumption increase
Solution Approach 1:
The patent divides the full block encoding into independent sub-block encodings. This segmentation allows the system to achieve complete data redundancy through localized operations, significantly reducing both computational effort and network bandwidth consumption compared to performing full block encoding.
Solution Approach 2:
The patent applies encoding only to the necessary sub-blocks rather than the entire data block. This partial action approach achieves sufficient redundancy for error correction while minimizing the computational and network overhead associated with encoding and transmitting redundant data.
Data Source
AI summary
An apparatus for generating encoded data includes processing circuitry configured to encode data using a Mojette transform (MT) based on generating encoded representations of data blocks. Generating the encoded representations of data blocks includes reading data in the form of a data block formatted according to specified settings to comprise rows and columns, creating a set of projections, and outputting the created set of projections to enable storage of the data in the form of the set of projections. The apparatus then transmits the encoded data over a network to another device. Additionally, creating the set of projections includes applying the Mojette transform on the data block, and creating a first number of projections based on mapping each row of the data block to a corresponding projection, wherein the first number of projections carries the same information as a corresponding row.