Data Block Reconstruction Using Mojette Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication solutions with high replication factors are inefficient in terms of storage space and cost, and erasure coding techniques, such as Reed-Solomon coding, introduce latency during the decoding process, especially when reconstructing data blocks.
Innovation Solution
The proposed solution utilizes the Mojette transform encoding scheme, which reduces algorithmic complexity and latency during the decoding stage by generating projection vectors based on a two-dimensional support, allowing for the reconstruction of missing data lines using the inverse Mojette transform.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Reed-Solomon coding is used for erasure coding, then data reliability is improved, but decoding latency increases
Solution Approach 1:
The patent replaces the traditional Reed-Solomon algebraic decoding mechanism with a neural network-based system. The neural network is trained offline to learn the decoding function, and during operation, it performs decoding through forward propagation, substituting complex algebraic computations with optimized neural network inference that runs faster on hardware accelerators.
Solution Approach 2:
The neural network is trained in advance during an offline phase using synthetic data generated from Reed-Solomon encoded data. This preliminary training allows the network to learn the decoding patterns beforehand, so that during actual decoding operations, the system can quickly reconstruct erased data without performing complex real-time algebraic computations.
2Reliability
If high replication factor is used, then data availability is improved, but storage cost increases
Solution Approach 1:
The system changes the redundancy parameter from high replication (3-7 copies) to erasure coding with lower redundancy ratio. By using neural network-based decoding, the system can efficiently handle erasure coding schemes that achieve the same data availability with significantly reduced storage overhead, as erasure coding stores only k data blocks plus n parity blocks instead of full replicas.
3Device complexity
If Mojette transform is used for encoding, then decoding complexity is reduced, but encoding optimality decreases
Solution Approach 1:
The patent replaces the Mojette transform geometric projection mechanism with a neural network-based system. The neural network learns the optimal decoding function during training and can handle various erasure coding schemes universally, providing both low complexity during inference and maintaining encoding optimality through learned representations rather than fixed geometric transformations.
Data Source
Figure 1
Figure 2a~2c
Figure 3
AI summary
A method for reconstructing a data block of size N is proposed. The data block was encoded using an erasure code to generate a set of NS systematic symbol vectors and a set of Np parity projection vectors from a mapping of the data block onto a two-dimensional convex support. The method comprises: for each input vector that contains at least an erasure, updating the value of each erased symbol to a predetermined value; mapping the NS input vectors with updated values onto the two-dimensional convex support, generating a reconstruction projection vector from the mapping of the NS input vectors with updated values onto the two-dimensional convex support using an encoding projection direction; and generating an updated parity projection vector from the reconstruction projection vector and the parity projection vector generated using said encoding projection direction.