Multi-Code Distributed Storage Joint Decoding for Data Migration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face challenges in combining and migrating data between different Forward Error Correction (FEC) codes, particularly due to differences in coding algorithms, finite fields, and source block sizes, which limits their ability to provide redundancy and data availability across diverse storage systems.
Innovation Solution
The method involves jointly decoding coded symbols from multiple FEC codes with different generator matrices, finite fields, and block sizes, using techniques such as constructing a joint generator matrix, mapping symbols to a common sub-field, and concatenating submatrices to recover files, allowing for the combination and processing of coded data across various FEC codes, including Reed-Solomon and Random Linear Network codes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in multiple distributed storage systems using different FEC codes, then data redundancy and availability are improved, but the complexity of combining and decoding coded data increases
Solution Approach 1:
The patent introduces a joint generator matrix as an intermediary structure that mediates between multiple FEC codes with different generator matrices. This joint generator matrix enables the combination of coded data from different storage systems by providing a unified decoding framework that can process symbols encoded with different codes (Reed-Solomon, RLNC, etc.) without requiring separate decoding processes for each code type.
Solution Approach 2:
The joint generator matrix serves multiple functions: it can decode data from any single FEC code type, combine data from multiple different FEC codes, and work with various finite fields and block sizes. This universal decoding mechanism eliminates the need for separate decoding logic for each code type, thereby reducing overall system complexity while maintaining support for diverse storage systems.
2Adaptability or versatility
If different FEC codes with different finite fields and block sizes are used, then storage system versatility is improved, but the difficulty of combining coded data increases
Solution Approach 1:
The patent employs parameter changes by dynamically adjusting the joint generator matrix construction based on the specific parameters of the input FEC codes. When combining coded data from different sources with varying finite fields and block sizes, the system modifies the joint generator matrix parameters (such as field size and matrix dimensions) to accommodate the least common multiple or maximum of the input parameters, enabling universal compatibility across different code configurations.
Solution Approach 2:
The patent resolves the difficulty of combining different FEC codes by introducing an additional dimension in the form of a joint generator matrix that operates in a higher-dimensional space. This matrix can simultaneously represent and process symbols from multiple codes with different parameters by embedding them in a unified mathematical framework, effectively transforming the combination problem from a conflicting parameter space to a compatible higher-dimensional space.
3Loss of energy
If gradual data migration between FEC codes is implemented, then storage operation cost is reduced, but the complexity of data transformation increases
Solution Approach 1:
The patent enables continuous data migration between different FEC codes by allowing the joint generator matrix to process and combine coded data from multiple code types simultaneously. This continuous processing capability allows data to be gradually migrated from one code type to another without interrupting storage operations, as the system can continuously decode from the source code and encode to the target code using the unified joint generator matrix framework.
Data Source
AI summary
Systems and techniques described herein include jointly decoding coded data of different codes, including different coding algorithms, finite fields, and/or source blocks sizes. The techniques described herein can be used to improve existing distributed storage systems by allowing gradual data migration. The techniques can further be used within existing storage clients to allow application data to be stored within diverse different distributed storage systems.