Parity-Based Erasure Coding for Silent Data Corruption Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Erasure coding systems are impractical for RAID systems with more than two check drives due to computational complexity and hardware requirements, and they fail to detect and correct silent data corruptions effectively, especially in large-scale storage environments.
Innovation Solution
A practical erasure coding system that uses a precomputed master encoding matrix for efficient data encoding and decoding, supports up to 256 drives, and includes a fast solution matrix algorithm for Reed-Solomon codes, enabling error detection and correction, including silent data corruption, with optimized RAID cache management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional erasure coding systems are used with more than two check drives, then data redundancy and reliability improve, but computational complexity and hardware requirements become impractical
Solution Approach 1:
The patent transforms the computational problem by changing the mathematical parameters from traditional polynomial-based Reed-Solomon codes to matrix-based codes over finite fields. This parameter change enables efficient computation even with many check drives (M>2) by using pre-computed encoding matrices and avoiding complex polynomial multiplication, thus resolving the contradiction between reliability through multiple check drives and computational complexity
Solution Approach 2:
The system performs preliminary action by pre-computing and storing encoding matrices before actual data encoding operations. These pre-computed matrices contain all necessary transformation information, allowing rapid encoding during operation without repeating complex calculations. This preliminary preparation resolves the computational complexity issue while maintaining high reliability through multiple check drives
2Reliability
If traditional erasure coding systems are used, then error correction capability is provided, but they fail to detect and correct silent data corruptions effectively
Solution Approach 1:
The patent applies universality by designing a single matrix-based encoding system that simultaneously provides multiple functions: traditional error correction, silent data corruption detection, and data reconstruction. The same encoding matrices and syndrome computation mechanism handle all these tasks, improving measurement precision for detecting subtle corruptions while maintaining error correction capability across diverse failure modes
3Reliability
If more check drives are added to increase reliability, then the system can handle more drive failures, but the computational overhead and hardware requirements become impractical
Solution Approach 1:
By changing from polynomial-based to matrix-based encoding parameters, the system achieves linear time complexity O(N) for encoding operations regardless of the number of check drives M. This parameter transformation allows adding many check drives to increase drive failure tolerance without proportionally increasing computational overhead, as each encoding operation processes data blocks independently using pre-computed matrix rows
Solution Approach 2:
The encoding process is segmented into independent operations on data blocks, where each block is encoded separately using the same encoding matrix. This segmentation allows parallel processing and distributes computational overhead, enabling the system to handle many check drives without excessive computational burden, as each segment can be processed efficiently and independently
Data Source
AI summary
A system for software error-correcting code (ECC) protection or compression of original data using ECC data in a first memory is provided. The system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The software ECC protection or compression includes: a data matrix for holding the original data in the first memory; a check matrix for holding the ECC data in the first memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the ECC data; and a thread for executing on the processing core. The thread includes a Galois Field multiplier for multiplying entries of the data matrix by an entry of the encoding matrix, and a sequencer for ordering operations using the Galois Field multiplier to generate the ECC data.