Erasure Coding with Slice-Based Reconstruction for Large Disks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As hard disk capacity increases, the time and resources required to reconstruct data in a storage system with erasure codes (EC) become inefficient due to the need for larger data read volumes and increased computational demands.
Innovation Solution
The EC-based data processing method divides each data chunk into two slices, encodes them using a parity matrix with specific settings, and stores the slices across multiple storage devices, reducing the number of input/output requests and computational resources needed for reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If hard disk capacity becomes larger, then storage capacity is improved, but data read volume and reconstruction time increase
Solution Approach 1:
The patent divides data into multiple chunks and further segments each chunk into multiple slices (e.g., 4 slices per chunk). This segmentation allows the reconstruction process to read only specific slices from specific chunks rather than reading entire chunks, significantly reducing the data read volume during reconstruction despite large hard disk capacities.
2Volume of stationary object
If hard disk capacity becomes larger, then storage capacity is improved, but computational resources required for reconstruction increase
Solution Approach 1:
By segmenting data into chunks and slices with specific mathematical relationships defined by encoding matrices, the patent enables selective reconstruction of only the required slices using fewer computational operations. The encoding structure allows efficient calculation of missing slices from available slices without processing the entire dataset.
3Reliability
If conventional EC encoding is used, then fault tolerance is achieved, but reconstruction process occupies many computing resources
Solution Approach 1:
The patent applies segmentation by dividing data into chunks and slices, where each slice is independently encoded with specific parity information. This structured segmentation enables targeted reconstruction operations that require fewer computational resources compared to conventional EC encoding that treats entire chunks as single units.
Solution Approach 2:
The patent changes the encoding parameters by using specific encoding matrices (e.g., Vandermonde matrices) and defining precise mathematical relationships between slices within chunks. These parameter changes optimize the balance between fault tolerance and reconstruction efficiency, allowing reliable data recovery with reduced computational complexity.
Data Source
AI summary
An erasure code (EC)-based data processing method implemented by a storage controller includes obtaining K data chunks, dividing each of the K data chunks into two data slices, encoding the 2*K data slices based on a parity matrix including 2*M rows and 2*(K+M) columns of elements, to obtain 2*M parity slices, and separately storing the K data chunks and the M parity chunks in different storage devices.