Multilevel Erasure Coding for Concurrent Storage Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High Performance Computing (HPC) systems face performance degradation due to storage node failures, which existing storage systems fail to address effectively without significant reconstruction impacts.
Innovation Solution
Implementing a hierarchical storage system with multilevel erasure coding across storage nodes and devices, allowing collaborative reconstruction processes to maximize durability and performance by distributing and optimizing data reconstruction across multiple levels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional single-level erasure coding is used for data redundancy, then data protection is provided, but system performance degrades significantly during reconstruction operations
Solution Approach 1:
The patent divides the storage system into multiple hierarchical levels (storage devices, storage nodes, storage clusters) with erasure coding applied at each level. This segmentation allows reconstruction operations to be localized to specific levels rather than affecting the entire system, thereby maintaining performance while providing comprehensive data protection.
Solution Approach 2:
The patent introduces a hierarchical dimension to traditional erasure coding by organizing storage units into multiple levels. This dimensional transformation enables parallel reconstruction operations across different hierarchy levels, reducing the performance impact of any single reconstruction event while maintaining strong data protection guarantees.
2Reliability
If storage nodes are marked as failed during reconstruction to ensure data integrity, then data protection is maintained, but system availability decreases
Solution Approach 1:
By segmenting the failure domain into hierarchical levels, the patent allows storage units at one level to be marked as failed without necessarily marking entire upper-level units as failed. This enables partial availability where some storage nodes can continue serving requests even during reconstruction operations elsewhere in the hierarchy.
Solution Approach 2:
The patent implements dynamic failure marking where storage units are marked as failed only when necessary for data integrity, and can be dynamically unmarked once reconstruction is complete. This dynamic approach optimizes the balance between maintaining data integrity and preserving system availability during different phases of operation.
3Reliability
If hierarchical storage system with multilevel erasure coding is implemented, then durability and performance are maximized through collaborative reconstruction, but system complexity increases
Solution Approach 1:
The patent manages complexity by segmenting the erasure coding implementation into independent hierarchical levels, where each level can be configured and managed separately. This modular approach allows administrators to control the complexity at each level while benefiting from the cumulative durability improvements across all levels.
Data Source
AI summary
A file system stores data in a hierarchical storage system comprising a plurality of levels of storage units. If an indication of failure of a storage unit at a lower level is received, the storage system indicates a storage unit at the higher level as failed. The storage system initiates reconstruction of data at the storage unit at the higher level based on redundant data stored other storage units at the higher level. If the number of failed storage units at the higher level exceeds a threshold value, the storage system indicates that the storage unit at the higher level is available. The storage system initiates reconstruction of data at the storage unit at the lower level using redundant data stored in other storage units at the lower level. The storage system may perform data reconstruction at the higher level and lower level concurrently.