Sequential Resilvering for Multi-Device Storage Pool Failures
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional resilvering processes in large storage systems are inefficient due to the need for repeated data transfers and restarts when multiple storage devices fail, leading to increased resilvering time and resource burden.
Innovation Solution
The method involves sorting data blocks into buckets based on their physical location on the failed device, organizing them into a balanced tree structure, and reusing information from the first phase of resilvering for subsequent failed devices to minimize restarts and optimize sequential I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional resilvering processes are used to restore failed storage devices, then data redundancy is maintained, but resilvering time increases significantly and system performance degrades during the process
Solution Approach 1:
The system performs preliminary actions by discovering and sorting data blocks into buckets in the order they appear on the failed device before actual resilvering begins. This preliminary organization of data blocks into sequential buckets enables faster subsequent transfer operations, reducing overall resilvering time while maintaining data redundancy
Solution Approach 2:
The resilvering process is segmented into distinct phases: data block discovery, sorting into buckets, and sequential transfer. By dividing the failed device's data into manageable buckets organized by physical location, the system can process and transfer data more efficiently, reducing the time penalty associated with maintaining data redundancy
2Reliability
If all data is copied from synchronized mirrors to unsynchronized mirrors after device failure, then data synchronization is restored, but system performance is significantly reduced during resynchronization
Solution Approach 1:
Before performing the actual data copy operation, the system preliminarily discovers all data blocks that need to be transferred and sorts them into buckets based on their physical location order on the failed device. This preliminary sorting enables sequential I/O operations during the copy phase, maintaining data synchronization while minimizing performance impact
Solution Approach 2:
The system maintains continuous useful action by organizing data blocks into sequential buckets that can be transferred in a continuous stream rather than random access patterns. This continuity reduces I/O operations and maintains better system performance during the resynchronization process while ensuring data synchronization is restored
3Reliability
If repeated data transfers and restarts are performed when multiple storage devices fail, then all failed devices are restored, but resource utilization increases and resilvering time extends
Solution Approach 1:
When the first storage device fails, the system performs preliminary data block discovery and sorting into buckets before the second device failure occurs. This preliminary work can be reused when the second device fails, avoiding repeated discovery and sorting operations. This reduces resource utilization and accelerates restoration of multiple failed devices
Solution Approach 2:
The system recovers and reuses the data block discovery results and bucket sorting information from the first resilvering operation when a second device fails. Instead of discarding this intermediate work and starting over, the system recovers and applies the same sorting structure to the second device, reducing resource consumption and accelerating multi-device restoration
Data Source
AI summary
A method of resilvering a plurality of failed devices in storage pools includes detecting a failure of a first storage device in a storage pool, identifying data blocks that were stored on the first storage device that are also stored on other storage devices, and resilvering the first storage device by transferring the data blocks from the other storage devices. While resilvering the first storage device, the method includes detecting a failure of a second storage device in the storage pool, identifying a subset of the data blocks that were stored on the first storage device that were also stored on the second storage device, and reusing a set of sequential I/O commands to resilver at least a portion of the second storage device with the subset of the data blocks.