Storage Segment Identifier for Partial Data Rebuild
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In storage systems, data recovery processes consume resources and increase response times due to the need to read and process data from operational units, especially when a storage unit fails, leading to a degraded mode of operation and vulnerability to permanent data loss, with full or partial rebuilds requiring significant processing and resource allocation.
Innovation Solution
The use of a log structured file system with segment identifiers to determine which data segments need to be rebuilt during a partial rebuild process, where segment identifiers are stored in persistent storage, allowing the system to identify and rebuild only necessary segments, reducing the resource overhead and recovery time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a full rebuild is performed to recover all data from a failed storage unit, then data reliability is improved, but resource consumption and recovery time increase significantly
Solution Approach 1:
The system performs preliminary actions by storing the maximum segment identifier in persistent storage before a failure occurs. This pre-stored information enables the system to immediately identify which data segments need reconstruction after a failure, eliminating the need to scan through all data to determine what needs rebuilding, thus significantly reducing recovery time while maintaining completeness.
Solution Approach 2:
Instead of always performing a full rebuild, the system performs a partial rebuild by using the pre-stored maximum segment identifier to determine only the subset of data segments that need reconstruction. This partial action approach reduces resource consumption and time while ensuring all necessary data is recovered.
2Reliability
If data recovery operations are performed to replace stale data, then data reliability is improved, but storage system response time to host requests increases
Solution Approach 1:
The maximum segment identifier is preliminarily stored in persistent storage before failures occur. When a failure is detected, this pre-stored value immediately enables the system to calculate the exact range of segments needing reconstruction, allowing recovery operations to start quickly and complete faster, thus minimizing the impact on host request response times.
Solution Approach 2:
The system uses its own pre-stored maximum segment identifier information to autonomously determine the reconstruction scope without requiring external intervention or extensive scanning. This self-service capability enables rapid, efficient recovery that minimizes disruption to normal storage operations and host requests.
3Productivity
If the storage system operates in degraded mode with reduced redundancy, then resource availability for normal operations is improved, but vulnerability to permanent data loss increases
Solution Approach 1:
The system preliminarily stores the maximum segment identifier in persistent storage, which enables rapid identification of data needing reconstruction. This allows the system to quickly perform targeted rebuilds of only the necessary segments, restoring full redundancy as soon as possible after a failure, thereby minimizing the duration of degraded mode operation and reducing exposure to data loss risks.
Data Source
AI summary
The failure of a storage unit in a storage array of a storage system may render the storage unit unresponsive to any requests. Any writes to the storage system that occur during the failure of the storage unit will not be reflected on the failed unit, rendering some of the failed unit's data stale. Assuming the failed unit's data is not corrupted but is just stale, a partial rebuild may be performed on the failed unit, selectively reconstructing only data that is needed to replace the stale data. Described herein are techniques for storing information that identifies the data that needs to be rebuilt. When the storage unit fails, the segment identifier associated with the last data segment written to the storage system may be stored. Upon the storage unit recovering, the storage system can rebuild only those data segments whose identifier is greater than the stored segment identifier.