Drive Path Quarantining for Storage I/O Error Diagnosis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems incorrectly handle I/O operation failures, often prematurely identifying storage devices as failing components due to backend issues like cable disconnections, leading to unnecessary service interruptions and data loss.
Innovation Solution
Implementing path quarantining, where storage processors transition problematic paths to a quarantined state to determine the cause of failures, preventing further I/O operations until the issue is resolved, allowing the platform manager to diagnose and communicate the reason to the system manager for proper handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage node immediately retries I/O operations on alternate paths upon failure, then system availability is improved, but false positives in identifying storage device failures increase
Solution Approach 1:
The system performs preliminary diagnostics on the alternate path before initiating full I/O operations. The management processor checks whether the alternate path is functional and identifies the root cause of the original failure before switching paths, preventing premature conclusions about storage device failures.
Solution Approach 2:
The management processor acts as an intermediary between the storage processor and storage devices. It receives I/O requests, determines the appropriate path based on failure analysis, and coordinates the switching between paths, ensuring that path failures are distinguished from device failures.
2Productivity
If the storage node quickly switches to alternate paths upon failure, then service continuity is improved, but unnecessary service interruptions increase
Solution Approach 1:
Before switching paths, the system performs preliminary checks to determine if the failure is due to a path issue or a storage device issue. This preliminary analysis ensures that switching to an alternate path is appropriate and will not cause unnecessary service interruptions.
Solution Approach 2:
The system implements feedback mechanisms where the management processor continuously monitors path status and I/O operation results. This feedback allows the system to adaptively switch paths based on real-time conditions while maintaining service stability through informed decision-making.
3Measurement precision
If the system performs comprehensive diagnostics before path switching, then false positives are reduced, but response time increases
Solution Approach 1:
The diagnostic process is segmented into multiple levels: quick checks for immediate path availability, followed by more comprehensive diagnostics only when needed. This segmentation allows the system to achieve accurate failure diagnosis without always incurring the full time cost of comprehensive diagnostics.
Solution Approach 2:
The system performs partial diagnostics - sufficient to distinguish between path failures and device failures - rather than exhaustive diagnostics in all cases. This partial action approach achieves the necessary diagnostic precision while minimizing time loss.
Data Source
AI summary
A technique involves, in response to encountering a predefined number of consecutive I/O errors using a drive path to a storage drive, transitioning the drive path from online to quarantined to temporarily deny further I/O operations from being processed using the drive path. The technique further involves starting a quarantine timer that defines a quarantine time period. The technique further involves performing an update operation that updates the drive path. The update operation (i) changes the drive path from quarantined to removed to continue denying further I/O operations from being processed using the drive path when a removal notification is received before the quarantine time period expires, and (ii) changes the drive path from quarantined to back to online to allow further I/O operations to be processed using the drive path when a removal notification is not received before the quarantine time period expires.