Multi-path Layer Mitigates Flaky SAN Link Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face performance degradation due to 'flaky' links with intermittent IO failures, often attributed to hardware issues in the SAN, leading to excessive IO retries and degraded application performance, requiring costly and inefficient manual intervention.
Innovation Solution
A multi-path layer in host devices, comprising MPIO drivers, detects and mitigates link performance issues by maintaining IO operation counts and determining error conditions for initiator-target pairs, updating link performance information, and adjusting path selection to avoid faulty links.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual intervention is used to detect and resolve link performance issues, then reliability of detection is improved, but loss of time and productivity deteriorate
Solution Approach 1:
The system implements self-service by automatically detecting link performance issues through the MPIO driver monitoring IO operation counts and error conditions. The multi-path layer autonomously identifies flaky links and reroutes IO operations without requiring manual intervention, thereby maintaining detection reliability while eliminating time loss associated with human intervention.
2Reliability
If IO retries are implemented for flaky links, then reliability of data transmission is improved, but productivity deteriorates due to excessive retries
Solution Approach 1:
The system uses feedback by continuously monitoring IO operation counts and error conditions for each initiator-target pair through the MPIO driver. When error rates exceed thresholds, the system automatically identifies the link as flaky and reroutes subsequent IO operations through alternative paths, maintaining data transmission reliability while preventing productivity degradation from excessive retries on faulty links.
Solution Approach 2:
The system applies dynamics by adaptively adjusting path selection based on real-time link performance conditions. The MPIO driver dynamically reroutes IO operations away from identified flaky links to healthy alternative paths, allowing the system to maintain high data transmission reliability while preserving application performance by avoiding static retry configurations that would degrade productivity.
3Ease of operation
If multi-path layer with MPIO drivers is implemented, then ease of operation is improved, but device complexity increases
Solution Approach 1:
The MPIO driver implements multi-functionality by combining multiple capabilities into a single component: it manages multiple paths to storage devices, monitors IO operation counts, detects error conditions, identifies flaky links, and performs automatic rerouting. This universal approach improves ease of operation by consolidating link performance management functions while managing device complexity through integrated rather than separate implementations.
Data Source
AI summary
A processing device controls delivery of input-output (IO) operations from a host device to a storage system over selected paths through a network, and maintains counts of IO operations for each of a plurality of initiator-target pairs, the initiators being implemented on the host device and the targets being implemented on the storage system, each initiator-target pair being associated with a corresponding subset of the paths. Responsive to detection of at least a threshold number of errors relating to the IO operations for a given one of the initiator-target pairs, the processing device determines whether or not the detected errors satisfy one or more specified conditions based at least in part on the maintained count for that initiator-target pair, and responsive to the detected errors for the given initiator-target pair satisfying the one or more specified conditions, the processing device updates link performance issue information for the given initiator-target pair.