InfiniBand I/O Path Failover via Proactive Rerouting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In storage management environments with multiple I/O paths, failures can cause widespread I/O errors and performance degradation due to the need for reactive rerouting of I/O operations, which drains computing resources and is inefficient in large cluster setups.
Innovation Solution
Implement a proactive I/O rerouting system in an InfiniBand managed storage environment by registering callbacks for InfiniBand events and polling hardware components to detect failures, allowing for immediate rerouting of all I/O operations from a failing path set to a separate path set on a different InfiniBand channel, without waiting for I/O errors to occur.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reactive I/O rerouting is used to handle failures, then I/O operations can be redirected to alternate paths, but computing resources are drained and I/O performance degrades due to continuous error detection and rerouting
Solution Approach 1:
The system performs preliminary actions by monitoring path health status and proactively rerouting I/O operations before failures occur. The host level multi-path component detects potential failures through path monitoring and initiates rerouting in advance, preventing the need for reactive error handling and reducing computing resource consumption.
2Reliability
If reactive I/O rerouting is used, then failed I/Os can be redirected, but new incoming I/Os may be scheduled on failing paths that have not yet returned errors, causing additional rerouting needs
Solution Approach 1:
The system monitors path status continuously and proactively identifies failing paths before they cause I/O errors. By detecting path failures in advance and pre-rerouting I/O operations, the system eliminates the delay associated with reactive error detection and ensures continuous I/O operation without unnecessary rerouting.
3Reliability
If multiple I/O paths are used in large cluster setups, then redundancy and availability are improved, but a single failure can cause widespread I/O errors requiring extensive rerouting
Solution Approach 1:
The host level multi-path component implements proactive path monitoring and pre-rerouting capabilities. When a path failure is detected, the system immediately redirects I/O operations to alternate paths without waiting for error propagation, thereby reducing the impact of failures in large cluster environments and simplifying path management complexity.
Data Source
AI summary
An InfiniBand managed storage environment is made up of processor nodes containing HCAs and managed storage devices containing TCAs and exposing a plurality of LUNs and volumes. For each InfiniBand channel between a specific HCA and a specific TCA, the paths between the HCA and any LUN or volume exposed by the TCA are grouped into a set. Occurrence of failures on specific paths of specific sets on specific channels are determined, for example by registering for callbacks or polling for occurrence of events which adversely affect communication between endpoints. Also, I/O operations executed by processor nodes are tracked and failures thereof are detected. When the occurrence of a failure on a specific path of a set is determined, all I/O operations on all paths of the set are proactively rerouted to a separate set on a separate channel that connects the same processor node and storage device.