Cluster I/O Failover via Grouped Path Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-availability clusters face performance issues due to high CPU usage caused by large numbers of network messages during I/O path failure protocols in large cluster configurations with many nodes and LUNs, leading to slow failover times and potential business disruptions.
Innovation Solution
Implementing a proactive I/O path failover method that includes generating a cluster-wide I/O failure message to identify a faulted controller, queuing I/O operations through the faulted controller, and committing to an alternate controller, thereby reducing protocol overhead and improving failover speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a cluster wide I/O failover protocol is implemented to detect path failures and choose best common paths on all nodes, then reliability of data access is improved, but CPU usage increases significantly due to large numbers of network messages exchanged between nodes
Solution Approach 1:
The patent segments the cluster into I/O groups where each group shares common I/O paths to storage devices. By organizing nodes into groups based on their I/O path relationships, the system can perform failover detection and path selection at the group level rather than requiring all nodes to communicate with each other individually, thereby reducing the total number of network messages and CPU usage while maintaining reliability.
Solution Approach 2:
The patent introduces a new dimension of organization by creating I/O groups that are separate from the traditional node-based cluster structure. This dimensional change allows the system to manage failover protocols more efficiently by operating within the I/O group dimension, reducing the complexity of inter-node communication while preserving data access reliability across the entire cluster.
2Measurement precision
If nodes exchange large numbers of network messages to determine best available paths for each LUN during failover, then path selection accuracy is improved, but failover performance deteriorates due to high protocol overhead
Solution Approach 1:
The patent performs preliminary actions by pre-establishing I/O groups and identifying common I/O paths before failures occur. During normal operation, nodes within each I/O group already have knowledge of their shared paths and relationships. When a failure occurs, the system can quickly determine alternative paths within the group without requiring extensive real-time communication, thus maintaining path selection accuracy while improving failover performance.
Solution Approach 2:
The patent merges the path selection and failover decision processes into the I/O group structure. Instead of each node independently evaluating all possible paths to all LUNs across the entire cluster, nodes within an I/O group collaboratively determine paths for their shared storage devices. This merging reduces the search space and message overhead while maintaining accurate path selection for the affected LUNs.
Data Source
AI summary
A method for fast I/O path failure detection and cluster wide failover. The method includes accessing a distributed computer system having a cluster including a plurality of nodes, and experiencing an I/O path failure for a storage device. An I/O failure message is generated in response to the I/O path failure. A cluster wide I/O failure message broadcast to the plurality of nodes that designates a faulted controller. Upon receiving I/O failure responses from the plurality of nodes, an I/O queue message is broadcast to the nodes to cause the nodes to queue I/O through the faulted controller and switch to an alternate controller. Upon receiving I/O queue responses from the plurality of nodes, an I/O failover commit message is broadcast to the nodes to cause the nodes to commit to a failover and un-queue their I/O.