I/O Queue Path Selection for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise storage systems face inefficiencies in managing input/output (I/O) operations due to imbalances in data transfer paths, leading to increased latency and reduced throughput, as I/O queues become overloaded, causing some paths to become idle while others are busy processing large queues.
Innovation Solution
The system determines the data transfer maximum for each path, calculates the cumulative amount of data pending on each path, and selects a target path for transmitting I/O operations based on these metrics to optimize data transfer and balance path loads, using a queue management module to dynamically adjust the data transfer maximum and path selection to improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If I/O operations are transmitted through multiple paths without load balancing, then path utilization becomes unbalanced, but implementing path selection based on cumulative data transfer increases system complexity
Solution Approach 1:
The system implements feedback by continuously monitoring the cumulative amount of data transferred on each path and using this information to dynamically select appropriate paths for new I/O operations. The queue management module receives feedback about path utilization and adjusts path selection accordingly, creating a closed-loop control system that optimizes throughput while managing complexity through automated decision-making.
Solution Approach 2:
The path selection mechanism serves itself by automatically monitoring path statistics and making routing decisions without external intervention. The queue management module independently tracks cumulative data transfers and selects paths based on predefined criteria, eliminating the need for complex external load balancers or manual path management while improving throughput.
2Productivity
If data transfer maximum is set high for all paths, then bandwidth utilization increases, but path overload occurs leading to increased latency
Solution Approach 1:
The system applies dynamics by making the data transfer maximum adaptive rather than static. The queue management module dynamically adjusts the effective data transfer maximum for each path based on real-time conditions, selecting paths where the cumulative data transfer plus new I/O data does not exceed the path's data transfer maximum. This dynamic adjustment allows the system to maximize throughput while preventing overload-induced latency.
Solution Approach 2:
The system changes parameters by adjusting which paths are selected for new I/O operations based on the cumulative data transfer parameter. Rather than changing the inherent data transfer maximum of paths, the system changes the effective utilization by selecting paths where parameter constraints are satisfied, thereby optimizing throughput while avoiding latency caused by path overload.
3Ease of operation
If path selection is based on cumulative data transfer, then load balancing improves, but measurement and tracking requirements increase system complexity
Solution Approach 1:
The queue management module serves multiple functions: it manages I/O queues, selects paths for new operations, and tracks cumulative data transfers. By combining these functions in a single component, the system achieves effective load balancing without adding separate measurement and tracking systems, thereby improving ease of operation while minimizing the increase in system complexity.
4Productivity
If I/O queues are allowed to grow large, then throughput increases, but path imbalance occurs causing some paths to become idle
Solution Approach 1:
The system performs preliminary action by evaluating the cumulative data transfer on each path before assigning new I/O operations. The queue management module proactively selects paths that have lower cumulative transfers, preventing queue imbalance before it occurs. This preliminary assessment ensures balanced path utilization while maintaining high throughput, as new I/Os are distributed to paths that can handle them without creating overload.
Data Source
AI summary
Managing input/output (‘I/O’) queues in a data storage system, including: receiving, by a host that is coupled to a plurality of storage devices via a storage network, a plurality of I/O operations to be serviced by a target storage device; determining, for each of a plurality of paths between the host and the target storage device, a data transfer maximum associated with the path; determining, for one or more of the plurality of paths, a cumulative amount of data to be transferred by I/O operations pending on the path; and selecting a target path for transmitting one or more of the plurality of I/O operations to the target storage device in dependence upon the cumulative amount of data to be transferred by I/O operations pending on the path and the data transfer maximum associated with the path.