NVMe Storage Load Balancing via Dynamic I/O Queue Reassignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current NVMe storage systems experience performance degradation due to I/O queue overlap, leading to imbalanced CPU core loads and increased latency, as multiple hosts' queues often overlap on individual CPU cores, resulting in reduced parallelism and increased I/O latency.
Innovation Solution
A method for detecting overlapping I/O queues and rebalancing their assignments across CPU cores by monitoring CPU core utilization and workload, identifying overloaded cores, and shifting I/O traffic to underutilized cores, using NVMe-oF architecture and Advanced Error Reporting messages to notify hosts and adjust queue pathing policies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple hosts' I/O queues are assigned to individual CPU cores, then queue processing efficiency is improved, but CPU core load becomes imbalanced and latency increases
Solution Approach 1:
The patent implements dynamic load balancing by continuously monitoring CPU core utilization metrics and adjusting I/O queue assignments in real-time. The system transitions from static queue-to-core mapping to dynamic reassignment based on current load conditions, allowing the system to adapt to changing workload patterns and maintain balanced CPU core utilization.
Solution Approach 2:
The patent employs feedback mechanisms by monitoring CPU core utilization metrics and using this information to make informed decisions about queue reassignment. The system collects performance data, analyzes load patterns, and adjusts queue assignments based on feedback from the actual system state, creating a closed-loop control system that optimizes load distribution.
2Speed
If I/O queues are concentrated on fewer CPU cores, then processing speed increases, but system latency increases due to overload
Solution Approach 1:
The system dynamically adjusts queue assignments to prevent any single CPU core from becoming overloaded, which would increase latency. By continuously monitoring utilization and redistributing queues from high-load to low-load cores, the system maintains optimal processing speed while avoiding latency penalties associated with core overload.
Solution Approach 2:
The patent changes the assignment parameters of I/O queues based on CPU core utilization metrics. When a core's utilization exceeds thresholds, the system modifies the queue-to-core mapping parameters to redistribute load, thereby maintaining processing speed while preventing latency increases from overload conditions.
3Reliability
If I/O queues are distributed across all CPU cores, then load balancing is improved, but parallelism efficiency decreases
Solution Approach 1:
The system implements dynamic load balancing that adapts to actual workload patterns, maintaining parallelism efficiency by only redistributing queues when necessary. The dynamic approach allows the system to preserve effective parallelism during high-utilization periods while maintaining load balance, avoiding unnecessary queue movements that would reduce parallelism efficiency.
Solution Approach 2:
The patent selectively changes queue assignment parameters based on monitored CPU core utilization thresholds. Rather than uniformly distributing queues across all cores, the system modifies assignments only when load imbalance is detected, thereby maintaining parallelism efficiency while achieving necessary load balance through targeted parameter adjustments.
Data Source
AI summary
In an approach to storage level load balancing, the load level of a storage system is monitored, where the load level is a utilization percentage of a plurality of CPU cores in the storage system. An overload condition is detected based on the utilization percentage of one or more CPU cores exceeding a threshold, where the overload condition is caused by an overlap of one or more I/O queues from multiple host computers accessing a single CPU core. Responsive to detecting the overload condition, a new I/O queue is selected on a second CPU core, where the second CPU core has a utilization percentage less than a second threshold. A recommendation is sent to a host computer, where the recommendation is to move I/O traffic from the first CPU core to the new I/O queue on the second CPU core to rebalance the load level of the storage system.