Networked Storage IOPS Management for Bully Workload Mitigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems fail to efficiently manage resource usage in networked and cloud-based storage systems by not understanding the correlation between resources and workloads, leading to inefficiencies and negative impacts on other workloads.
Innovation Solution
A machine learning-based model using queuing theory is employed to identify bully and victim workloads, which are then addressed through automated corrective actions such as workload migration, quality of service policies, and capacity adjustments based on historical demand and real-time changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional systems monitor and analyze individual resources without understanding correlation between resources and workloads, then resource monitoring is simple and straightforward, but resource usage efficiency deteriorates and negative impacts on other workloads occur
Solution Approach 1:
The system segments resource monitoring by creating separate data structures for different resource types (storage, compute, network) and their associated workloads. Each resource is tracked with its own utilization metrics and workload correlations, allowing detailed analysis without overwhelming system complexity.
Solution Approach 2:
The system implements continuous feedback loops that monitor resource utilization and workload correlations in real-time. When resource contention is detected, the system automatically adjusts resource allocation and provides feedback to stakeholders, improving efficiency through dynamic adaptation.
2Reliability
If automated corrective actions are implemented to address bully and victim workloads, then resource contention is reduced and performance is enhanced, but system complexity and automation extent increase
Solution Approach 1:
The system implements self-service mechanisms where the resource management system automatically detects resource contention, identifies bully and victim workloads, and executes corrective actions without human intervention. The system serves itself by autonomously optimizing resource allocation based on real-time conditions.
Solution Approach 2:
Manual resource management and intervention are replaced with automated machine learning models and algorithms. The system uses computational algorithms to analyze resource utilization patterns, identify contention issues, and execute corrective actions, substituting mechanical human operations with automated digital systems.
Data Source
AI summary
Methods and systems for a networked storage system are provided. One method includes predicting an IOPS limit for a plurality of storage pools based on a maximum allowed latency of each storage pool, the maximum allowed latency determined from a relationship between the retrieved latency and a total number of IOPS from a resource data structure; identifying a storage pool whose utilization has reached a threshold value, the utilization based on a total number of IOPS directed towards the storage pool and a predicted IOPS limit; detecting a bully workload based on a numerical value determined from a total number of IOPS issued by the bully workload for the storage pool and a rising step function; and implementing a corrective action to reduce an impact of the bully workload on a victim workload.