Storage Fabric Traffic Control for Priority Fan-In Congestion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems with multiple processing nodes experience increased likelihood of starvation of lower-priority operations due to fan-in congestion, where higher-priority operations exacerbate execution delays, leading to timeouts and inefficiencies.
Innovation Solution
Implement techniques to identify and manage sustained high rates of high-priority operations by maintaining counts and calculating exponential moving averages (EMAs) for each process, taking corrective actions when thresholds are exceeded to prevent starvation of lower-priority operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If higher-priority operations are given precedence in execution, then system responsiveness and throughput for critical operations are improved, but lower-priority operations experience execution delays and starvation
Solution Approach 1:
The system monitors the rate of high-priority operations from each process and compares it against thresholds. When a process exceeds the threshold for sustained periods, feedback is used to adjust its priority level, preventing it from monopolizing system resources and causing starvation of lower-priority operations.
Solution Approach 2:
Process priorities are made dynamic rather than static. The system automatically adjusts process priority levels based on real-time monitoring of operation rates, allowing processes that are generating excessive high-priority operations to have their priorities reduced, while other processes can maintain or increase their priorities as needed.
2Ease of operation
If rate limiting is applied to high-priority operations to prevent starvation, then fairness among processes is improved, but system responsiveness to critical operations deteriorates
Solution Approach 1:
Rate limiting is applied partially rather than universally. Only processes that exceed the configured threshold for sustained periods are subject to priority reduction, while other processes continue to execute high-priority operations without restriction. This ensures fairness is improved only where needed, without compromising overall system responsiveness.
3Reliability
If monitoring and corrective actions are implemented for each process, then starvation prevention is improved, but system complexity increases
Solution Approach 1:
The traffic management system implements universal monitoring and control mechanisms that apply to all processes in a unified manner. The same threshold-based monitoring, EMA calculation, and priority adjustment logic is used for every process, simplifying the overall system architecture compared to implementing process-specific complex control algorithms.
Data Source
AI summary
For each source process running on a source node, a rate of high-priority operations initiated by the first process during the period that require accessing a target node can be calculated. An average rate (e.g., an exponential moving average (EMA)) of high-priority operations initiated by the first process over a larger period of time that require accessing the target node can be calculated. Similarly, a rate of high-priority operations initiated by any process executing on a processing node during the period that require accessing the target node can be calculated, and an average rate (e.g., an EMA) of high-priority operations initiated by any such process over the larger period of time that require accessing the target node can be calculated. If one or more of the rates for the period or the average rates reach or exceed respective thresholds, corrective actions can be taken.