Neural Network Layer Scheduling for QoS and Idle-Time Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices face challenges in efficiently utilizing hardware resources of accelerators due to idle times when executing multiple neural networks with varying workloads, leading to significant tail-latency and decreased performance.
Innovation Solution
A processor-implemented method and apparatus that schedules neural networks in units of layers based on quality of service (QoS) information and idle time, minimizing idle times by optimizing the execution order of layers within the accelerator.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple neural networks are executed on an accelerator with varying workloads, then the accelerator can handle diverse computational tasks, but idle times occur between layer executions leading to decreased hardware utilization
Solution Approach 1:
The scheduler predicts idle time for each candidate layer before execution by analyzing accelerator state (memory usage, computational resource availability) and model characteristics. This preliminary prediction allows the system to proactively select layers that minimize idle time, rather than reactively responding to idle periods after they occur.
Solution Approach 2:
The scheduling system dynamically adapts to changing accelerator states by continuously monitoring memory usage, computational resource availability, and model execution progress. The scheduler adjusts layer selection in real-time based on current accelerator conditions, making the system flexible rather than static.
2Productivity
If layers are scheduled based on idle time minimization, then hardware utilization improves, but quality of service requirements may be violated for time-sensitive models
Solution Approach 1:
The system changes the scheduling parameter from pure idle time minimization to a composite metric that incorporates both idle time and QoS slack time. By adjusting this parameter combination, the scheduler can shift focus between maximizing hardware utilization and ensuring QoS compliance depending on current system conditions.
Solution Approach 2:
The scheduler uses predicted QoS slack time as feedback to adjust layer selection. When QoS slack time becomes critical (approaching zero), the scheduler prioritizes layers from models with tighter QoS constraints, ensuring service level agreements are met while still maintaining reasonable hardware utilization.
3Loss of time
If the scheduler explores more candidate layers to find optimal execution order, then idle time reduction improves, but computational overhead of scheduling increases
Solution Approach 1:
Instead of exhaustively evaluating all possible layer permutations, the scheduler performs partial exploration by considering only the most promising candidate layers based on predicted idle time and QoS metrics. This partial action approach achieves sufficient idle time reduction without the prohibitive computational cost of complete optimization.
Data Source
AI summary
A processor-implemented method with scheduling includes: receiving one or more execution requests for a plurality of models executed independently of each other in an accelerator; predicting, for each of the plurality of models, quality of service (QoS) information corresponding to the model; and scheduling the plurality of models in units of layers of the plurality of models based on, for each of the plurality of models, either one or both of the QoS information and an idle time occurring in response to a candidate layer to be scheduled in the model being executed in the accelerator.