Neural Network Layer Scheduling for QoS and Idle-Time Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices face challenges in efficiently scheduling multiple neural networks on accelerators, leading to significant idle times and tail-latencies due to the nature of workload differences, which results in suboptimal utilization of hardware resources and increased service latency.
Innovation Solution
A processor-implemented method and apparatus that schedules neural networks in units of layers based on quality of service (QoS) information and idle time, minimizing idle times by prioritizing layers with minimal idle time and considering workload characteristics, memory availability, and data dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple neural networks are scheduled on accelerators without considering idle time and QoS, then device complexity is reduced, but productivity decreases due to significant idle times and tail-latencies
Solution Approach 1:
The patent changes the scheduling parameters by introducing QoS information and idle time metrics as key decision factors. The scheduler predicts QoS slack time and standalone execution time for each model, using these parameters to dynamically adjust scheduling decisions. This transforms the scheduling approach from simple task queue management to parameter-driven optimization, resolving the contradiction by making productivity improvements through systematic parameter analysis rather than complex adaptive algorithms.
Solution Approach 2:
The patent applies preliminary action by predicting QoS information and idle time before actual scheduling decisions are made. The system pre-calculates QoS slack time and standalone execution time for each model, allowing the scheduler to make informed decisions in advance. This preliminary prediction phase enables efficient scheduling without requiring complex real-time adjustments, thereby improving productivity while keeping the scheduling mechanism manageable.
2Loss of time
If layers are scheduled based on minimal idle time, then loss of time is reduced, but device complexity increases due to the need to track and predict idle time for multiple models
Solution Approach 1:
The patent implements self-service by having each model provide its own QoS information and idle time characteristics to the scheduler. Each model's execution requests include predicted QoS slack time and standalone execution time, allowing the scheduler to make decisions based on self-reported data from each model. This eliminates the need for the scheduler to complexly track and predict idle time for all models, reducing scheduling mechanism complexity while still achieving minimal overall idle time.
3Reliability
If QoS information is predicted for each model, then reliability of service quality is improved, but loss of time increases due to the prediction process
Solution Approach 1:
The patent applies preliminary action by performing QoS prediction during the request submission phase, before actual execution begins. The QoS slack time and standalone execution time are predicted in advance and attached to each execution request, allowing the scheduler to use this information immediately without additional prediction delays during scheduling decisions. This ensures reliable QoS management while minimizing the time cost of predictions.
Data Source
AI summary
A processor-implemented method with scheduling includes: receiving one or more execution requests for a plurality of models executed independently of each other in an accelerator; predicting, for each of the plurality of models, quality of service (QOS) information corresponding to the model; and scheduling the plurality of models in units of layers of the plurality of models based on, for each of the plurality of models, either one or both of the QoS information and an idle time occurring in response to a candidate layer to be scheduled in the model being executed in the accelerator.