Neural Network Layer Scheduling for Accelerator Idle-Time Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network execution systems face challenges in efficiently utilizing hardware resources due to idle times and tail latencies caused by unsynchronized execution of tasks across different neural networks, leading to suboptimal performance and throughput.
Innovation Solution
A method and device that perform layer-wise scheduling of neural network models based on estimated idle times of operation and memory access resources, allowing for real-time queuing and execution in an accelerator to minimize idle times and maximize resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If memory block prefetching is used to reduce memory bandwidth idleness, then memory bandwidth utilization is improved, but processing element array idleness increases when prefetching is not enough to resolve compute block dependencies
Solution Approach 1:
The patent implements dynamic scheduling that adjusts memory block prefetching strategies based on real-time compute block dependency status. When compute blocks are ready, prefetching is intensified; when dependencies block execution, the system dynamically switches to compute block merging to keep processing elements busy, thus resolving the contradiction between memory bandwidth utilization and processing element utilization.
Solution Approach 2:
The patent merges multiple memory blocks and compute blocks together in a unified scheduling framework. By combining memory block prefetching with compute block merging, the system ensures that when memory operations complete, compute blocks are ready to immediately utilize the processed data, eliminating processing element idle time while maintaining high memory bandwidth utilization.
2Productivity
If compute block merging is applied to improve processing element array utilization, then processing element utilization is improved, but on-chip memory footprint increases
Solution Approach 1:
The patent employs dynamic scheduling that adapts compute block merging intensity based on current on-chip memory availability. When memory pressure is high, the system reduces merging size; when memory is available, it increases merging to maximize processing element utilization, thus dynamically balancing utilization and memory footprint.
Solution Approach 2:
The patent changes the parameter of compute block size dynamically based on system state. By adjusting block size parameters according to memory availability and workload characteristics, the system achieves high processing element utilization without excessively increasing on-chip memory footprint beyond acceptable thresholds.
3Productivity
If multiple neural network models are executed in parallel to improve throughput, then system throughput is improved, but resource scheduling complexity increases
Solution Approach 1:
The patent segments the scheduling problem into independent per-layer decisions rather than attempting to schedule entire models at once. Each layer is scheduled independently based on its own idle time characteristics, which simplifies the scheduling algorithm while still achieving high parallel throughput across multiple neural network models.
Solution Approach 2:
The patent uses idle time as a key parameter to simplify scheduling decisions. By calculating and comparing idle times for different layers across multiple models, the system makes straightforward scheduling choices that maximize throughput without requiring complex scheduling algorithms, thus managing multiple models efficiently.
Data Source
Figure 1a
Figure 1b
Figure 2
AI summary
A scheduler, a method of operating the scheduler, and an electronic device including the scheduler are disclosed. The method of operating the scheduler configured to determine a model to be executed in an accelerator includes receiving one or more requests for execution of a plurality of models to be independently executed in the accelerator, and performing layer-wise scheduling on the models based on an idle time occurring when a candidate layer which is a target for the scheduling in each of the models is executed in the accelerator