Multi-Engine AI Processing Unit Dynamic Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network models, especially deep neural networks, face inefficiencies due to their large scale and complexity, which makes them difficult to run efficiently on general hardware platforms, as they often require a fixed number of compute units to be activated regardless of being compute or memory bound, leading to suboptimal throughput and power usage.
Innovation Solution
A multi-engine processing unit system that includes a compiler to determine whether operations/threads are compute or memory bound, a thread scheduler to allocate and schedule these operations across a plurality of compute units, and an executor to dynamically adjust the number of active compute units based on runtime performance, thereby optimizing uRate and power usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a fixed number of compute units are activated for neural network operations, then the system structure is simple and easy to control, but the throughput and power usage efficiency deteriorate
Solution Approach 1:
The system dynamically adjusts the number of activated compute units based on runtime performance monitoring and operation characteristics. The executor can change the number of active CUs by activating or deactivating units based on whether operations are compute-bound or memory-bound, transforming a static system into a dynamic one that adapts to workload requirements.
Solution Approach 2:
The compute units are divided into multiple independent engines that can be selectively activated. Each compute unit operates as an independent engine with its own buffer, allowing fine-grained control over resource allocation. This segmentation enables the system to activate only the necessary number of units for each specific operation.
2Productivity
If all compute units are activated for every operation, then the system can handle maximum workload, but power consumption increases unnecessarily
Solution Approach 1:
The system applies partial action by activating only the necessary number of compute units required for each operation rather than all units. The executor determines the optimal number of active CUs based on operation characteristics (compute-bound vs. memory-bound), activating fewer units when sufficient and maximizing unit utilization when needed, thus avoiding excessive power consumption.
Solution Approach 2:
The system changes the parameter of active compute unit count based on operation type. For compute-bound operations, more units are activated to maximize processing capacity, while for memory-bound operations, fewer units are activated since memory bandwidth is the limiting factor. This parameter adjustment optimizes the trade-off between execution speed and power consumption.
3Adaptability or versatility
If the system uses static scheduling for compute units, then the control logic is simple, but the system cannot adapt to different operation characteristics
Solution Approach 1:
The system incorporates feedback through runtime performance monitoring. The performance monitor tracks metrics such as CU utilization and memory bandwidth usage, and this feedback is used by the executor to dynamically adjust the number of active compute units. This closed-loop control enables the system to adapt to different operation characteristics while maintaining manageable complexity through automated decision-making.
Solution Approach 2:
The system performs self-service by automatically determining the appropriate number of active compute units based on monitored performance metrics and operation characteristics. The executor autonomously makes scheduling decisions without requiring complex external control logic, allowing the system to adapt to different operation types while keeping the control mechanism relatively simple.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects of the present disclosure provide an apparatus (300). For example, the apparatus (300) can include a compiler (310) configured to compile a neural network, NN, model to generate a plurality of operations/threads and determine whether each of the operations/threads is compute bound or memory bound, and a memory (320) coupled to the compiler (310) and configured to store the operations/threads. The apparatus (300) can also include a thread scheduler (330) coupled to the memory (320) and configured to schedule the operations/threads of the NN model. The apparatus (300) can also include a multi-engine processing unit (350) that includes a plurality of compute units, CUs, and an executor (340) coupled between the thread scheduler (330) and the multi-engine processing unit (350). The executor (340) can be configured to allocate the operations/threads of the NN model and activate a number of the CUs of the multi-engine processing unit (350) for each of the operations/threads based on whether the operation/thread is compute bound or memory bound.