Dynamic MAC Scheduler for AI Engine Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The static mappings between layers of a neural network model and MAC units in AI engines lead to low utilization of MAC units, resulting in idle resources due to non-multiple MAC requirements, sparsity, and latency in data or instruction fetch, which affects the performance of AI engines.
Innovation Solution
Implementing a dynamic scheduling mechanism using an O(1) scheduler that maintains status information of MAC units through a bit array, allowing for the allocation of idle MAC units to process feature map slices based on availability, and utilizing linked lists to efficiently manage feature map slices and weights without physical partitioning of SRAM.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If static mappings are used between layers and MAC units, then the mapping is simple and deterministic, but MAC unit utilization is low due to idle resources
Solution Approach 1:
The patent implements dynamic scheduling that allows MAC units to be reassigned to different feature map slices based on real-time availability and workload, transforming the static mapping into a dynamic allocation system that adapts to varying computational demands and improves resource utilization
Solution Approach 2:
The patent introduces a scheduler as an intermediary component that manages the allocation between feature map slices and MAC units, using data structures like bit arrays and linked lists to coordinate resource distribution and resolve the contradiction between simple mapping and high utilization
2Productivity
If the number of MACs required is not a multiple of total MAC units, then some MACs become idle, but increasing MAC units increases hardware cost
Solution Approach 1:
The patent makes MAC units universal by allowing them to process different feature map slices from different layers through dynamic scheduling, enabling the same set of MAC units to handle variable workload requirements without needing additional hardware for each specific case
3Adaptability or versatility
If static mappings are used, then resource allocation is deterministic, but MAC units cannot adapt to sparsity and latency variations
Solution Approach 1:
The patent implements feedback mechanisms where the scheduler monitors MAC unit completion status and dynamically reallocates idle MAC units to pending feature map slices, creating a closed-loop system that adapts to sparsity patterns and latency variations in real-time
Data Source
AI summary
Embodiments of the invention disclose an integrated circuit and a method for improving utilization of multiply and accumulate (MAC) units on the integrated circuit in an artificial intelligence (AI) engine. In one embodiment, the integrated circuit can include a scheduler for allocating the MAC units to execute a neural network model deployed on the AI engine to process input data. The scheduler includes status information for the MAC units, and can select one or more idle MAC units based on the status information for use to process the feature map slice. The integrated circuit can dynamically map idle MAC units to an input feature map, thereby improving utilization of the MAC units. A pair of linked list, each with a reference head, can be provided in a static random access memory (SRAM) to store only feature map slices and weights for a layer that is currently being processed. When processing a next layer, the two reference heads can be swapped so that output feature map slices for the current layer can be used as input feature maps for the next layer.