Global Modulo Allocation for Neural Network Accelerator Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network hardware accelerators face inefficiencies due to underutilization or overutilization of computation and memory resources, leading to suboptimal execution times, especially when shared by multiple tenants with varying resource allocations.
Innovation Solution
A compiler determines a global modulo allocation scheme to schedule repetitive operations based on available computation and memory resources, assigning iterations into batches for parallel execution while optimizing memory usage, thereby maximizing parallelism and resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network hardware accelerators are shared by multiple tenants with varying resource allocations, then resource utilization efficiency is improved, but execution time and performance deteriorate due to resource contention and suboptimal scheduling
Solution Approach 1:
The patent segments the neural network computation into multiple batches, where each batch is assigned to specific computation resources. This segmentation allows different tenants' workloads to be divided into manageable units that can be scheduled independently, resolving the conflict between sharing resources and maintaining execution efficiency. The compiler determines batch assignments based on available resources and data dependencies, enabling efficient multi-tenant operation without excessive resource contention.
2Speed
If more computation resources are allocated to a tenant, then execution speed is improved, but memory resource utilization deteriorates due to overallocation or underutilization
Solution Approach 1:
The patent implements dynamic resource allocation where the compiler determines batch assignments based on the actual availability of computation and memory resources at compile time. This dynamic approach allows the system to adapt resource allocation to current conditions, optimizing both execution speed and memory utilization. The global modulo allocation scheme dynamically adjusts how iterations are distributed across batches, ensuring that memory resources are neither overallocated nor underutilized while maintaining efficient execution speed.
3Productivity
If iterations are assigned into batches for parallel execution, then resource utilization is maximized, but memory footprint increases due to storing multiple batches
Solution Approach 1:
The patent employs a discarding and recovering strategy where intermediate results of processed batches are discarded from memory after use, and memory is recovered for reuse by subsequent batches. This approach allows the system to maintain high resource utilization through batched parallel execution while controlling memory footprint. The compiler optimizes the allocation and deallocation of memory resources across batches, ensuring that memory is freed efficiently after each batch completes its computation, preventing excessive memory accumulation.
4Productivity
If global modulo allocation is used to schedule repetitive operations, then parallelism is maximized, but data dependency management becomes more complex
Solution Approach 1:
The patent applies preliminary action by determining the global modulo allocation scheme and batch assignments during the compilation phase, before actual execution. This preliminary planning allows the compiler to pre-calculate optimal batch assignments that maximize parallelism while accounting for data dependencies. By resolving dependency management issues in advance during compilation, the system achieves high parallelism during execution without the complexity of dynamic dependency management, as the allocation strategy is predetermined based on static analysis of the neural network operations.
Data Source
AI summary
In one example, a method performed by a compiler comprises: receiving a dataflow graph of a neural network, the neural network comprising a neural network operator; receiving information of computation resources and memory resources of a neural network hardware accelerator intended to execute the neural network operator; determining, based on the dataflow graph, iterations of an operation on elements of a tensor included in the neural network operator; determining, based on the information, a mapping between the elements of the tensor to addresses in the portion of the local memory, and a number of the iterations of the operation to be included in a batch, wherein the number of the iterations in the batch are to be executed in parallel by the neural network hardware accelerator; and generating a schedule of execution of the batches of the iterations of the operations.