Neural Network Kernel Allocation for Accelerator Resource Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently executing multiple neural network models on accelerators due to resource contention and idle time, leading to degraded utilization rates and increased latency.
Innovation Solution
A method and apparatus that generate multiple candidate kernels for each layer of a neural network model, selecting and allocating a single kernel based on accelerator status and resource use to maximize utilization and minimize contention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple neural network models are executed concurrently on accelerators, then throughput is improved, but resource contention occurs leading to degraded utilization rates
Solution Approach 1:
The system segments the execution of neural network models by generating multiple candidate kernels for each layer and selecting appropriate kernels based on accelerator status. This segmentation allows different models to use different kernel configurations, reducing resource contention while maintaining high throughput concurrent execution.
Solution Approach 2:
The system dynamically selects and allocates candidate kernels based on real-time accelerator status information. This dynamic adaptation allows the system to optimize resource utilization during runtime by choosing kernels that best match the current accelerator state, preventing resource contention while maintaining high throughput.
2Productivity
If multiple neural network models are executed concurrently on accelerators, then throughput is improved, but idle time increases leading to degraded utilization rates
Solution Approach 1:
The system performs preliminary actions by generating multiple candidate kernels in advance for each layer of neural network models. These candidate kernels are prepared with different resource usage characteristics, allowing the scheduler to quickly select an appropriate kernel without causing idle time during runtime, thus maintaining high throughput.
Solution Approach 2:
The system uses feedback from accelerator status information to guide kernel selection. By continuously monitoring accelerator state and selecting candidate kernels based on this feedback, the system minimizes idle time and ensures continuous productive operation, maintaining high throughput without resource wastage.
3Reliability
If candidate kernels are selected based on accelerator status, then resource utilization is maximized, but system complexity increases
Solution Approach 1:
The system manages complexity by changing parameters of candidate kernels (such as number of cores, data reuse patterns, and tiling factors) rather than creating entirely different kernel implementations. This parameter-based approach allows flexible adaptation to accelerator status while keeping the overall system structure manageable.
4Reliability
If adaptive resource allocation is performed, then resource utilization is maximized, but processing overhead increases
Solution Approach 1:
The system performs adaptive resource allocation efficiently by preparing multiple candidate kernels in advance with different resource usage profiles. This preliminary preparation eliminates the need for complex runtime optimization, reducing processing overhead while maintaining high resource utilization through pre-computed kernel options.
Data Source
AI summary
A processor-implemented data processing method includes: receiving a request for executing a neural network model on an accelerator; generating a plurality of candidate kernels for each of a plurality of layers comprised in the model; and allocating, to the accelerator, a single candidate kernel that is selected from among a plurality of candidate kernels for a layer to run on the accelerator based on corresponding kernel information and status information of the accelerator.