GPU Kernel Resource Allocation for Concurrent Multipath Neural Network Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current GPU platforms face inefficiencies in resource utilization and sequential execution of operations, leading to high latency and underutilization of resources, especially in multipath neural networks where concurrent execution of kernels is hindered by limited memory and core occupancy, resulting in suboptimal throughput.
Innovation Solution
A system and method for dynamically allocating computing resources by allowing a minimum and maximum number of cores to be allocated to kernels based on concurrent or solo execution, using a causal resource profiling technique to optimize resource allocation and enable concurrent execution of multiple GPU operations, thereby reducing execution time and improving throughput in multipath neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple GPU kernels are executed concurrently, then resource utilization improves, but current GPU platforms prevent concurrent execution due to core occupancy limitations
Solution Approach 1:
The system dynamically adjusts the number of blocks allocated to each kernel based on real-time resource availability and kernel requirements. Instead of static block allocation, the GPU scheduler monitors core occupancy and dynamically reconfigures block distribution to enable concurrent kernel execution while maintaining optimal resource utilization.
Solution Approach 2:
The invention changes the parameter of block allocation from fixed to variable, allowing the number of blocks per kernel to be adjusted based on concurrent execution needs. This parameter change enables the system to transition from sequential execution (100% cores for single kernel) to concurrent execution (shared cores among multiple kernels) without sacrificing performance.
2Power
If GPU kernels are optimized for specific inputs and microarchitectures, then execution efficiency improves, but resource utilization for multiple algorithms becomes insufficient
Solution Approach 1:
The GPU platform is designed to support multiple GPU algorithms simultaneously by creating a universal execution environment where different kernels can share resources. The system allocates blocks and cores dynamically to accommodate various algorithms (e.g., convolutions, GEMM operations) with different resource requirements, making the hardware universally applicable to multiple workloads rather than optimized for a single algorithm.
Solution Approach 2:
The system allows kernels to use partial core resources when executing concurrently, rather than requiring full core occupancy. This partial action approach enables multiple kernels to run simultaneously with reduced individual resource consumption, improving overall DRAM utilization while maintaining acceptable computational efficiency for each algorithm.
3Loss of time
If GPU operations are executed sequentially in arrival order, then latency of earlier operations is reduced, but average latency for multiple applications increases
Solution Approach 1:
The GPU scheduler implements feedback mechanisms to monitor kernel execution progress, resource usage, and completion status. Based on this feedback, the scheduler dynamically adjusts the execution order and resource allocation of subsequent kernels, allowing it to optimize for average latency by interleaving executions of multiple applications rather than strictly following arrival order.
Solution Approach 2:
The system performs preliminary scheduling and resource allocation decisions before kernel execution begins. By pre-configuring block allocations and identifying potential concurrent execution opportunities, the scheduler can prepare resource distribution strategies that minimize average latency across multiple applications while ensuring that early-arriving kernels still receive timely execution.
Data Source
AI summary
Computing resources are optimally allocated for a multipath neural network using a multipath neural network analyzer that includes an interface and a processing device. The interface receives a multipath neural network that includes two or more paths. A first path includes one or more layers. A first layer of the first path corresponds to a first kernel that runs on a compute unit that includes two or more cores. The processing device allocates to the first kernel a minimum number of cores of the compute unit and a maximum number of cores of the compute unit. The minimum number of cores of the compute unit is allocated based on the first kernel being run concurrently with at least one other kernel on the compute unit and the maximum number of cores of the compute unit is allocated based on the first kernel being run alone on the compute unit.