AI Chip Thread Management for GPU Parallel Processing Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Heterogeneous hardware platforms face challenges in efficiently processing large batch sizes of data due to difficulties in finer-grained splitting of model inference tasks, particularly when utilizing a combination of central processing units (CPUs) and graphic processing units (GPUs, where the GPU's parallel processing capability is not fully utilized.
Innovation Solution
An apparatus and method that determine an initial number of threads based on the data amount and storage unit capacity, allowing for efficient execution of tasks in parallel by utilizing the 3rd-level high-speed dynamic random access memory (DRAM) to store input, weight, and output data, thereby optimizing bandwidth and capacity usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model inference tasks are processed on heterogeneous hardware platforms (CPU+GPU), then processing capacity is improved, but finer-grained splitting of tasks becomes difficult and parallel processing capability of GPU is not fully utilized
Solution Approach 1:
The patent segments model inference tasks into multiple independent sub-tasks that can be executed in parallel. By dividing the input data into batches and creating corresponding inference tasks, the system enables fine-grained parallel processing on GPU while maintaining manageable task complexity through systematic segmentation strategies.
Solution Approach 2:
The patent introduces a new dimension of task organization by mapping tasks to a two-dimensional grid structure (batch dimension × sequence dimension). This dimensional transformation enables more flexible task scheduling and resource allocation, allowing better utilization of GPU parallel processing capabilities while simplifying the complexity of task management.
2Productivity
If the initial number of threads is increased to process larger data amounts, then processing capacity is improved, but thread management complexity and resource allocation difficulty increase
Solution Approach 1:
The patent implements dynamic thread number adjustment based on real-time data amount and storage unit capacity. The system calculates the optimal initial number of threads using the formula considering available memory resources, and dynamically adapts thread configuration during task execution. This dynamic approach enables the system to handle varying data scales efficiently while maintaining manageable thread management complexity through automated resource assessment and adaptive allocation.
Data Source
AI summary
An apparatus and a method of processing data, an electronic device, and a storage medium are provided, which relate to a field of artificial intelligence, and in particular to fields of chip and multi-thread parallel technologies. The apparatus includes: a first target storage unit; and a processor configured to: determine an initial number of threads according to a data amount of target data and a capacity of the first target storage unit in response to determining that the data amount is less than or equal to the capacity of the first target storage unit, where the target data includes input data to be processed, weight data to be processed, and output data; and determine a first number of executable tasks according to the initial number of threads in response to determining that the initial number of threads is greater than or equal to a predetermined number of threads.