GPU On-Chip Memory Preloading for Faster Model Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional CPU hardware devices struggle to meet the computing needs of machine learning and deep learning tasks, particularly in large model inference phases, where the performance bottleneck is not model computing but the data loading time from GPU global memory to computing units.
Innovation Solution
Implementing a data processing method that utilizes two GPU threads, where one thread loads writable data and another thread preloads read-only data from GPU global memory to on-chip memory, such as L2 cache, shared memory, or registers, to reduce data loading time and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data are loaded from GPU global memory to GPU computing unit during model inference, then model computing can be performed, but data loading time becomes the performance bottleneck reducing overall processing efficiency
Solution Approach 1:
The patent preloads read-only model parameter data from GPU global memory to GPU on-chip memory before the actual computing task executes. By using a second GPU thread to perform data preloading in advance, the system ensures that data are ready in fast on-chip memory when the computing thread needs them, eliminating data loading delays during computation.
Solution Approach 2:
The patent divides data into two categories: writable data loaded by the first GPU thread and read-only model parameter data loaded by the second GPU thread. This segmentation allows different data types to be handled by specialized threads, with read-only data being preloaded in advance to optimize access patterns and reduce bottlenecks.
2Speed
If GPU global memory is used to store model parameter data, then large capacity is available, but access speed is insufficient creating a performance bottleneck
Solution Approach 1:
The patent employs a hierarchical memory structure where GPU on-chip memory (faster, smaller) is nested within the GPU memory system that includes GPU global memory (slower, larger). Read-only model parameter data are transferred from the outer layer (global memory) to the inner layer (on-chip memory) for faster access during computation, while writable data remain in or are transferred from global memory as needed.
3Productivity
If single GPU thread is used for both data loading and computing, then system complexity is reduced, but data loading time delays computing execution
Solution Approach 1:
The patent merges data loading and computing operations into a coordinated multi-threaded system where a first GPU thread loads writable data and a second GPU thread preloads read-only data. This combining of multiple functions into an integrated thread coordination system allows overlapping of data loading and computing preparation, improving throughput despite increased thread management complexity.
Data Source
AI summary
Methods, apparatuses, and systems for data processing based on graphics processing unit (GPU) on-chip memories are described. A data obtaining operation for first data is initiated on a first GPU thread. The first data include writable data needed by a GPU computing task. When the first GPU thread performs the data obtaining operation, a data preloading process of preloading second data from a GPU global memory to the GPU on-chip memory is initiated on a second GPU thread. The second data include read-only data that are needed by the GPU computing task and that are stored in the GPU global memory. The GPU computing task is executed on the second GPU thread based on the first data and the second data in response to that a data obtaining process of the first data and the data preloading process of the second data are completed.