Thread Block Preloading With Split Compute Units for GPU Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The time delay in loading data from external storage space significantly affects the execution efficiency of graphics processing units (GPUs) due to serial execution of data loading operations, which cannot be hidden during data calculation.
Innovation Solution
Decompose a thread block into two sub-thread blocks, with one block obtaining data and the other executing tasks, utilizing independent computing units within a GPU to pre-load data from external storage into a local share memory, reducing the time delay by parallelizing data loading and calculation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional GPU architectures are used for neural network training, then general-purpose computing capability is maintained, but training efficiency and speed are insufficient for large-scale models
Solution Approach 1:
The patent divides the computing system into specialized units: tensor processing units (TPUs) for neural network operations, separate from traditional CUDA cores. This segmentation allows optimized hardware paths for different computational workloads, improving training efficiency without requiring complete system redesign.
Solution Approach 2:
The patent introduces a memory hierarchy with specialized buffers and caches between the CPU, GPU, and storage systems. This intermediary memory structure reduces data transfer bottlenecks and enables faster access to training data, directly improving training throughput.
2Productivity
If more computing resources are allocated to accelerate training, then training speed improves, but system cost and power consumption increase
Solution Approach 1:
The patent implements dynamic precision adjustment for different computation types, using lower precision (e.g., FP16, INT8) for operations where high precision is not critical. This parameter change reduces computational complexity and power consumption while maintaining adequate model training quality.
Solution Approach 2:
The patent employs mixed-precision training where only critical parts of the computation graph use full precision, while other parts use reduced precision. This partial application of high-resource operations achieves acceptable training speed improvement without proportional power consumption increase.
3Speed
If data is stored in CPU memory for fast access during training, then data access speed improves, but available storage capacity is limited
Solution Approach 1:
The patent introduces a multi-level memory hierarchy that extends storage capacity beyond CPU memory by incorporating GPU memory, HBM (High Bandwidth Memory), and persistent storage layers. This dimensional expansion of the memory architecture provides both large capacity and fast access paths through hierarchical data movement.
Solution Approach 2:
The patent implements data preloading and caching mechanisms that prepare training data in advance in faster memory layers before it is needed for computation. This preliminary action reduces data access latency during actual training operations without requiring all data to reside in fast memory simultaneously.
4Ease of manufacture
If traditional CUDA programming model is used, then compatibility with existing software is maintained, but programming complexity and development time increase for optimized performance
Solution Approach 1:
The patent maintains the CUDA programming interface as a universal layer that can address both traditional GPU operations and specialized TPU operations. This multi-functional interface allows developers to write code once and achieve both compatibility and optimized performance without learning separate programming models.
Solution Approach 2:
The patent introduces a software layer that mediates between the standard CUDA API and the specialized hardware operations. This intermediary software handles the complexity of optimized operations transparently, allowing developers to benefit from high performance without directly managing the underlying complexity.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
The embodiments of the present disclosure provide a processing system of a thread block, a method and a relative device. The processing system includes: a first computing unit for running the first sub-thread block and a second computing unit for running the second sub-thread block; the first computing unit is used for obtaining the data to be processed of the thread block, and the second computing unit is used for executing the processing task of the thread block according to the data to be processed obtained by the first computing unit. The processing system of the thread block provided by the embodiments of the present disclosure can effectively reduce the time delay of loading data to be processed, especially reduce the time delay of loading data to be processed from external storage space, and improve the processing efficiency of thread blocks.