GPU Rendering Pipeline Inference for Texture I/O Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI inference methods on mobile devices incur high I/O overheads due to the need for data format conversion between texture and SSBO data types, which cannot be directly performed on the GPU, requiring CPU intervention and increasing memory usage.
Innovation Solution
Implement model inference using a graphics rendering pipeline that directly processes texture data on the GPU, avoiding the need for data format conversion by utilizing a GPU rendering pipeline to perform inference and converting the inference result back to texture format, thereby reducing I/O memory overheads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data format conversion between texture and SSBO is performed on the CPU, then model inference can be executed, but I/O overheads increase significantly
Solution Approach 1:
The patent introduces a shader storage buffer object (SSBO) as an intermediary data structure that allows texture data to be processed directly by the GPU without CPU intervention. The SSBO serves as a mediator between the texture format and the neural network inference engine, enabling format conversion and inference execution to occur entirely on the GPU side, thereby eliminating the need for CPU-based data transfer and conversion operations.
Solution Approach 2:
The patent replaces the mechanical CPU-based data conversion and transfer mechanism with a GPU-based shader program mechanism. Instead of using the CPU to read texture data, convert it to SSBO format, and transfer it to the GPU, the system uses GPU shaders to directly process texture data in the desired format, substituting a more efficient parallel processing mechanism for the sequential CPU operations.
2Reliability
If data format conversion is performed on the CPU, then correct data format for inference is achieved, but memory usage increases
Solution Approach 1:
The SSBO acts as an intermediary buffer that resides in GPU memory, allowing texture data to be converted to the required format directly in GPU memory space. This eliminates the need to maintain multiple copies of data in CPU memory and GPU memory simultaneously, reducing overall memory usage while ensuring data format correctness through the shader-based conversion process.
Solution Approach 2:
The patent changes the memory location parameter where data format conversion occurs, moving it from CPU memory to GPU memory. By performing conversion operations in GPU memory through shaders, the system avoids the memory overhead associated with CPU-based conversion, as the converted data remains in GPU memory and does not require additional CPU memory allocation.
3Productivity
If multiple data transfers between CPU and GPU are performed, then inference computation can be executed, but energy efficiency decreases
Solution Approach 1:
The patent enables continuous GPU operation by keeping texture data in the GPU's address space throughout the inference process. The shader programs continuously process the data without interruption from CPU data transfers, maintaining continuous useful action on the GPU side. This eliminates the idle periods and energy-wasting data transfers that would otherwise occur when moving data between CPU and GPU memory spaces.
Solution Approach 2:
The patent substitutes the energy-intensive CPU-GPU data transfer mechanism with an energy-efficient GPU-only processing mechanism. By using shader programs to handle all data conversion and processing operations within the GPU, the system eliminates the need for repeated data transfers between CPU and GPU, significantly reducing the energy consumption associated with memory access and data movement operations.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
This application provides a model inference method and apparatus based on a graphics rendering pipeline, and a storage medium. The method includes: obtaining an instruction stream in a render thread; extracting and saving texture data information from the instruction stream, where the texture data information includes texture data; and inputting the texture data information to a graphics processing unit GPU rendering pipeline, where the GPU rendering pipeline is used to perform GPU model-based inference on the texture data to obtain an inference result of the texture data, and the GPU model is a model running in a GPU. This implements model inference on the texture data, and avoids conversion of the texture data into another data type, thereby reducing I/O memory overheads.