Tile-Based Image Rendering With GPU–NPU Pipeline Staging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image rendering technologies face challenges in efficiently handling complex computations required for graphics processing and artificial intelligence tasks, particularly in converting three-dimensional models into image frames, due to the lack of optimal integration between graphic processing units (GPUs) and neural processing units (NPUs).
Innovation Solution
A method and apparatus that utilize a GPU for generating a first-rendering result in tile units, followed by a NPU for performing a second-rendering operation, where the first and second operations are specialized in scalar/vector and matrix operations respectively, with a pipeline architecture allowing partial rendering results to be generated and stored before complete input data processing is finished.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If image rendering is performed using only a GPU with scalar/vector operations, then the rendering process is simple to implement, but the computational efficiency and power consumption are suboptimal for complex AI-based rendering tasks
Solution Approach 1:
The rendering process is segmented into multiple stages: first rendering by GPU generates initial image data, then selective tiles are processed by NPU for AI-based enhancements like super-resolution. This segmentation allows complex AI operations to be applied only where needed rather than to the entire image, improving computational efficiency while managing pipeline complexity.
Solution Approach 2:
The patent introduces an intermediary control mechanism that manages data flow between GPU and NPU. The control logic determines which tiles require NPU processing based on complexity metrics, acting as a mediator that coordinates the heterogeneous processing units and optimizes the pipeline without requiring full complexity of AI processing for all image data.
2Measurement precision
If the entire first-rendering result is processed by the NPU before second rendering, then the NPU can operate on complete data, but the memory usage and latency increase
Solution Approach 1:
The GPU performs first rendering in advance to generate initial image data and identify complex tiles that require AI processing. This preliminary action allows the system to prepare and select only necessary tiles for NPU processing, rather than waiting for complete image generation, thereby reducing latency while maintaining rendering quality.
Solution Approach 2:
The patent applies partial processing by selecting only complex tiles for NPU-based second rendering, rather than processing the entire image. This partial action approach maintains high rendering quality for critical areas while reducing overall processing time and latency compared to comprehensive AI processing of all image data.
3Manufacturing precision
If high-resolution rendering is performed for the entire image, then the image quality is improved, but the power consumption and computational load increase significantly
Solution Approach 1:
The patent applies different processing qualities to different regions of the image based on complexity analysis. High-resolution AI-based second rendering is applied locally only to complex tiles that require enhanced detail, while simpler tiles use standard rendering. This local quality approach improves overall image quality where needed while significantly reducing power consumption compared to uniform high-resolution processing of the entire image.
Data Source
AI summary
A method and an apparatus with image rendering are provided. A method includes generating a first-rendering result corresponding to input graphic data of a rendering target, wherein the first-rendering result is rendered in units of first tiles by performing first-rendering including a first type of operation using a first processing unit, storing, in memory, the first-rendering result as units of first tiles, generating a second-rendering result of second tiles by, for each first tile of the first-rendering result, performing second-rendering including a second type of operation using a second processing unit, storing, in the memory, the second-rendering result as units of second tiles, and determining a final rendering result based on the second-rendering result.