Graphics Processing System Foveated Rendering Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graphics processing systems that perform remote rendering face significant latency issues, particularly in gaming and virtual reality applications, due to the need to generate frames in real-time, which can lead to a poor user experience, especially when the user's location is far from the cloud server.
Innovation Solution
The graphics processing system generates frame sequences in advance for multiple possible future states of an application, with regions of the frame being rendered at different qualities, such as higher quality for the foveal region and lower quality for peripheral regions, to reduce latency and processing burden.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If frames are generated in real-time for remote display, then processing can be performed remotely in the cloud, but input latency increases significantly
Solution Approach 1:
The system generates frame sequences in advance for multiple possible future states before they are actually needed. By pre-rendering frames for various potential application states and storing them locally, the system eliminates the need to wait for real-time cloud processing when the user interacts with the application, thus reducing input latency while maintaining remote processing capabilities.
2Manufacturing precision
If high-quality frames are generated for all regions, then rendering quality is improved, but processing power and bandwidth requirements increase
Solution Approach 1:
The system divides the frame into different regions and applies different quality levels to each region. The foveal region (where users are most likely to look) is rendered at high quality, while peripheral regions are rendered at lower quality. This selective quality approach maintains acceptable overall rendering quality while significantly reducing processing power and bandwidth requirements.
Solution Approach 2:
Instead of rendering all regions at maximum quality, the system applies quality rendering only to the necessary foveal region. This partial action approach provides sufficient visual quality for user interaction while avoiding the excessive processing and bandwidth costs of rendering the entire frame at high quality.
3Loss of time
If frame sequences are generated in advance for multiple future states, then input latency is reduced, but memory requirements increase
Solution Approach 1:
The system pre-generates and stores frame sequences for multiple possible future states in local memory. When a user interaction occurs, the system can immediately retrieve the pre-computed frame sequence corresponding to the actual future state, eliminating latency. The memory trade-off is acceptable because the pre-stored frames are compact and the alternative (real-time cloud processing) would be too slow.
Solution Approach 2:
The system generates frame sequences for multiple possible future states (more than strictly necessary) to ensure that the correct sequence is available when needed. This excessive preparation covers uncertainty about which future state will actually occur, reducing latency at the cost of increased memory usage for storing multiple frame sequences.
Data Source
AI summary
When a graphics processor is processing data for an application on a host processor, the graphics processor generates in advance of their being required for display by the application a plurality of frame sequences corresponding to a plurality of different possible “future states” for the application. The graphics processing system, when producing a frame in a sequence of frames corresponding to a given future state for the application, determines one or more region(s) of the frame that are to be produced at a first, higher quality, and producing the determined region(s) of the frame at a first, higher quality, whereas other regions of the frame are produced at a second, lower quality.