Foveated Rendering Latency Reduction via Multi-Resolution Frame Reprojection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems for augmented and virtual reality head-mounted displays face challenges in efficiently rendering images that accurately reflect the user's head orientation changes during the rendering process, leading to latency issues and inefficient resource utilization.
Innovation Solution
A method and system that generate plural different resolution versions of frames, where the display processor reads and combines these versions based on data indicating which resolution to use for respective regions, transforming the frames to match the predicted head orientation, thereby reducing latency and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If frames are re-projected based on predicted head orientation to match user's current head orientation, then display accuracy is improved, but rendering time and latency increase
Solution Approach 1:
The system pre-renders multiple resolution versions of frames before the actual display moment. These pre-rendered versions are prepared in advance at different resolutions, allowing the display processor to quickly select and combine appropriate versions based on predicted head orientation, avoiding time-consuming real-time re-projection operations.
Solution Approach 2:
The frame rendering is divided into multiple resolution versions (e.g., high resolution and low resolution portions). The display processor selectively combines these segmented versions based on the predicted head orientation and foveated rendering requirements, processing only necessary portions at high resolution while using lower resolution for peripheral areas.
2Productivity
If multiple different resolution versions of frames are generated and combined, then rendering efficiency is improved, but device complexity increases
Solution Approach 1:
The display processor is designed to perform multiple functions: it can read from multiple different resolution versions, selectively combine them based on region and orientation, and output the final composite frame. This multi-functional processor handles various rendering scenarios (different resolutions, orientations, and foveated regions) using a single integrated unit, managing complexity through functional consolidation.
3Use of energy by moving object
If the display processor reads and combines multiple resolution versions based on predicted head orientation, then resource utilization is improved, but processing complexity increases
Solution Approach 1:
The system applies different quality levels (resolutions) to different regions of the frame based on where the user is likely to look. High resolution is applied only to the foveal region (center of gaze) while peripheral regions use lower resolution. This local differentiation optimizes resource usage by concentrating computational resources on visually critical areas only.
Solution Approach 2:
Instead of rendering the entire frame at high resolution, the system performs partial rendering actions by generating only the necessary high-resolution portions corresponding to the predicted foveal region, while using pre-rendered lower resolution versions for the remainder of the frame, reducing overall processing requirements.
Data Source
AI summary
In a data processing system, when displaying a foveated image, a producer processing unit generates plural different resolution versions of the frame to be displayed. A display processor then generates a view orientation transformed output version of the frame to be displayed using data from the plural different resolution versions of the frame to be displayed generated by the producer processing unit based on data indicative of which resolution version of the frame is to be used for respective regions of the view orientation transformed output version of the frame to be displayed provided to the display processor.