Adaptive Foveation Processing and Rendering for Low-Latency VST XR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
VST XR systems face challenges such as high computational load and latency due to processing and rendering of high-resolution images, leading to user discomfort and motion sickness.
Innovation Solution
Adaptive foveation processing and rendering techniques that identify the user's focus region, generate masks with varying resolutions and shapes, and map image data onto a mesh to reduce computational load and latency, enhancing image quality where the user focuses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If high-resolution images are processed and rendered in VST XR systems, then image quality is improved, but computational load and latency increase
Solution Approach 1:
The patent applies local quality by rendering different regions of the display at different resolutions. The foveal region (center of vision) is rendered at high resolution while peripheral regions are rendered at lower resolution. This is achieved through eye-tracking data to identify the foveal region, then selectively applying high-resolution rendering only to that area, thereby reducing overall computational load and latency while maintaining perceived image quality where the user actually looks.
2Manufacturing precision
If high-resolution images are processed and rendered in VST XR systems, then image quality is improved, but computational load increases
Solution Approach 1:
The patent reduces computational load by applying local quality principles - only the foveal region is processed at high resolution while peripheral regions use lower resolution. This selective processing significantly reduces the number of pixels that require high-computation rendering operations, thereby lowering power consumption and computational load while maintaining image quality in the visually critical foveal region.
Solution Approach 2:
The patent segments the display area into different resolution zones based on eye-tracking data. The foveal region is segmented as a high-resolution zone while surrounding areas are segmented as lower-resolution zones. This segmentation allows the system to allocate computational resources efficiently, processing only the necessary high-resolution data for the foveal region rather than uniformly processing the entire display at high resolution.
3Power
If adaptive foveation processing is applied, then processing load is reduced, but system complexity increases
Solution Approach 1:
The patent employs eye-tracking functionality that can serve multiple purposes - not only identifying the foveal region for adaptive resolution rendering but also potentially providing gaze-based interaction cues and user attention data. This multi-functionality justifies the added complexity by providing additional system capabilities beyond just resolution adaptation.
Data Source
AI summary
A method includes obtaining, using at least one processing device, images of a scene captured using one or more imaging sensors of a video see-through (VST) extended reality (XR) device. The method also includes identifying, using the at least one processing device, a region of the scene on which a user is focused. The method further includes generating, using the at least one processing device, a mask for each image based on the region of the scene on which the user is focused, where different masks are associated with different resolutions and/or different shapes. The method also includes mapping, using the at least one processing device, at least some image data of each image onto a mesh based on the mask associated with that image. In addition, the method includes rendering, using the at least one processing device, final views of the scene using the mapped image data.