Foveated Processing for Holographic XR Displays
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current extended reality (XR) devices face challenges in efficiently managing processing power and battery life due to the need for high-quality image rendering across entire frames, which is not necessary as users typically focus on specific regions, leading to increased energy consumption and reduced battery life.
Innovation Solution
Implementing a foveated processing system that uses an eye-tracking subsystem and neural networks to determine the focus region of a user's gaze, allowing for high-quality rendering only in the focus area and lower-quality rendering in peripheral regions, thereby reducing processing power and energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If high-quality image rendering is performed across the entire frame, then visual quality is improved, but energy consumption increases and battery life decreases
Solution Approach 1:
The patent applies local quality by rendering different regions of the display at different quality levels. The foveal region (where the user's gaze is focused) is rendered at high quality, while peripheral regions are rendered at lower quality. This resolves the contradiction by maintaining high visual quality where needed (improving image rendering quality) while reducing processing requirements in less critical areas (decreasing energy consumption).
Solution Approach 2:
The patent segments the display area into multiple regions based on the user's gaze position, specifically identifying a foveal region and peripheral regions. This segmentation allows the system to apply different rendering strategies to different segments, rendering high quality only in the foveal region and lower quality in peripheral regions, thereby reducing overall energy consumption while maintaining visual quality where the user is actually looking.
2Manufacturing precision
If high-quality image rendering is performed across the entire frame, then visual quality is improved, but processing power requirements increase
Solution Approach 1:
The system applies local quality by concentrating processing power on rendering the foveal region at high quality while using reduced processing for peripheral regions. This resolves the contradiction by maintaining high image rendering quality in the visually critical foveal area while significantly reducing the total processing power required compared to rendering the entire frame at high quality.
Solution Approach 2:
The display frame is segmented into a foveal region and peripheral regions based on eye tracking data. This segmentation enables the system to allocate processing power selectively, dedicating high processing resources to the foveal region where visual quality is most important and using minimal processing resources for peripheral regions, thereby reducing overall processing power requirements while maintaining visual quality where needed.
3Use of energy by moving object
If foveated processing is applied to holograms, then energy consumption is reduced, but visual quality outside the focus region deteriorates
Solution Approach 1:
The patent applies local quality by rendering holographic content at high quality specifically within the foveal region while using lower quality rendering for peripheral regions. This resolves the contradiction by maintaining high visual quality for holograms where the user is actually looking (in the foveal region) while reducing energy consumption through lower quality rendering in peripheral areas where visual acuity is naturally reduced.
Data Source
AI summary
An image processing system for an extended reality, XR, device comprising an eye-tracking subsystem, for determining a focus region of the eye, and a processor. The processor is configured to process application data to render image content for an application for display on the XR device, and obtain metadata indicating that a virtual object is to be generated as a hologram as part of the image content for display. Based on a determination that the virtual object belongs to a predetermined class of objects and is to be displayed in the focus region, the processor performs, using a neural network corresponding to the predetermined class of objects, foveated processing of the image content, including at least part of the hologram, such that relatively high-quality image content is generated for display in the focus region and relatively low-quality image content is generated for display outside the focus region.