Foveated Rendering for Head-Mounted Displays

Solution Overview

Problem

Artificial reality systems face challenges in reducing power consumption and computational resource usage due to demanding graphic rendering processes for full-resolution display content, particularly in head-mounted displays with limited battery power and resources.

Innovation Solution

Implementing a foveated rendering process that renders display content with full resolution in the user's foveal region and reduced resolutions in surrounding areas, casting fewer rays for tile/surface pairs and using lower sampling resolutions for color channels in non-foveal regions, thereby reducing computational resources and memory reading bandwidth.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If full-resolution rendering is used for the entire display, then image quality is maintained, but power consumption and computational resource usage increase

Engineering Contradiction:

Improveimage qualityVSAvoidpower consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent applies local quality by differentiating rendering resolution across different regions of the display based on user gaze location. The foveal region receives full-resolution rendering to maintain image quality where the user is looking, while peripheral regions use reduced resolution to lower computational load and power consumption. This spatially adaptive approach resolves the contradiction by concentrating computational resources where they are most needed for perceived quality.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent segments the display into multiple regions (foveal and peripheral regions) based on distance to the user's gazing point. Each region is rendered at different sampling resolutions, allowing the system to maintain high quality in the foveal region while reducing computational expenditure in peripheral regions. This segmentation enables the system to balance image quality and power consumption across the entire display.

Inventive Principle:

Principle #1Segmentation

2Measurement precision

If full-resolution rendering is used for the entire display, then image quality is maintained, but computational resource usage increases

Engineering Contradiction:

Improveimage qualityVSAvoidcomputational resource usage

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent applies local quality by differentiating rendering resolution across different regions of the display based on user gaze location. The foveal region receives full-resolution rendering to maintain image quality where the user is looking, while peripheral regions use reduced resolution to lower computational load and power consumption. This spatially adaptive approach resolves the contradiction by concentrating computational resources where they are most needed for perceived quality.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent segments the display into multiple regions (foveal and peripheral regions) based on distance to the user's gazing point. Each region is rendered at different sampling resolutions, allowing the system to maintain high quality in the foveal region while reducing computational expenditure in peripheral regions. This segmentation enables the system to balance image quality and power consumption across the entire display.

Inventive Principle:

Principle #1Segmentation

3Productivity

If reduced sampling resolution is used in peripheral regions, then computational resources and memory bandwidth are reduced, but image quality in peripheral areas deteriorates

Engineering Contradiction:

Improvecomputational resource usageVSAvoidimage quality in peripheral regions

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent applies local quality by differentiating rendering resolution across different regions of the display based on user gaze location. The foveal region receives full-resolution rendering to maintain image quality where the user is looking, while peripheral regions use reduced resolution to lower computational load and power consumption. This spatially adaptive approach resolves the contradiction by concentrating computational resources where they are most needed for perceived quality.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent applies partial action by providing full-resolution rendering only to the foveal region rather than the entire display. The peripheral regions receive reduced-resolution rendering, which is sufficient for the user experience since human visual acuity is lower in peripheral vision. This partial application of full resolution achieves the desired balance between computational resources and perceived image quality.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS11862128B2Systems and methods for foveated rendering

Publication Date: 2024.01.02 META PLATFORMS TECHNOLOGIES LLC

AI summary

In one embodiment, a computing system may determine a focus point of a viewer based on received sensor data. The system may determine, for a current frame, a first viewing region encompassing a focus point of the viewer and a second view region excluding the first viewing region. The system may determine, for the current frame, color values for the first viewing region using respective first sampling resolutions, and color values for the second viewing region using respective second sampling resolutions. At least one second sampling resolution may be lower than a corresponding first sampling resolution associated with a same color channel. At least two of the second sampling resolutions for the color channels of the second viewing region may be different from each other. The system may output the color values for the first viewing region and the second viewing region of the current frame for display.