Distributed Foveated Rendering for HMD Bandwidth Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face challenges in efficiently rendering content on multiple devices due to data transmission limitations, such as bandwidth constraints and encoder limitations, leading to distorted displays in head-mounted devices (HMDs) when using other devices like laptops or smartphones.
Innovation Solution
Implementing foveated rendering techniques that adapt image resolution based on user gaze, generating warped images with non-uniform pixel distributions to reduce data requirements, allowing for efficient data compression and transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-resolution image data is transmitted from the second device to the first device, then image quality is maintained, but bandwidth consumption and computational demands increase
Solution Approach 1:
The patent applies local quality by transmitting only the region of interest (ROI) at full resolution while transmitting the rest of the image at reduced resolution. The first device identifies the ROI based on gaze data or content importance, then selectively processes and transmits different quality levels for different regions of the image, optimizing both bandwidth usage and visual quality where it matters most.
Solution Approach 2:
The image is segmented into multiple regions including the region of interest and surrounding regions. Each region is processed independently with appropriate resolution levels. The segmentation allows the system to divide the full-resolution data transmission task into manageable parts, transmitting only necessary high-resolution data for the ROI while using lower resolution for peripheral areas.
2Measurement precision
If image processing is performed on the first device (HMD), then display quality is improved, but computational complexity and processing time increase
Solution Approach 1:
The second device performs preliminary actions by pre-processing the image to identify the region of interest and pre-calculating the warped image data before transmission. This shifts computational burden to the source device, allowing the first device (HMD) to receive pre-processed data that requires minimal additional processing, thus reducing its computational complexity while maintaining display quality.
Solution Approach 2:
The patent extracts only the necessary image data (region of interest) from the full image before transmission. By taking out and transmitting only the relevant portion at full resolution along with warp parameters, the system reduces the amount of data the first device needs to process, thereby reducing computational complexity while preserving display quality for the critical regions.
3Quantity of substance
If warp parameters are transmitted to enable foveated rendering, then bandwidth efficiency is improved, but image distortion may occur if not properly corrected
Solution Approach 1:
The system uses feedback through gaze tracking data to dynamically adjust the region of interest and warp parameters. The first device receives gaze information from the second device and uses this feedback to accurately determine which regions require high-resolution transmission. This feedback mechanism ensures that the warping and resolution adjustment are precisely targeted, maintaining image accuracy in the ROI while optimizing bandwidth efficiency.
Data Source
Figure 1
Figure 2
Figure 3A~3D
AI summary
In one implementation, a method of displaying a display image is performed by a first device having a first display, one or more processors, and non-transitory memory. The method includes obtaining gaze data indicative of a gaze of a user. The method includes transmitting, to a second device having a second display, warp parameters based on the gaze data. The method includes receiving, from the second device, display data based on the warp parameters. The method includes displaying, on the first display, a display image based on the display data.