Foveated Sensing for Extended Reality Bandwidth Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extended reality (XR) devices face challenges in efficiently processing high-resolution images and videos, leading to increased bandwidth and power consumption, which can limit the performance of visual see-through (VST) applications.
Innovation Solution
The implementation of foveated sensing systems, which use gaze prediction algorithms to determine the region of interest (ROI) in a scene, allowing for higher resolution capture and processing of central areas while reducing resolution in peripheral areas, thereby conserving bandwidth and power.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-resolution image capture and processing is performed across the entire frame, then image quality is improved, but bandwidth consumption and power consumption increase
Solution Approach 1:
The patent applies local quality by capturing and processing the ROI (region of interest) at high resolution while capturing peripheral regions at lower resolution. This is achieved through the foveated sensing system that identifies the ROI based on gaze prediction and processes only that specific region with full image quality, thereby reducing overall power consumption while maintaining perceived image quality in the important region.
Solution Approach 2:
The patent segments the image frame into two distinct portions: the ROI (first portion) and peripheral regions (second portion). Each portion is processed differently - the ROI receives high-resolution capture and full processing, while peripheral regions receive reduced-resolution capture. This segmentation allows the system to optimize resource allocation and reduce bandwidth consumption without sacrificing critical image quality.
2Measurement precision
If high-resolution image capture and processing is performed across the entire frame, then image quality is improved, but computational complexity increases
Solution Approach 1:
The system applies local quality by concentrating computational resources on processing only the ROI at high resolution, while using simplified processing for peripheral regions. The foveated sensing system determines which region requires full computational processing based on gaze prediction, thereby reducing overall computational complexity while maintaining high image quality where it matters most.
Solution Approach 2:
The patent segments the processing workload into two parts: full-resolution processing for the ROI and reduced-resolution processing for peripheral regions. This segmentation of computational tasks significantly reduces the total computational complexity required, as the majority of the frame does not require intensive processing, thereby simplifying the device requirements.
3Measurement precision
If high-resolution image capture and processing is performed across the entire frame, then image quality is improved, but bandwidth consumption increases
Solution Approach 1:
The patent applies local quality by transmitting and processing high-resolution image data only for the ROI region, while transmitting reduced-resolution data for peripheral regions. This approach significantly reduces the total bandwidth consumption, as high-quality data is transmitted only for the specific region that requires it, rather than for the entire frame.
Solution Approach 2:
The patent segments the image data into two portions with different resolution requirements. The ROI (first portion) is captured and transmitted at high resolution, while the peripheral regions (second portion) are captured and transmitted at lower resolution. This segmentation reduces the total quantity of data that needs to be transmitted over the bandwidth, thereby reducing bandwidth consumption while maintaining necessary image quality.
Data Source
AI summary
Systems and techniques are described for performing foveated sensing. In some aspects, a method (e.g., implemented by an image sensor) can include capturing sensor data for a frame associated with a scene, obtaining information corresponding to a region of interest (ROI) associated with the scene, generating a first portion (having a first resolution) of the frame corresponding to the ROI, generating a second portion of the frame having a second resolution, and outputting (e.g., to an image signal processor (ISP)) the first portion and the second portion. In some aspects, a method (e.g., implemented by an ISP) can receive, from an image sensor, sensor data for a frame associated with a scene, generating a first version (having a first resolution) of the frame based on an ROI associated with the scene, and generating a second version of the frame having a second resolution (lower than the first resolution).