Foveated Camera Binning and Region of Interest Compositing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Augmented reality head-mounted displays face challenges in processing high-resolution images due to limited computing power and battery capacity, making it impractical to use high-resolution cameras like those required for 50 megapixels, while still seeking to provide an immersive experience with high-resolution images.
Innovation Solution
Implementing a foveated camera system that uses binning and region of interest capture, where the camera captures a full frame at reduced resolution and focuses on a high-resolution region of interest, which is then composited with the binned image, along with late-stage re-projection and feathering to enhance the image quality and reduce computational load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a 50 megapixel camera is used to provide high resolution images for immersive augmented reality, then image resolution is improved, but processing power requirements and battery consumption increase tremendously
Solution Approach 1:
The patent applies local quality by capturing high-resolution images only in the foveal region (center of visual attention) while using lower resolution for peripheral regions. This is achieved through eye tracking to identify the fovea, then using a region of interest mask to guide the camera to capture detailed images only where the user is looking, rather than uniformly across the entire field of view.
Solution Approach 2:
The patent segments the visual field into different resolution zones based on eye tracking data. The foveal region is segmented as high-priority for high-resolution capture, while peripheral regions are segmented as low-priority for lower-resolution capture. This segmentation allows the system to allocate computational resources selectively rather than processing the entire image at high resolution.
2Measurement precision
If a 50 megapixel camera is used to provide high resolution images for immersive augmented reality, then image resolution is improved, but computing power requirements increase tremendously
Solution Approach 1:
The system applies local quality by processing high-resolution data only for the foveal region while using lower-resolution processing for peripheral regions. The region of interest mask derived from eye tracking data enables the processing pipeline to focus computational effort on the small central area where high resolution is actually needed, dramatically reducing overall processing requirements.
Solution Approach 2:
The patent segments the image processing task into high-priority foveal regions and low-priority peripheral regions. By using eye tracking to identify which regions require high-resolution processing, the system can allocate computing power selectively, processing only the necessary portions at high resolution while using simpler processing for the rest of the field of view.
3Power
If binning mode is used to reduce processing load, then processing power consumption is reduced, but image resolution decreases
Solution Approach 1:
The patent applies local quality by using binning mode selectively only in peripheral regions where lower resolution is acceptable, while maintaining full resolution in the foveal region. The region of interest mask enables the system to apply different processing modes to different spatial locations, combining binning for power efficiency with full-resolution capture where needed.
Solution Approach 2:
The system dynamically adjusts the camera's operating mode based on real-time eye tracking data. The region of interest mask is continuously updated as the user's gaze moves, allowing the system to dynamically switch between binning mode and full-resolution mode in different regions of the field of view, optimizing the balance between power consumption and image quality adaptively.
Data Source
AI summary
Using a foveated camera for video augmented reality via a head mounted display is provided. A camera may be placed in a binning mode; a full frame binned image may be captured using the camera and the binning mode of the camera may be turned off. A region of interest may be selected within the full frame; the region of interest may be captured at a higher resolution than the resolution of the camera when in binning mode; the region of interest may be composited with the full frame binned image; and the composited image may be displayed.