Foveated Temporal Filtering for ROI Resolution and Compute Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixed-reality systems face challenges in efficiently performing temporal filtering and motion compensation on high-resolution images due to limited computational resources, particularly in mobile devices like HMDs, leading to reduced angular pixel resolution and difficulty in identifying distant objects.
Innovation Solution
Implementing foveated temporal filtering and motion compensation techniques that utilize a dual layer representation of image content, focusing on maintaining high resolution in the region of interest while using lower resolution elsewhere, and performing image processing in parallel to optimize computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If temporal filtering is performed on high-resolution images to enhance scene information, then image quality is improved, but computational resources are exceeded and processing time increases
Solution Approach 1:
The image is divided into two resolution layers: a high-resolution base layer and a low-resolution detail layer. Temporal filtering is applied selectively - the base layer undergoes full temporal filtering to maintain angular pixel resolution, while the detail layer uses reduced filtering to save computational resources. This segmentation allows the system to achieve high image quality where needed while maintaining overall computational efficiency.
Solution Approach 2:
Different regions of the image are assigned different quality levels. The base layer maintains high resolution and undergoes comprehensive temporal filtering to preserve angular pixel resolution for critical viewing areas. The detail layer uses lower resolution and reduced filtering for less critical regions, optimizing the trade-off between image quality and computational cost in each local area.
2Measurement precision
If temporal filtering is applied to capture sufficient scene information in low light conditions, then image interpretability is improved, but processing time increases
Solution Approach 1:
The temporal filtering process is segmented into two parallel processing paths: one for the high-resolution base layer and one for the low-resolution detail layer. This allows scene information capture to proceed simultaneously at different resolution levels, reducing total processing time while still capturing sufficient scene information through the combined output of both layers.
Solution Approach 2:
Instead of applying full temporal filtering to the entire high-resolution image (excessive action), the system applies temporal filtering partially - to the base layer at full resolution and to the detail layer at reduced resolution. This partial application achieves sufficient scene information capture without the excessive processing time that would result from full filtering of the complete high-resolution image.
3Manufacturing precision
If motion compensation is performed on all image frames to maintain alignment, then image accuracy is improved, but computational load increases
Solution Approach 1:
Motion compensation is segmented and applied differently to the base layer and detail layer. The base layer receives full motion compensation to maintain alignment accuracy for critical viewing areas, while the detail layer receives reduced or selective motion compensation. This segmentation maintains image alignment accuracy where it matters most while reducing overall computational load.
Solution Approach 2:
Different levels of motion compensation are applied to different regions. The base layer receives comprehensive motion compensation for high accuracy in critical areas, while the detail layer uses reduced motion compensation for less critical regions, optimizing the balance between alignment accuracy and computational load in each local area.
Data Source
AI summary
A system for generating foveated imagery is configurable to: (i) access an image; (ii) identify an ROI of the image; (iii) generate a downsampled image by performing downsampling on the image; (iv) generate a transformed downsampled image by performing one or more image processing operations on the downsampled image; (v) generate a transformed image by performing upsampling on the transformed downsampled image; and (vi) generate a foveated image by using the ROI of the image to define a corresponding ROI of the transformed image.