Foveated Imaging ISP Statistics for Gaze-Driven Region Changes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image signal processor (ISP) architectures fail to efficiently support foveated imaging, leading to image quality degradation due to delays in statistics collection and computation when foveated and blend regions change per frame, especially in extended reality (XR) applications.
Innovation Solution
A multi-context ISP architecture with a shared lookup table (LUT) is employed to provide instant adjustments for foveated and blend regions based on eye gaze changes, reducing delays and improving image quality while minimizing power, area, and DDR bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If separate statistics collection paths are maintained for foveated and blend regions, then image processing accuracy is improved, but device complexity and power consumption increase
Solution Approach 1:
The patent merges the statistics collection paths for foveated and blend regions into a single shared path. The image sensor collects statistics for the entire image frame once, and the processor divides these statistics into different regions (foveated, blend, peripheral) based on region-of-interest masks. This eliminates the need for separate statistics collection paths while maintaining processing accuracy for different regions.
Solution Approach 2:
The shared statistics collection path serves multiple functions by providing statistics for the entire image frame that can then be selectively applied to different regions. The same statistics collection mechanism supports both foveated region processing and blend region processing, making the system more efficient while reducing complexity.
2Loss of energy
If statistics are collected for the entire image frame, then DDR bandwidth is reduced, but image processing delay increases
Solution Approach 1:
The system performs preliminary statistics collection for the entire image frame in advance, storing the statistics in memory. When processing is needed, the processor quickly retrieves the pre-collected statistics and applies region-specific masks to generate the required region statistics. This preliminary action eliminates the need for real-time statistics collection for each region, reducing processing delay while maintaining efficient bandwidth usage.
3Stability of the object's composition
If region-of-interest masks are updated per frame, then image quality consistency is improved, but computational workload increases
Solution Approach 1:
Instead of computing completely new statistics for each region every frame, the system uses pre-collected frame-level statistics and applies region-of-interest masks to extract the necessary region information. This partial action approach maintains image quality consistency across frames while significantly reducing the computational workload compared to full re-computation.
Data Source
AI summary
Systems and techniques are described for image processing. For example, a computing device can obtain, from an image sensor, a peripheral image frame of a scene, a blend image frame, and a foveated image frame. The peripheral image frame includes the blend image frame and the foveated image frame. The computing device can determine, based on the peripheral image frame, statistics for the peripheral image frame that include a respective pixel value, a respective luma value, and/or a respective chroma value associated with each pixel of a plurality of pixels of the peripheral image frame. The computing device can determine, based on a first starting location within the statistics, a first configuration for the blend image frame and can determine, based on a second starting location within the statistics, a second configuration for the foveated image frame.