Foveated Sensor Readout Using ROI Frames to Cut Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image sensing technologies in artificial reality devices suffer from high latency and power consumption due to the inefficient transmission and processing of full-resolution image data, which reduces the enjoyment and utility of the experience.
Innovation Solution
Implementing foveated sensing techniques that vary image resolution across regions of interest (ROIs) within the image, using ROI configurations to read, process, and transmit image data efficiently, reducing data transfer bandwidth, latency, and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full-resolution image data is transmitted from the image sensor to the computing system, then image quality is maintained, but data transfer bandwidth and power consumption increase
Solution Approach 1:
The image sensor is divided into multiple regions of interest (ROIs) with different resolution requirements. The pixel array is segmented such that a first portion corresponds to a first ROI with higher resolution and a second portion corresponds to a second ROI with lower resolution, allowing selective transmission of data based on importance
Solution Approach 2:
Different regions of the image are assigned different quality levels and resolution parameters. The first ROI receives higher resolution imaging signals with more detailed data, while the second ROI receives lower resolution signals with less detailed data, optimizing the balance between image quality and power consumption by allocating resources locally based on need
2Loss of information
If full-resolution image data is transmitted, then complete visual information is provided, but latency increases reducing artificial reality enjoyment
Solution Approach 1:
The patent extracts and transmits only the most critical visual information from the image sensor. By identifying and prioritizing regions of interest, the system extracts essential visual data while discarding or compressing less critical information, thereby reducing transmission time and latency while maintaining adequate visual information for the artificial reality experience
3Measurement precision
If high-resolution imaging signals are captured across the entire pixel array, then detailed image data is obtained, but the quantity of data to be processed and transmitted increases
Solution Approach 1:
The pixel array is segmented into multiple regions with different sampling rates. The first region captures high-resolution signals with fine detail, while the second region captures lower-resolution signals with coarser detail. This segmentation reduces the total quantity of data captured and transmitted while preserving critical image details in the priority regions
Solution Approach 2:
The system applies partial high-resolution capture only to specific regions of interest rather than the entire image. By using excessive detail only where necessary (first ROI) and adequate but lower detail elsewhere (second ROI), the system optimizes the ratio of data quality to data quantity
Data Source
AI summary
Systems and readout methods for foveated sensing may include a pixel array, readout circuitry, and processing logic. The pixel array may have a number of pixels. The readout circuitry may be configured to read image data from the pixel array for each of the plurality of pixels. The processing logic may be configured to identify a number of regions of interest (ROIs) within the pixel array. The processing logic may be configured to associate the image data for the pixels with one or more ROIs. The processing logic may be configured to arrange the image data into data frames. The data frames may include the image data ordered by ROI. Image data for inactive pixels may be removed from the image data and data frames prior to transmission. The processing logic may be configured to transmit the data frames in an order that is based on the ROIs.