Hierarchical Image Sensing for VR Power Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge is to implement a high-resolution image sensing and processing system for VR/AR/MR applications on mobile/wearable platforms, which face limitations in power consumption, computation, and memory resources due to the need for high spatial and temporal resolutions, leading to inefficiencies in capturing and processing full-pixel image frames.
Innovation Solution
A dynamic and hierarchical imaging system that configures pixel cells and processing operations based on image sensing configurations, switching between high-complexity and low-complexity modes to optimize power and resource usage, using a two-tier feedback system to capture full-pixel frames initially and then sparse frames at high rates for targeted object tracking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-resolution image frames are generated and processed at high frame rate, then spatial and temporal resolution are improved, but power consumption and computation resources increase significantly
Solution Approach 1:
The image sensor array is divided into multiple blocks, with only selected blocks actively capturing images at high frame rates while other blocks remain inactive or capture at lower rates. This segmentation allows high temporal resolution for specific regions of interest while reducing overall power consumption and computation resources.
Solution Approach 2:
Different frame rates and image qualities are applied to different spatial regions based on their importance. Regions containing objects of interest receive high-resolution, high-frame-rate capture, while less important regions use lower resources, optimizing the balance between measurement precision and energy consumption.
2Measurement precision
If high-resolution image frames are generated and processed at high frame rate, then object detection accuracy is improved, but computation and memory resources increase significantly
Solution Approach 1:
The processing system is segmented into multiple levels: a first level processes only selected blocks at high frame rates for objects of interest, while a second level processes other blocks at lower rates. This hierarchical segmentation reduces the total computation and memory resources required while maintaining high object detection accuracy for critical regions.
Solution Approach 2:
Instead of processing all image blocks at maximum resolution and frame rate, the system applies partial processing only to blocks containing objects of interest. This partial action approach maintains high detection accuracy where needed while significantly reducing overall computation and memory resource requirements.
3Loss of information
If full-pixel image frames are captured and processed, then complete scene information is obtained, but power waste increases due to processing unnecessary pixel data
Solution Approach 1:
The system extracts and processes only the necessary subset of pixel data from each image frame. By identifying blocks containing objects of interest and selectively processing only those blocks at high frame rates, the system obtains sufficient scene information while eliminating power waste associated with processing unnecessary pixel data from other regions.
4Speed
If high frame rate image capture is performed, then temporal resolution is improved, but power consumption increases
Solution Approach 1:
The image sensor array is segmented into multiple blocks with different operational modes. Selected blocks capture images at high frame rates to achieve high temporal resolution for tracking moving objects, while other blocks capture at lower frame rates or remain inactive, significantly reducing overall power consumption while maintaining high temporal resolution where needed.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces power consumption and resource requirements while maintaining high temporal resolution, improving the efficiency and performance of image sensing and processing operations by selectively focusing on regions of interest, thereby enhancing the capabilities of VR/AR/MR applications.
Implementation Method 1
Each pixel cell may include a photodiode to sense light by converting photons into charge (e.g., electrons or holes)
Data Source
AI summary
In one example, an apparatus is provided. The apparatus is part of a mobile device and comprises: an array of pixel cells each configured to perform an image sensing operation, one or more attributes of the image sensing operation being configurable based on an image sensing configuration; a sensor data processor configured to execute a hierarchical set of processing operations; and a controller configured to: receive, from the sensor data processor, a first-level processing output from a first-level processing operation of the hierarchical set of processing operations; and based on the first-level processing output satisfying a condition, control the sensor data processor to execute a second-level processing operation on at least one image frame captured by the array of pixel cells to generate a second-level processing output.