Scene Segmentation via ROI Processing for Portable AR Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional scene segmentation techniques for computer vision applications, such as augmented and extended reality, are computationally expensive and can degrade battery life and performance on portable devices.
Innovation Solution
A method and apparatus that utilize a combination of image data from an image sensor and sparse depth data from a depth sensor, processed through multiple neural networks to obtain object regions of interest, feature maps, and dense depth maps, which are then aligned and passed to a fully convolutional network for segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional machine learning-based scene segmentation techniques are used, then segmentation accuracy is improved, but computational expense increases leading to degraded battery life and frame rate on portable devices
Solution Approach 1:
The patent divides the scene segmentation task into multiple processing stages: initial object detection, ROI identification, dense depth map generation, and final segmentation. By segmenting the processing workflow, the system performs computationally intensive operations only on relevant regions rather than the entire image, reducing overall computational expense and energy consumption while maintaining segmentation accuracy
Solution Approach 2:
The patent applies different processing quality levels to different regions of the image. Full-resolution processing is applied only to identified regions of interest containing objects, while other regions receive reduced or no processing. This local quality approach maintains high segmentation accuracy for objects while significantly reducing computational load and energy usage on portable devices
2Measurement precision
If traditional machine learning-based scene segmentation techniques are used, then segmentation accuracy is improved, but frame rate and processing speed deteriorate due to computational expense
Solution Approach 1:
The patent segments the image processing workflow into discrete stages with intermediate results stored. By processing only regions of interest at full resolution and using lower-resolution processing for other areas, the system increases processing speed and frame rate while maintaining segmentation accuracy for objects of interest
Solution Approach 2:
The patent performs preliminary object detection and ROI identification before executing the full segmentation algorithm. This preliminary action identifies which regions require intensive processing, allowing the system to skip unnecessary computations in other regions and thereby improving frame rate and processing throughput
Data Source
AI summary
A method includes obtaining, from an image sensor, image data of a real-world scene; obtaining, from a depth sensor, sparse depth data of the real-world scene; and passing the image data to a first neural network to obtain one or more object regions of interest (ROIs) and one or more feature map ROIs. Each object ROI includes at least one detected object. The method also includes passing the image data and sparse depth data to a second neural network to obtain one or more dense depth map ROIs; aligning the one or more object ROIs, one or more feature map ROIs, and one or more dense depth map ROIs; and passing the aligned ROIs to a fully convolutional network to obtain a segmentation of the real-world scene. The segmentation contains one or more pixelwise predictions of one or more objects in the real-world scene.