Image Segmentation Using Depth and RGB Data for Transparent Foreground
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video conferencing technologies face challenges in accurately segmenting transparent or semi-transparent foreground elements and small portions of background encircled by foreground during video conferencing, leading to inaccuracies in background modification processes.
Innovation Solution
Combining RGB image data with depth information from a depth-capable camera, such as a stereo infrared sensor, and utilizing facial detection to identify a face, perform depth cutoff, and process the images using a convolutional neural network (CNN) for improved segmentation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If object segmentation is performed using only RGB image data, then the processing speed is faster and the system is simpler, but the segmentation accuracy is insufficient especially for transparent or semi-transparent foreground elements
Solution Approach 1:
The patent combines RGB image data with depth map data to perform object segmentation. The RGB data provides color and texture information while the depth map provides depth information, allowing the system to accurately segment transparent or semi-transparent foreground elements that cannot be reliably detected using RGB data alone. This merging of multiple data sources resolves the contradiction by improving segmentation accuracy without requiring a completely new system architecture.
Solution Approach 2:
The patent uses depth map data as an intermediary to enhance the segmentation process. The depth map serves as an additional information layer that mediates between the RGB image data and the segmentation result, providing depth context that helps distinguish transparent foreground elements from the background. This intermediary data source enables more accurate segmentation while maintaining system simplicity.
2Measurement precision
If depth information is added to improve segmentation accuracy, then the segmentation precision improves, but the processing time increases
Solution Approach 1:
The patent segments the depth map data into a foreground depth map and a background depth map based on the segmented foreground mask. This segmentation allows the system to process only the relevant portions of the depth information rather than the entire depth map, reducing processing time while maintaining segmentation precision for transparent and small background elements.
Solution Approach 2:
The patent applies different processing quality levels to different regions of the image. For transparent or semi-transparent foreground elements and small background portions, the system uses enhanced processing with depth information. For other regions, standard processing is applied. This local quality approach optimizes processing time by applying computational resources only where needed to achieve high segmentation precision.
3Reliability
If traditional object segmentation is used, then the system is simpler and faster, but it fails to accurately handle transparent or semi-transparent foreground elements and small background portions
Solution Approach 1:
The patent merges RGB image data with depth map data to create a more reliable segmentation system. The depth map provides additional information about the three-dimensional structure and depth of objects, enabling the system to reliably segment transparent or semi-transparent foreground elements and small background portions that traditional RGB-only methods cannot handle accurately.
Solution Approach 2:
The patent adds the depth dimension to the traditional two-dimensional RGB image data. By incorporating depth information, the system transitions from two-dimensional image processing to three-dimensional scene understanding, enabling accurate segmentation of transparent and small elements that are invisible or indistinguishable in the two-dimensional RGB image alone.
Data Source
AI summary
In some examples in accordance with the present description, an electronic device is provided. The electronic device includes a controller to implement an image segmentation process. The controller is to obtain color information of an image. The controller also is to obtain depth information of the image. The controller also is to determine a depth of a face represented in the color information. The controller also is to segment a foreground of the image from a background of the image according to the color information and the depth information based on the depth of the face.