Video Foreground Segmentation Using Depth and Color Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image and video segmentation technologies face challenges such as color camouflage, moving backgrounds, shadows, scenes without foreground objects, and inaccurate depth data, which hinder effective separation of foreground and background.
Innovation Solution
A computer-implemented method that receives video frames with depth and color data, generates an initial segmentation mask based on depth data, determines a trimap to classify pixels as known background, known foreground, or unknown, calculates weights for unknown pixels, and performs fine segmentation using color data, the trimap, and weight map to obtain a binary mask for each frame, ultimately creating a foreground video.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If depth data is used for segmentation, then segmentation speed is improved, but segmentation accuracy deteriorates due to inaccurate depth data
Solution Approach 1:
The patent combines depth-based segmentation with color-based refinement. The depth data provides quick initial segmentation, while color data from the same or different views is integrated to correct errors and handle cases where depth data is unreliable, such as when foreground and background have similar colors or when depth data is noisy.
Solution Approach 2:
The patent introduces an intermediary refinement process that mediates between the fast but inaccurate depth-based segmentation and the more accurate but slower color-based segmentation. This intermediary step uses color information to correct depth-based errors without requiring complete color-based segmentation, thus balancing speed and accuracy.
2Measurement precision
If color data is used for segmentation, then segmentation accuracy is improved, but segmentation speed deteriorates due to computational complexity
Solution Approach 1:
The patent segments the segmentation process into two stages: a fast initial segmentation using depth data, and a selective refinement stage using color data. This divides the computationally intensive color-based segmentation into only the necessary regions, rather than processing the entire image with color data.
Solution Approach 2:
The patent applies color-based refinement only partially, specifically to regions where depth-based segmentation is likely to fail (such as regions with similar depth values or uncertain classifications), rather than applying it to the entire image. This reduces computational load while maintaining accuracy where it matters most.
3Device complexity
If simple depth-based segmentation is used, then device complexity is reduced, but segmentation reliability deteriorates in challenging conditions
Solution Approach 1:
The patent creates a multi-functional segmentation system that can operate effectively in various conditions. The system automatically adapts by using depth data when it is reliable, color data when depth is unreliable, or both together in challenging conditions such as color camouflage, moving backgrounds, or poor lighting.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Implementations described herein relate to methods, systems, and computer-readable media to render a foreground video. In some implementations, a method includes receiving a plurality of video frames that include depth data and color data. The method further includes downsampling the frames of the video. The method further includes, for each frame, generating an initial segmentation mask that categorizes each pixel of the frame as foreground pixel or background pixel. The method further includes determining a trimap that classifies each pixel of the frame as known background, known foreground, or unknown. The method further includes, for each pixel that is classified as unknown, calculating and storing a weight in a weight map. The method further includes performing fine segmentation to obtain a binary mask for each frame. The method further includes upsampling the plurality of frames based on the binary mask for each frame to obtain a foreground video.