Multimodal Foreground Background Segmentation Framework
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video processing segmentation techniques, such as chroma keying and background subtraction, are not robust enough for complex scenarios like multiple camera studios, and depth-based segmentation is limited by noise, making them inadequate for accurately separating foreground objects from backgrounds in varying conditions.
Innovation Solution
A multimodal segmentation framework that combines RGB, infrared, and depth data to determine probability values for each pixel, using a global binary segmentation mechanism to achieve robust and accurate foreground/background separation by integrating contribution factors from different modalities like chroma keying, background subtraction, and depth-based methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If chroma key segmentation is used, then segmentation can be achieved under controlled conditions, but performance deteriorates when illumination or background color changes
Solution Approach 1:
The patent combines multiple segmentation modalities (chroma key, background subtraction, depth-based methods) into a unified framework that processes contribution factors from each modality. This merging allows the system to maintain reliable segmentation across varying conditions by integrating complementary strengths of different approaches rather than relying on a single method.
Solution Approach 2:
The segmentation framework is designed to be universal by accepting contribution factors from multiple modalities and adapting to different scenarios. The system can handle chroma key scenarios, background subtraction scenarios, depth-based scenarios, or any combination thereof, making it versatile across diverse lighting and background conditions without requiring separate specialized systems.
2Reliability
If background subtraction is used, then foreground objects can be separated from background, but the method fails when foreground and background are similar or image is blurry
Solution Approach 1:
The patent merges background subtraction with other modalities such as chroma key and depth-based methods. When background subtraction alone fails due to similarity between foreground and background, the combined framework compensates by incorporating additional contribution factors from other modalities, thereby maintaining segmentation accuracy in challenging scenarios.
3Reliability
If depth-based segmentation is used, then foreground objects can be separated using depth data, but noise in depth data reduces segmentation quality
Solution Approach 1:
The patent combines depth-based segmentation with other modalities that do not suffer from the same noise issues. By merging depth contribution factors with chroma key and background subtraction contributions, the system compensates for depth data noise through the complementary information from other modalities, maintaining robust segmentation despite measurement imperfections.
4Reliability
If chroma keying is used, then segmentation works when a screen can be placed in background, but the method is limited when screen placement is not practical
Solution Approach 1:
The segmentation framework is designed to be universally applicable by incorporating multiple modalities. When chroma key conditions are met (green screen available), the system utilizes chroma key contribution factors. When screen placement is not practical, the system seamlessly transitions to using background subtraction and/or depth-based modalities, maintaining segmentation effectiveness across diverse real-world scenarios without requiring physical screen modifications.
Data Source
AI summary
The subject disclosure is directed towards a framework that is configured to allow different background-foreground segmentation modalities to contribute towards segmentation. In one aspect, pixels are processed based upon RGB background separation, chroma keying, IR background separation, current depth versus background depth and current depth versus threshold background depth modalities. Each modality may contribute as a factor that the framework combines to determine a probability as to whether a pixel is foreground or background. The probabilities are fed into a global segmentation framework to obtain a segmented image.