Foreground Subject Detection Using Depth Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Accurately identifying and separating a human or other subject from the background in a digital image is challenging due to the complexity of distinguishing foreground and background pixels.
Innovation Solution
The use of depth sensors in conjunction with primary image sensors to generate a refined depth image, which is then used to classify pixels and separate foreground objects from the background through a segmentation process involving skeletal modeling and graph-cut algorithms, refining the segmentation map with temporal feedback to enhance accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors and primary image sensors are used together to classify pixels, then foreground subject identification accuracy is improved, but device complexity increases
Solution Approach 1:
The patent combines depth sensors and primary image sensors into a unified sensor array that captures both depth and color information simultaneously. The processing pipeline merges depth data and color data through registration and classification processes, allowing the system to leverage multiple sensor types for improved foreground identification while managing complexity through integrated processing.
2Measurement precision
If segmentation processes are applied to separate foreground from background, then subject separation accuracy is improved, but computational expense increases
Solution Approach 1:
The patent divides the pixel classification process into distinct segments: first classifying pixels as foreground, background, or test pixels based on depth and color data; then applying graph-cut algorithms specifically to test pixels at boundaries; and finally refining the segmentation map through temporal feedback. This segmented approach focuses computational resources on ambiguous regions rather than processing all pixels uniformly.
Solution Approach 2:
The patent applies computationally intensive graph-cut algorithms only to test pixels located at foreground-background boundaries, rather than to all pixels in the image. This partial application of the algorithm significantly reduces computational expense while maintaining separation accuracy in critical regions where foreground and background meet.
3Measurement precision
If graph-cut algorithms are used for segmentation refinement, then segmentation accuracy is improved, but processing time increases
Solution Approach 1:
The patent applies graph-cut algorithms locally only to test pixels that are located at the boundaries between foreground and background regions, rather than applying the computationally intensive algorithm to all pixels. This localized application maintains high segmentation accuracy at critical boundary regions while significantly reducing overall processing time.
4Reliability
If temporal feedback is used to refine segmentation maps, then classification reliability is improved, but device complexity increases
Solution Approach 1:
The patent incorporates temporal feedback by using classification results from previous video frames to refine the segmentation map in current frames. The system maintains a segmentation map that is continuously updated over time, using temporal consistency to improve classification reliability and reduce flickering artifacts while managing complexity through iterative refinement.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Classifying (300) pixels in a digital image includes receiving (302) a primary image from one or more image sensors. The primary image includes a plurality of primary pixels. A depth image from one or more depth sensors is also received (308). The depth image includes a plurality of depth pixels, each depth pixel registered to one or more primary pixels. The depth image and the primary image are cooperatively used to identify (324) whether a primary pixel images a foreground subject or a background subject.