Whole-Body Segmentation From Monocular Video Without Depth Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems require depth sensors to modify images, increasing device cost and complexity, and struggle to accurately segment and replace backgrounds for whole-body images, especially when users are at a distance or multiple users are present.
Innovation Solution
A machine learning-based approach segments a user's whole body from a single image using a first technique and refines the segmentation with a second technique based on previous frames, allowing background replacement and visual effects without depth sensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth sensors are used to modify images in augmented reality systems, then image modification capability is improved, but device cost and complexity increase
Solution Approach 1:
The patent extracts the depth sensing function from dedicated hardware sensors and implements it through software-based monocular depth estimation using machine learning models. This removes the need for physical depth sensors while maintaining the capability to obtain depth information for image modification in augmented reality applications.
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a computational approach using neural networks that process monocular images to estimate depth maps. This substitution eliminates complex hardware while achieving the same functional goal through algorithmic processing.
2Measurement precision
If depth sensors are used to segment whole body images, then segmentation accuracy is improved, but device cost increases
Solution Approach 1:
The patent creates a virtual depth map copy from monocular images using trained neural networks. This computational copy replicates the information that would be obtained from actual depth sensors, enabling accurate whole-body segmentation without requiring expensive hardware sensors.
3Measurement precision
If machine learning techniques are used for whole body segmentation, then background replacement accuracy is improved, but computational resource usage increases
Solution Approach 1:
The patent performs preliminary training of specialized neural network models offline using large datasets. The pre-trained models are then deployed for inference during runtime, where they require significantly fewer computational resources compared to training models in real-time. This separates the computationally intensive training phase from the resource-constrained deployment phase.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; smoothing, using the video feed, the segmentation of the whole body generated based on the monocular image to provide a smoothed segmentation; and applying one or more visual effects to the monocular image based on the smoothed segmentation.