Whole Body Visual Effects Using ML Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality systems that modify images to replace backgrounds without depth sensors struggle with accurately recognizing and segmenting whole bodies, leading to poor image quality and failure in multi-user scenarios, as they rely on specialized face recognition techniques that are ineffective beyond a certain distance or with multiple users.
Innovation Solution
The use of machine learning techniques to segment a user's body from a single image and improve the segmentation over time by comparing current frames with previous frames, allowing for the application of visual effects such as background replacement without the need for depth sensors, thereby enhancing the augmented reality experience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If face recognition techniques are used for background replacement, then single-user close-range segmentation is improved, but whole-body segmentation and multi-user recognition deteriorate
Solution Approach 1:
The patent applies a universal segmentation model that can handle multiple tasks (face segmentation, whole-body segmentation, multi-user segmentation) instead of specialized face recognition techniques. This allows the system to adapt to different scenarios including single-user and multi-user cases, close-range and distance scenarios, achieving both accuracy and versatility
Solution Approach 2:
The patent segments the user's body into multiple regions (face, upper body, lower body, arms, legs) using a trained segmentation model. This segmentation approach enables precise identification of different body parts and users in the image, allowing for accurate background replacement even in multi-user scenarios where face recognition alone would fail
2Measurement precision
If depth sensors are added to improve segmentation accuracy, then measurement precision is improved, but device complexity and cost increase
Solution Approach 1:
The patent replaces the mechanical/optical depth sensing system with a computational approach using machine learning segmentation models. The system uses standard RGB cameras combined with trained neural networks to achieve accurate depth-aware segmentation without physical depth sensors, thereby reducing device complexity while maintaining segmentation precision
Solution Approach 2:
The patent changes the approach from physical parameter measurement (depth sensing) to computational parameter extraction (segmentation from 2D images). By using trained segmentation models that infer depth and body boundaries from standard images, the system achieves accurate segmentation without adding complex hardware sensors
3Speed
If specialized face recognition algorithms are used, then processing speed for faces is improved, but whole-body processing efficiency deteriorates
Solution Approach 1:
The patent segments the image processing task into different body regions (face, upper body, lower body, arms, legs) and applies efficient segmentation algorithms to each region simultaneously. This region-based segmentation approach maintains processing speed while improving whole-body segmentation efficiency compared to processing the entire image as a single task
Data Source
AI summary
Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image by applying one or more machine learning techniques; receiving input that selects a visualization mode; and applying one or more visual effects corresponding to the visualization mode to the monocular image based on the segmentation.