Machine-Learning Clothing Segmentation for Depth-Free Facial-Expression AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems require depth sensors to modify images, increasing device cost and complexity, and struggle to accurately segment and apply visual effects to user clothing, especially when users are at a distance or multiple users are present.
Innovation Solution
A machine learning technique is used to segment articles of clothing from the background and apply augmented reality elements based on facial expressions without depth sensors, allowing for accurate tracking and interaction with the clothing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth sensors are used to modify images in augmented reality systems, then image modification capability is improved, but device cost and complexity increase
Solution Approach 1:
The patent extracts the depth sensing function from the hardware level and implements it through software-based machine learning techniques. The system removes the requirement for depth sensors by using 2D image data combined with ML algorithms to infer depth information and perform segmentation, thereby reducing device complexity while maintaining image modification capability
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a computational approach using machine learning. Instead of using physical depth sensors to capture 3D information, the system uses 2D images processed through ML models to achieve similar segmentation and modification results, substituting hardware with software intelligence
2Speed
If traditional image processing methods are used to segment clothing, then processing speed is maintained, but segmentation accuracy deteriorates when users are at a distance or multiple users are present
Solution Approach 1:
The patent changes the parameters used for segmentation by incorporating multiple features beyond simple color and texture. The ML model considers spatial relationships, depth information (when available), contextual cues, and multi-scale features to improve segmentation accuracy while maintaining processing speed through optimized model architecture
Solution Approach 2:
The patent combines multiple data sources and feature types into a composite segmentation approach. By integrating information from different modalities (2D images, depth maps when available, contextual data) and processing them through a unified ML framework, the system achieves robust segmentation accuracy across various scenarios including distance and multi-user situations
3Device complexity
If machine learning techniques are used to segment clothing without depth sensors, then device complexity is reduced, but computational resource usage increases
Solution Approach 1:
The patent implements a tiered approach where the full ML model is used when computational resources are abundant, but simplified or pre-processed versions are used when resources are constrained. The system can operate with partial processing (using only 2D data without depth) or excessive processing (with depth maps and full ML models) depending on available resources, optimizing the balance between complexity reduction and resource usage
Solution Approach 2:
The patent performs preliminary processing of images to extract key features before applying the full ML model. By pre-computing feature representations, reducing image resolution where appropriate, and using efficient model architectures, the system reduces the computational burden of the ML inference while maintaining segmentation quality, thereby lowering energy consumption
4Ease of operation
If facial expressions are used to control AR elements on clothing, then user interaction intuitiveness is improved, but system complexity increases
Solution Approach 1:
The patent implements a system where the user's natural facial expressions automatically control the AR elements without requiring explicit commands or complex interfaces. The ML model continuously monitors facial expressions and autonomously adjusts the AR clothing elements based on detected emotions or gestures, making the system self-responsive and highly intuitive while keeping the control logic embedded within the existing ML pipeline
Data Source
AI summary
Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item worn by the person depicted in the image; identifying a facial expression of the user depicted in the image; and in response to identifying the facial expression, applying one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.