Garment Recoloring via Segmentation and Lighting Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality systems in messaging applications face challenges in recognizing and modifying whole-body users without depth sensors, leading to poor image quality and failure in applying visual effects, especially when users are at a distance or multiple users are in the image.
Innovation Solution
The system employs machine learning techniques to segment articles of clothing and estimate light and shadows, allowing for the application of visual effects to specific body parts like shirts, while preserving original lighting conditions, without the need for depth sensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used to recognize and modify whole-body users, then measurement precision and reliability are improved, but device complexity and cost increase
Solution Approach 1:
The patent extracts the depth sensing function from the hardware level and implements it through software-based monocular depth estimation using machine learning models. This removes the physical depth sensor requirement while maintaining depth-related functionality for virtual try-on applications.
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a computational approach using neural networks that process standard RGB images to estimate depth information, thereby substituting physical sensors with algorithmic processing.
2Adaptability or versatility
If visual effects are applied to users at a distance or when multiple users are present, then adaptability is improved, but measurement precision and effect application accuracy deteriorate
Solution Approach 1:
The patent segments the image to identify and isolate individual users and their clothing items, enabling precise application of visual effects to each user regardless of distance or number of users in the scene.
Solution Approach 2:
The patent applies different processing quality and attention to different regions of the image, focusing computational resources on identifying and processing each user's clothing area with high precision while maintaining overall scene awareness.
3Manufacturing precision
If machine learning techniques are used to segment clothing and estimate lighting, then manufacturing precision and effect application accuracy are improved, but use of energy and computational resources increase
Solution Approach 1:
The patent performs preliminary actions by pre-training machine learning models offline and preparing segmentation masks and lighting estimates before the actual virtual try-on application, reducing real-time computational energy requirements.
Solution Approach 2:
The system uses the input image itself to generate the segmentation and lighting estimates through self-supervised learning approaches, eliminating the need for additional sensors or external data sources and reducing overall system resource requirements.
4Ease of operation
If augmented reality elements are applied without preserving original lighting, then ease of operation is improved, but realism and visual quality deteriorate
Solution Approach 1:
The patent estimates lighting parameters (direction, intensity, color temperature) from the original image and applies these parameters to the virtual clothing elements, preserving the original scene's lighting characteristics while maintaining ease of automated operation.
Data Source
AI summary
Methods and systems are disclosed for performing operations for recoloring a fashion item. The operations include receiving an image that includes a depiction of a person wearing a fashion item. The operations include generating a segmentation of the fashion item worn by the person depicted in the image. The operations include extracting a portion of the image corresponding to the segmentation of the fashion item. The operations include estimating lights and shadows being cast on the fashion item in the portion of the image. The operations include applying one or more augmented reality elements to the fashion item in the image based on the lights and shadows being cast on the fashion item.