Animatable Garment Extraction from Monocular 3D Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing VR and AR systems require depth sensors for accurate 3D mesh generation, increasing device cost and complexity, and current methods fail to provide pixel-accurate representations of users, especially in facial regions, limiting the implementation of animatable XR fashion items.
Innovation Solution
A method using a first machine learning model to predict a volumetric reconstruction tensor and pose from a monocular image, followed by a second model to generate a 3D mesh in canonical space, allowing for pixel-accurate extraction and projection of XR fashion items.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used to accurately depict users in VR and AR systems, then measurement precision is improved, but device complexity and cost increase
Solution Approach 1:
The patent extracts the depth sensing functionality from physical hardware sensors and replaces it with a computational approach using machine learning models that process monocular images to infer 3D geometry, thereby eliminating the need for depth sensors while maintaining measurement precision
Solution Approach 2:
The patent substitutes the mechanical/optical depth sensing system with a computational vision system that uses neural networks to predict 3D mesh from 2D images, replacing physical sensing mechanisms with algorithmic processing
2Measurement precision
If depth sensors are added to mobile devices for XR applications, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent removes the requirement for depth sensors from mobile device hardware configurations by implementing a software-based solution that achieves 3D reconstruction using only standard monocular camera inputs already present in mobile devices
Solution Approach 2:
The patent enables mobile devices to perform 3D mesh reconstruction using their existing monocular camera for multiple purposes (photography, 3D reconstruction, AR/VR applications), eliminating the need for dedicated depth sensing hardware
3Ease of manufacture
If existing 3D mesh reconstruction technologies are used, then processing is simplified, but manufacturing precision is worsened
Solution Approach 1:
The patent changes the approach parameters by using end-to-end differentiable rendering and gradient-based optimization to adjust mesh vertices, achieving pixel-accurate representations while maintaining computational tractability through modern ML training techniques
4Productivity
If volumetric reconstruction and pose prediction are performed simultaneously using machine learning, then productivity is improved, but use of energy increases
Solution Approach 1:
The patent combines volumetric reconstruction and pose prediction into a single unified machine learning model that processes monocular images to generate both 3D mesh and pose information simultaneously, improving processing efficiency and reducing computational overhead compared to separate models
Data Source
AI summary
Methods and systems are disclosed for generating an animatable garment from a single image. The system accesses a monocular image depicting a person wearing a fashion item and generates a three-dimensional (3D) mesh representing the person in a canonical space. The system determines a pose of the person depicted in the monocular image and modifies a pose of the 3D mesh to match the determined pose of the person. The system extracts a portion of the 3D mesh corresponding to the fashion item and generates an extended reality (XR) item using the extracted portion of the 3D mesh.