3D Upper-Garment Tracking Without Depth Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems require depth sensors to modify images, increasing device cost and complexity, and struggle to recognize and apply visual effects to a user's whole body, especially when multiple users are present or at varying distances from the camera.
Innovation Solution
A system that segments articles of clothing using machine learning techniques to create a whole-body model, allowing for the application of AR elements without depth sensors, and tracks the position of clothing separately from body parts for intuitive interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth sensors are used to modify images in AR systems, then image modification capability is improved, but device cost and complexity increase
Solution Approach 1:
The patent extracts the depth sensing function from dedicated hardware sensors and implements it through software-based monocular depth estimation using standard RGB cameras. This removes the need for complex depth sensors while maintaining the capability to modify images in AR applications.
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a computational approach using machine learning models that process standard camera images to infer depth information, thereby substituting physical sensors with software-based solutions.
2Measurement precision
If depth sensors are used for whole body recognition, then recognition accuracy is improved, but device cost increases
Solution Approach 1:
The patent creates a virtual depth map copy from standard RGB images using trained neural networks. This computational copy provides the necessary depth information for whole body recognition without requiring physical depth sensors, thereby reducing device cost while maintaining recognition accuracy.
Solution Approach 2:
The patent changes the approach from direct physical measurement using depth sensors to indirect computational inference using machine learning. The system transforms standard 2D images into 3D depth representations through learned parameter transformations, achieving accurate whole body recognition without expensive hardware.
3Ease of operation
If traditional AR systems track body parts together, then tracking simplicity is maintained, but clothing interaction accuracy deteriorates
Solution Approach 1:
The patent segments the tracking system into two independent components: body part tracking and clothing tracking. This allows each component to be optimized independently - body tracking maintains simplicity while clothing tracking achieves high accuracy by separately monitoring garment position, orientation, and deformation.
Solution Approach 2:
The patent adds a separate tracking dimension for clothing items independent of body parts. By treating clothing as a distinct tracked object with its own spatial coordinates and transformation parameters, the system achieves accurate clothing interaction while maintaining simple body tracking.
Data Source
AI summary
Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a person wearing a fashion item; generating a three-dimensional (3D) model of the person depicted in the video; generating a segmentation of the fashion item worn by the person depicted in the video; outlining a portion of the 3D model based on the segmentation of the fashion item; tracking movement of the portion of the 3D model in the video; and in response to tracking the movement of the portion of the 3D model in the video, modifying a display position of one or more augmented reality elements on the fashion item in the video.