External Mesh Deformation Using Body Mesh Tracking for Mobile AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR systems require depth sensors to modify images, increasing device cost and complexity, and struggle to recognize and apply visual effects to a user's whole body, especially at distances beyond the camera's threshold, leading to inefficient processing and inaccurate background replacement.
Innovation Solution
Generate a body mesh of a user in an image and deform an external mesh based on body mesh changes to apply AR elements efficiently without generating a rig, using deformation models based on attributes like garment location and external forces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used to modify images in AR systems, then image modification accuracy is improved, but device cost and complexity increase
Solution Approach 1:
The patent extracts the depth sensing function from dedicated hardware sensors and implements it through software-based monocular depth estimation using standard cameras. This removes the need for complex depth sensors while maintaining the capability to obtain depth information for AR image modification.
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a computational approach using machine learning models that process standard camera images to infer depth information, thereby substituting physical sensors with software-based processing.
2Measurement precision
If depth sensors are used to modify images in AR systems, then image modification accuracy is improved, but processing power requirements increase
Solution Approach 1:
The patent performs preliminary depth estimation and body part segmentation before applying visual effects. By pre-processing the image to identify depth information and body regions of interest, the system reduces the computational burden during the actual effect application phase.
Solution Approach 2:
The patent applies visual effects selectively to specific body parts rather than processing the entire image uniformly. This localized approach reduces overall processing power requirements by concentrating computational resources only where needed.
3Adaptability or versatility
If visual effects are applied to the whole body, then AR experience completeness is improved, but processing complexity increases
Solution Approach 1:
The patent applies different visual effects to different body parts based on their characteristics. For example, clothes may have one effect while skin has another, allowing comprehensive body coverage with tailored processing that manages complexity through differentiation rather than uniform treatment.
Solution Approach 2:
The patent segments the body into distinct parts (head, torso, limbs, etc.) and processes each segment independently with appropriate visual effects. This segmentation reduces overall processing complexity by breaking down the complex task of whole-body processing into manageable independent segments.
Data Source
AI summary
Methods and systems are disclosed for performing operations for deforming an external mesh. The operations comprise receiving a video that includes a depiction of a real-world object. The operations comprise generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video. The operations comprise obtaining an external mesh associated with an Augmented-Reality (AR) element. The operations comprise accessing a plurality of deformation attributes associated with the external mesh, each attribute corresponding to a different deformation model. The operations comprise separately deforming, based on respective deformation models, a first portion of the external mesh and a second portion of the external mesh. The operations comprise modifying the video to include a display of the AR element based on the separately deformed first and second portions of the external mesh.