External Mesh Deformation via Body Mesh for Sensor-Free Mobile AR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing augmented reality systems require depth sensors for image modification, increasing device cost and complexity, and struggle to recognize and apply visual effects to the whole body of a user, especially at varying distances and in multiple-user scenarios, while also requiring complex rig generation for AR graphics adjustment.
Innovation Solution
Generate a body mesh of a user in an image or video and deform an external mesh in correspondence with the body mesh to apply AR elements efficiently, reducing the need for depth sensors and rig generation, and allowing seamless AR graphics addition on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used for image modification in AR systems, then measurement precision of user body is improved, but device complexity and cost increase
Solution Approach 1:
The patent extracts the depth sensing functionality from the mobile device itself and replaces it with an external depth map source. The system obtains depth maps from external sources (such as pre-captured depth data or server-provided depth information) rather than requiring the mobile device to have built-in depth sensors, thereby achieving accurate body measurement without increasing device complexity.
Solution Approach 2:
The patent uses 2D images combined with external depth maps to create accurate 3D body representations. Instead of relying on the device's own sensing capabilities, the system copies depth information from external sources and combines it with standard camera images to generate precise body meshes for AR applications.
2Measurement precision
If depth sensors are added to mobile devices for AR, then AR measurement accuracy is improved, but device cost increases
Solution Approach 1:
The patent removes the requirement for depth sensors from the mobile device hardware. By obtaining depth maps from external sources such as pre-captured depth data or server-provided information, the system achieves accurate AR measurements without the need to manufacture devices with expensive depth sensing capabilities.
Solution Approach 2:
The patent introduces external depth maps as an intermediary between the mobile device camera and the AR rendering process. These depth maps serve as a mediator that provides the necessary depth information for accurate AR measurements without requiring the device itself to have depth sensing hardware.
3Manufacturing precision
If complex rig generation is used for AR graphics adjustment, then AR graphic accuracy is improved, but processing time and complexity increase
Solution Approach 1:
The patent copies the body mesh structure directly from the generated 3D body model and uses it as the deformation target for AR graphics. This eliminates the need for complex rig generation and manual animation setup, as the body mesh itself serves as the structural framework for attaching and deforming AR clothing graphics.
Solution Approach 2:
The patent extracts the essential structural information directly from the body mesh vertices and uses this extracted data to drive AR graphic deformation. By taking out only the necessary geometric information from the body mesh rather than using complex rig systems, the system achieves accurate AR graphic adjustment with minimal processing overhead.
4Adaptability or versatility
If body mesh deformation is used for AR elements, then AR element adaptability is improved, but computational requirements increase
Solution Approach 1:
The patent segments the body into a mesh structure of vertices and uses this segmented representation to attach and deform AR graphics. By dividing the body into discrete mesh points, the system can efficiently calculate deformations at each vertex based on body pose changes, enabling adaptable AR elements while keeping computational requirements manageable through localized calculations.
Data Source
AI summary
Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; obtaining an external mesh associated with an augmented reality element; automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh; deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and modifying the video to include a display of the augmented reality element based on the deformed external mesh.