Machine Learning Volumetric Capture for 3D Body Modeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems face challenges in accurately and efficiently modeling real-world objects, particularly human bodies, for augmented and virtual reality applications, as they struggle with capturing and processing the complex movements and non-rigid deformations of subjects in real-time.
Innovation Solution
The implementation of a machine learning-based volumetric capture system that uses a storage facility and processing facility to create and update 3D reference models of bodies by accessing machine learning models, capturing 2D images, and generating 3D joint locations, enabling real-time modeling and non-rigid deformation handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional capture systems are used to model real-world objects, then the system structure is relatively simple, but the modeling accuracy and ability to handle non-rigid deformations deteriorate
Solution Approach 1:
The patent segments the body into multiple body parts (e.g., head, torso, limbs) and models each part separately using individual capture devices. This segmentation allows for more accurate tracking of non-rigid deformations in each body part while managing system complexity through modular processing of each segment's data independently.
Solution Approach 2:
The patent introduces machine learning models as intermediaries between the raw capture device data and the final 3D model. These ML models process and interpret the captured images to accurately detect body part boundaries, joint locations, and non-rigid deformations, thereby improving modeling accuracy without directly increasing the complexity of the capture hardware itself.
2Reliability
If real-time volumetric capture is performed without machine learning, then the processing speed is faster, but the ability to accurately model complex movements and non-rigid deformations deteriorates
Solution Approach 1:
The patent performs preliminary actions by pre-training machine learning models on large datasets of human body movements and deformations before real-time capture. During real-time operation, these pre-trained models quickly process the capture data, enabling both high reliability in modeling complex movements and maintained processing efficiency through the use of already-trained models.
Solution Approach 2:
The patent replaces traditional mechanical/image-processing methods with machine learning-based approaches. Instead of using complex computer vision algorithms and manual processing pipelines, the system uses trained neural networks to automatically detect and model body movements, achieving both high reliability and processing efficiency.
3Adaptability or versatility
If traditional 3D modeling methods are used, then the computational resources required are fewer, but the ability to handle non-rigid deformations and complex movements deteriorates
Solution Approach 1:
The patent changes the parameters of the modeling approach by using machine learning models that can adaptively adjust to different types of movements and deformations. The ML models learn to handle various degrees of non-rigid deformation through training, providing high adaptability while optimizing computational resource usage through efficient model architectures and processing strategies.
Data Source
AI summary
An illustrative volumetric capture system accesses a machine learning model associated with bodies of a particular body type, as well as a two-dimensional (2D) image captured by a capture device located at a real-world scene. The 2D image depicts a body of the particular body type that is present at the real-world scene. Using the machine learning model and based on the 2D image, the volumetric capture system identifies a 2D joint location, from a perspective of the capture device, of a particular joint of the body. The volumetric capture system also generates a three-dimensional (3D) reference model of the body that represents the particular joint of the body at a 3D joint location that is determined based on the 2D joint location identified using the machine learning model. Corresponding methods and systems are also disclosed.