Body Pose Estimation Using Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual reality (VR) and augmented reality (AR) systems require depth sensors to accurately render avatars, which increases the cost and complexity of user devices, limiting their applications.
Innovation Solution
The use of machine learning techniques to generate virtual objects directly from a single RGB image or video of a user's body, eliminating the need for depth maps and allowing devices with simple RGB cameras to render animated virtual objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth sensors are used to accurately render avatars in VR/AR systems, then measurement precision of body pose is improved, but device complexity and cost increase
Solution Approach 1:
The patent replaces the mechanical depth sensing system with a machine learning-based image processing system. Instead of using depth sensors to capture spatial information, the system uses trained machine learning models to process RGB images and infer body pose, skeletal joint positions, and depth information simultaneously, thereby eliminating the need for separate depth sensing hardware.
Solution Approach 2:
The patent introduces machine learning models as an intermediary between the camera and the avatar rendering system. The trained models process the camera images to extract pose information and depth data, serving as a computational mediator that replaces the physical depth sensor and enables accurate avatar rendering using only standard RGB cameras.
2Manufacturing precision
If depth sensors are installed in user devices, then avatar rendering accuracy is improved, but cost increases
Solution Approach 1:
The patent substitutes expensive depth sensing hardware with a software-based machine learning solution that runs on standard RGB cameras. The trained models process images to extract the necessary spatial and pose information, achieving accurate avatar rendering without requiring additional costly sensors, thereby reducing device manufacturing costs.
Solution Approach 2:
The patent creates a computational copy of depth information by processing RGB images through machine learning models. Instead of directly capturing depth with sensors, the system generates depth maps and pose information as digital representations derived from camera images, achieving the same functional outcome at lower cost.
3Device complexity
If machine learning techniques are used to generate virtual objects from single RGB images, then device complexity is reduced, but measurement precision may be compromised
Solution Approach 1:
The patent employs trained machine learning models that have been extensively trained on large datasets of annotated images and corresponding pose information. The models process RGB images and output pose estimates, depth maps, and skeletal joint positions through a feedback loop of image analysis and pose synthesis, achieving accurate measurements without additional hardware.
Solution Approach 2:
The patent changes the approach from physical measurement parameters (depth sensors measuring distance) to computational parameters (machine learning models processing image pixels). The trained models transform 2D image data into 3D pose representations by learning complex mappings from training data, achieving precise measurements through parameter transformation rather than physical sensing.
Data Source
AI summary
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for detecting a pose of a user. The program and method include receiving a monocular image that includes a depiction of a body of a user; detecting a plurality of skeletal joints of the body depicted in the monocular image; and determining a pose represented by the body depicted in the monocular image based on the detected plurality of skeletal joints of the body. A pose of an avatar is modified to match the pose represented by the body depicted in the monocular image by adjusting a set of skeletal joints of a rig of an avatar based on the detected plurality of skeletal joints of the body; and the avatar having the modified pose that matches the pose represented by the body depicted in the monocular image is generated for display.