3D Pose Estimation Training Using Image and Motion Capture Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network models struggle to accurately determine 3D pose and shape of humans in images due to the lack of large datasets of annotated 3D poses and shapes, which are costly and difficult to collect.
Innovation Solution
A training system that utilizes a first dataset of animal images with joint coordinates and a second dataset of motion capture data without images, fine-tuning batch normalization layers and incorporating synthetic data generation to enhance model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large datasets of annotated 3D poses and shapes are collected to train neural network models, then model accuracy in determining 3D pose and shape is improved, but data collection cost and complexity increase significantly
Solution Approach 1:
The patent creates synthetic 3D pose and shape annotations by rendering virtual human models with known ground truth parameters, then generating corresponding 2D images and detecting keypoints in those rendered images to create training annotations without requiring physical motion capture sessions. This copying approach generates unlimited training data with perfect ground truth labels.
Solution Approach 2:
The patent introduces an intermediary rendering pipeline that translates between 3D model parameters and 2D image observations. This intermediary system includes virtual camera models, lighting models, and keypoint detection algorithms that simulate real-world imaging conditions, allowing the neural network to learn from synthetic data that closely mimics real data distributions.
2Measurement precision
If motion capture data is used for training, then model training accuracy is improved, but the requirement for specialized equipment and controlled environments increases
Solution Approach 1:
The patent replaces physical motion capture sessions with virtual rendering of human models. Instead of requiring motion capture studios, cameras, and specialized equipment, the system generates synthetic training data by rendering 3D human models with known pose and shape parameters, eliminating the need for expensive specialized equipment while maintaining training quality.
Solution Approach 2:
The system generates its own training data automatically through the rendering pipeline. The synthetic data generation process is self-contained, requiring no external motion capture sessions, human subjects, or specialized recording equipment. The system creates unlimited training examples by varying parameters of virtual human models and rendering them under different conditions.
3Ease of operation
If 2D keypoint annotations are manually annotated without 3D pose data, then annotation process is simplified, but the ability to learn accurate 3D pose representation is limited
Solution Approach 1:
The patent generates synthetic 2D images from 3D models with known ground truth pose and shape parameters, then detects keypoints in these rendered images to create training annotations. This process creates a complete training dataset with both 2D keypoint locations and corresponding 3D pose labels, enabling the network to learn accurate 3D pose representation while maintaining automated annotation processes.
Solution Approach 2:
The patent bridges the gap between 2D images and 3D pose by rendering 3D human models with known parameters into 2D images, then detecting keypoints in the rendered 2D images. This dimensional transformation creates training data that connects 2D observations with 3D ground truth, allowing the network to learn 3D pose estimation from 2D image inputs without requiring manual 3D annotation.
Data Source
AI summary
A training system includes: a neural network model configured to determine three-dimensional coordinates of joints, respectively, representing poses of animals in images, where the neural network model is trained using a first training dataset including: images including animals; and coordinates of joints of the animals in the images, respectively; and a training module configured to, after the training of the neural network model using the first training dataset, train the neural network model using a second training dataset including motion capture data, where the motion capture data does not include images of animals and includes measured coordinates at points, respectively, on animals.