Monocular Video Body Tracking Using Neural Network Pose Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monocular video-based body tracking systems face challenges in dimensional extrapolation, real-time performance, and partial visibility, leading to inaccuracies and user-perceptible lag, especially on devices with limited computational power.
Innovation Solution
A computer-implemented method using a pre-trained neural network to extract 2D images from video frames, estimate 3D pose of upper body joints, determine confidence scores, and animate a 3D avatar, while applying temporal smoothing and calibrating for camera distortions to enhance tracking accuracy and reduce lag.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If monocular video-based body tracking is implemented on devices with limited computational power, then accessibility and cost are improved, but tracking accuracy and real-time performance deteriorate
Solution Approach 1:
The system segments the body tracking problem into two distinct stages: 2D pose estimation from video frames, and 3D pose extrapolation. By separating these tasks and applying specialized algorithms to each stage, the system achieves accurate 3D tracking results while maintaining computational efficiency suitable for mobile devices with limited processing power.
Solution Approach 2:
The system transitions from 2D video input to 3D pose output by implementing dimensional extrapolation. This involves estimating depth information and three-dimensional joint positions from two-dimensional image data, effectively adding the depth dimension to reconstruct accurate 3D body poses without requiring expensive depth sensors.
2Measurement precision
If complex algorithms for 3D pose estimation are used, then tracking accuracy is improved, but processing speed and real-time performance worsen
Solution Approach 1:
The system performs preliminary 2D pose estimation on video frames before conducting 3D extrapolation. By pre-processing the video data to extract 2D joint positions and using these as input for the 3D estimation stage, the system reduces the computational complexity of the final 3D pose calculation, enabling real-time performance while maintaining accuracy.
3Measurement precision
If traditional body tracking approaches with external hardware are used, then tracking accuracy is improved, but device complexity and cost increase
Solution Approach 1:
The system replaces mechanical/physical tracking hardware (such as motion capture suits, depth sensors, and external cameras) with computational methods running on standard mobile device cameras. By substituting physical tracking apparatus with software-based monocular video analysis and 3D extrapolation algorithms, the system achieves comparable tracking accuracy while eliminating the need for complex external hardware.
4Speed
If monocular video tracking is implemented without temporal smoothing, then responsiveness is improved, but tracking stability and accuracy worsen
Solution Approach 1:
The system implements temporal smoothing that uses feedback from previous frame predictions to refine current pose estimates. By incorporating historical pose data and using it to guide current 3D extrapolation, the system stabilizes tracking results and reduces noise while maintaining responsiveness to actual body movements.
Data Source
AI summary
Various implementations relate to methods, systems and computer readable media to provide body tracking from monocular video. According to one aspect, a computer-implemented method includes obtaining a video including a set of video frames depicting movement of a human subject; extracting 2D images of the human subject from the video frames; providing the 2D images as input to a pre-trained neural network model. The method further includes determining a pose of the subject based on the 2D images. The method further includes generating a 3D pose estimation of upper body joint positions of the human subject. The method further includes determining confidence scores, and selecting a set of keypoints of the upper body joints of the human subject based on the confidence scores. The method further includes animating a 3D avatar using at least the selected set of keypoints, and displaying the animated 3D avatar in a user interface.