3D Pose Estimation via Volumetric Heatmaps and Self-Supervision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 3D human pose estimation systems require either full 3D supervision or additional supervision like camera extrinsics, limiting their ability to accurately estimate poses from a single image without extensive annotation data.
Innovation Solution
A system utilizing a convolutional neural network with residual and deconvolution networks, generating volumetric heatmaps and applying a soft argmax function to estimate 3D poses from single images, leveraging epipolar geometry and 2D ground-truth poses to create self-supervision, eliminating the need for 3D supervision and camera extrinsics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full 3D supervision or additional supervision (camera extrinsics) is used, then measurement precision of 3D pose estimation is improved, but device complexity and data annotation requirements increase
Solution Approach 1:
The system uses 2D pose detection results to generate self-supervision signals through virtual multi-view synthesis. The 2D detector outputs are projected back into 3D space to create supervisory targets, allowing the network to learn 3D pose estimation without requiring external 3D annotations or camera calibration data. This self-service mechanism resolves the contradiction by eliminating complex supervision requirements while maintaining estimation accuracy.
Solution Approach 2:
The patent introduces an intermediary mechanism using 2D pose detection as a bridge between available 2D data and desired 3D pose estimation. The 2D detection results serve as an intermediate representation that is transformed through virtual multi-view projection to create supervisory signals, mediating the gap between limited 2D supervision and complex 3D pose estimation requirements.
2Measurement precision
If 3D ground-truth annotations are collected, then measurement precision is improved, but loss of time and annotation cost increase
Solution Approach 1:
The system performs preliminary 2D pose detection on available images before generating 3D pose estimates. By first obtaining 2D keypoints through detection and then using these as basis for virtual multi-view synthesis, the method prepares intermediate results that serve as self-supervision targets, eliminating the need for time-consuming 3D annotation collection while maintaining accurate 3D estimation.
Solution Approach 2:
The patent creates virtual copies of 2D pose detection results by projecting them into multiple view perspectives. These copied and transformed 2D detections serve as surrogate 3D supervision signals, replacing the need for actual 3D ground-truth annotations and significantly reducing annotation time and costs.
3Measurement precision
If additional supervision (unpaired 3D ground truth or subset labels) is used, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The system makes the 2D pose detector universal by using it for multiple purposes: both as the primary detection component and as a source for generating self-supervision signals. The same 2D detector outputs are used directly for 2D pose estimation and simultaneously projected to create 3D supervisory targets, eliminating the need for separate 3D detection components or additional supervision data streams.
Data Source
AI summary
A system for estimating a three dimensional pose of one or more persons in a scene is disclosed herein. The system includes at least one camera, the at least one camera configured to capture an image of the scene; and a data processor including at least one hardware component, the data processor configured to execute computer executable instructions. The computer executable instructions comprising instructions for: (i) receiving the image of the scene from the at least one camera; (ii) extracting features from the image of the scene for providing inputs to a convolutional neural network; (iii) generating one or more volumetric heatmaps using the convolutional neural network; and (iv) applying a maximization function to the one or more volumetric heatmaps to obtain a three dimensional pose of one or more persons in the scene.