Monocular Person Pose Depth Estimation With 3D CNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monocular RGB-based methods for estimating human joint depths suffer from limited accuracy and robustness due to missing visual cues and inherent 2D nature of CNN features, especially in challenging poses and environments.
Innovation Solution
A convolutional neural network architecture that incorporates 3D convolutional layers to reduce 2D spatial resolution and expand depth resolution, combining 2D joint positions with estimated depths to produce accurate 3D joint positions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If CNN-based methods use 2D convolutions to extract features from input images, then the model can process standard RGB image data without additional depth sensing hardware, but the extracted image features are inherently 2D and lack the 3D nature needed for accurate depth estimation
Solution Approach 1:
The patent transitions from 2D CNN convolutions to 3D CNN convolutions, adding the depth dimension to feature extraction. This allows the model to process volumetric data and capture three-dimensional spatial relationships, thereby obtaining 3D features that maintain the 3D nature of the output while still using standard 2D image input data
2Ease of operation
If lifting methods use fully-connected neural network models to generate 3D positions from 2D skeleton data, then the approach can work with basic 2D joint detections, but it misses critical visual cues such as shadows, occlusions, lighting and volume that are essential for correct depth determination
Solution Approach 1:
The patent merges the advantages of both approaches by combining 2D skeleton joint position inputs with 3D CNN-based visual feature extraction. The 3D CNN processes the input image to capture volumetric and visual cues (shadows, occlusions, lighting, volume), while the 2D skeleton provides structural constraints. These two sources of information are fused to produce accurate 3D joint positions
3Productivity
If CNN-based approaches infer 2D joint positions and depths simultaneously, then the model can process raw image data directly, but it lacks a priori knowledge of joint positions which reduces estimation robustness
Solution Approach 1:
The patent performs preliminary detection of 2D joint positions using skeleton detection algorithms before feeding the data into the 3D CNN model. This preliminary action provides a priori knowledge of joint locations, which then guides the depth estimation process. The 3D CNN uses this pre-detected skeleton information as input, allowing it to focus on inferring depth while maintaining robustness through the structural constraints provided by the skeleton
Data Source
AI summary
Systems and methods are provided for estimating the 3D joint location of skeleton joints from an image segment of an object and a 2D joint heatmaps comprising 2D locations of skeleton joints on the image segment. This includes applying the image segment and 2D joint heatmaps to a convolutional neural network containing at least one 3D convolutional layer block, wherein the 2D resolution is reduced at each 3D convolutional layer and the depth resolution is expanded to produce an estimated depth for each joint. Combining the 2D location of each kind of joint with the estimated depth of the kind of joint generates an estimated 3D joint position of the skeleton joint.