Monocular Human Depth Estimation With 3D Convolutional Joint Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monocular RGB-based methods for estimating 3D joint positions of humans suffer from limited accuracy and robustness due to missing visual cues and the inherent 2D nature of CNN features, as well as the lack of prior knowledge of 2D joint positions.
Innovation Solution
A convolutional neural network architecture incorporating 3D convolutional layers that reduce 2D resolution and expand depth resolution to estimate 3D joint positions by combining 2D joint locations with depth heatmaps, utilizing 3D features like object boundaries and spatial orderings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If CNN-based methods are used to estimate 3D joint positions from 2D images, then access to visual cues such as shadows, occlusions, lighting and volume is improved, but the extracted image features remain inherently 2D which limits the accuracy of 3D position estimation
Solution Approach 1:
The patent applies 3D convolutional layers to the CNN architecture, transforming the feature extraction from 2D to 3D. This allows the model to process volumetric information and spatial relationships directly from the image data, enabling the extraction of 3D-like features while maintaining access to 2D visual cues such as shadows, occlusions, lighting and volume. The 3D convolutions operate on volumetric data cubes, producing features that capture depth and spatial structure without requiring additional depth sensing hardware.
2Productivity
If lifting methods are used to generate 3D positions from 2D skeleton data, then computational efficiency is improved, but the method loses critical visual cues such as shadows, occlusions, lighting and volume that are essential for accurate depth determination
Solution Approach 1:
The patent merges the advantages of both lifting methods and CNN-based methods by combining 2D skeleton data with 3D convolutional feature extraction. The 3D CNN processes the original 2D image data to extract volumetric features and spatial relationships, while simultaneously using the 2D joint position predictions as constraints. This fusion allows the model to maintain computational efficiency while recovering lost visual cues through the 3D convolutional processing of the original images.
3Measurement precision
If 3D convolutional layers are added to the CNN architecture, then the ability to extract 3D features and improve estimation accuracy is improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent segments the computational process into distinct stages: first, 2D convolutional layers extract basic image features; second, 3D convolutional layers process these features to extract volumetric characteristics; third, the model integrates these features with 2D skeleton constraints to produce final 3D joint positions. This segmentation allows the model to manage computational complexity by processing information in manageable stages rather than attempting to perform all operations simultaneously.
Data Source
AI summary
Systems and methods are provided for estimating the 3D joint location of skeleton joints from an image segment of an object and a 2D joint heatmaps comprising 2D locations of skeleton joints on the image segment. This includes applying the image segment and 2D joint heatmaps to a convolutional neural network containing at least one 3D convolutional layer block, wherein the 2D resolution is reduced at each 3D convolutional layer and the depth resolution is expanded to produce an estimated depth for each joint. Combining the 2D location of each kind of joint with the estimated depth of the kind of joint generates an estimated 3D joint position of the skeleton joint.