First-Person 3D Pose Prediction With Joint-Limb Heatmaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for predicting a user's 3D pose from a first-person viewpoint suffer from information loss due to the compression of probabilistic heatmaps using CNN-based encoders, leading to inaccurate predictions, especially for body parts hidden by others, resulting in lower accuracy compared to third-person camera-based methods.
Innovation Solution
An apparatus and method that utilize a heatmap estimator to generate limb and joint heatmaps, a grid heatmap encoder to extract joint feature vectors, and a propagation network with a skeletal tree hierarchical structure to propagate relational feature vectors between neighboring joints, minimizing information loss and enhancing 3D pose prediction accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CNN-based encoder compresses heatmaps at once, then encoding efficiency is improved, but information loss occurs and prediction accuracy deteriorates
Solution Approach 1:
The patent segments the encoding process into two distinct stages: first encoding joint heatmaps to extract joint features, then encoding limb heatmaps to extract limb features. This segmentation prevents information loss by processing different types of heatmap information separately rather than compressing all heatmaps at once, thereby maintaining prediction accuracy while achieving encoding efficiency.
2Speed
If decoder performs prediction at once from compressed information, then processing speed is improved, but prediction accuracy deteriorates due to hidden body parts
Solution Approach 1:
The decoder performs pose prediction in two sequential stages: first predicting joint positions from joint features, then predicting limb positions from limb features while considering the skeletal hierarchy. This segmented approach allows the system to account for hidden body parts by propagating information through the skeletal tree, improving prediction accuracy without significantly compromising processing speed.
Solution Approach 2:
The patent performs preliminary encoding of joint heatmaps before decoding, extracting joint features that serve as foundational information for subsequent limb prediction. This preliminary action ensures that information about visible body parts is captured and preserved, which then informs the prediction of hidden body parts, thereby improving overall prediction accuracy.
3Device complexity
If conventional encoder compresses all heatmaps uniformly, then device complexity is reduced, but prediction accuracy for hidden body parts deteriorates
Solution Approach 1:
The encoder is segmented into distinct components: a joint heatmap encoder for processing joint heatmaps and a limb heatmap encoder for processing limb heatmaps. This segmentation allows each encoder to be optimized for its specific input type, improving prediction accuracy for hidden body parts while maintaining manageable device complexity through modular design.
Solution Approach 2:
The patent applies different encoding strategies to different parts of the heatmap data: joint heatmaps are encoded to capture joint position information, while limb heatmaps are encoded to capture limb configuration information. This local quality approach ensures that each part of the system is optimized for its specific function, improving overall prediction accuracy without excessive complexity.
4Loss of time
If probabilistic heatmap information is compressed once, then encoding time is reduced, but information loss increases leading to lower prediction accuracy
Solution Approach 1:
The encoding process is segmented into parallel streams: joint heatmap encoding and limb heatmap encoding. This segmentation allows probabilistic information to be preserved in both streams simultaneously rather than being compressed into a single representation, thereby reducing information loss while maintaining reasonable encoding time through efficient parallel processing.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed herein are an apparatus and method for predicting a three-dimensional (3D) pose. The apparatus for predicting a 3D pose includes: an input/output interface configured to receive a plurality of pieces of image data obtained by observing a user's body parts from a first-person viewpoint and output the results of computation processing of the image data; memory configured to store a program for performing a method of predicting a 3D pose; and a controller configured to predict the user's 3D pose based on the image data received through the input/output interface by executing the program. The control unit generates the plurality of pieces of image data as limb heatmaps and joint heatmaps, extracts a joint feature vector, outputs a propagation feature vector by propagating a relational feature vector between neighboring joints, and predicts the user's 3D pose based on the propagation feature vector and the joint feature vector.