Monocular 3D Human Motion Estimation With Global Temporal Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating temporally consistent 3D human shape and motion from monocular video fail to effectively capture long-term temporal dynamics and perform poorly under occlusions due to limitations in spatio-temporal feature aggregation and lack of exploitation of human body's geometric model and appearance.
Innovation Solution
A method that utilizes Continuous Surface Embeddings (CSE) for body-aware feature extraction, self-similarity and self-attention mechanisms for spatio-temporal feature aggregation, and Long Short-Term Memory (LSTM) refinement to predict and refine 3D human shape and motion, incorporating camera parameters for improved temporal consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If CNN or RNN modules are used for spatio-temporal feature aggregation from neighboring frames, then local temporal coherence is improved, but long-term temporal dynamics are lost
Solution Approach 1:
The patent extends the temporal aggregation beyond local neighboring frames by incorporating features from the entire video sequence, transitioning from a local temporal window to a global temporal dimension. This allows the model to capture both local coherence and long-term dynamics simultaneously.
Solution Approach 2:
The patent employs a unified spatio-temporal feature aggregation mechanism that serves multiple functions: capturing local temporal coherence through neighboring frames and capturing long-term temporal dynamics through the entire video sequence, making the model versatile for different temporal scales.
2Ease of manufacture
If generic deep features from RGB frames are used, then feature extraction is simplified, but human body geometric model prior knowledge is not exploited
Solution Approach 1:
The patent transforms generic RGB frame features into body-aware features by incorporating parameters from the human body geometric model (SMPL). This parameter transformation integrates prior knowledge about human body topology and geometry into the feature representation, improving measurement precision while maintaining reasonable complexity.
Solution Approach 2:
The patent introduces body-aware features as an intermediary between generic RGB features and the final 3D human pose estimation. These features serve as a bridge that incorporates human body geometric model prior knowledge into the deep learning pipeline, enhancing accuracy without requiring direct integration of complex geometric models.
3Productivity
If per-frame prediction is performed without joint-estimation, then processing speed is improved, but temporal consistency deteriorates
Solution Approach 1:
The patent performs joint-estimation of 3D human pose and shape across the entire video sequence rather than independent per-frame prediction. This continuous estimation approach ensures temporal consistency by considering all frames simultaneously, maintaining stable and coherent human motion representation throughout the video.
Solution Approach 2:
The patent performs preliminary spatio-temporal feature aggregation from the entire video sequence before final prediction. This preliminary action of aggregating features across all frames ensures that temporal information is captured early in the processing pipeline, enabling consistent predictions while maintaining reasonable processing efficiency.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
Estimating temporally consistent 3D human body shape, pose, and motion from a monocular video is a challenging task due to occlusions, poor lightning conditions, complex articulated body poses, depth ambiguity, and limited availability of annotated data. Embodiments of present disclosure provide a method for temporally consistent motion estimation from monocular video. A monocular video of person(s) is captured by a weak perspective camera and spatial features of body of the persons are extracted from each frame of the video. Then, initial estimates of body shape, body pose, and features of the weak perspective camera are obtained. The spatial features and initial estimates are then aggregated to obtain spatio-temporal features by a combination of self-similarity matrices between the spatial features, pose and the camera and selfattention maps of the camera features and the spatial features. The spatio-temporal aggregated features are then used to predict shape and pose parameters of the person(s).