Monocular 3D Human Shape Tracking Under Occlusion and Pose Ambiguity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating temporally consistent 3D human shape and motion from monocular video fail to capture long-term temporal dynamics and perform poorly under occlusions due to issues like occlusions, poor lighting, complex articulated body poses, and limited annotated data, while neglecting the fixed topology of the human body and lacking initialization from per-frame shape and pose predictions.
Innovation Solution
A method that extracts spatial features using a body-aware feature extractor, performs temporal aggregation through self-similarity and self-attention mechanisms, and refines predictions using LSTM to achieve spatio-temporal coherence, incorporating the fixed topology of the human body and utilizing per-frame initialization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If CNN or RNN modules are introduced to perform spatio-temporal feature aggregation from neighboring frames, then local temporal coherence is improved, but long-term temporal dynamics are not captured and performance under partial occlusion remains poor
Solution Approach 1:
The video sequence is divided into multiple temporal segments or clips, where each segment is processed independently to capture local temporal coherence. This segmentation allows the model to focus on short-term dynamics while maintaining overall temporal structure, resolving the contradiction between local coherence and long-term dynamics capture.
Solution Approach 2:
The patent introduces a temporal dimension by processing multiple video clips sequentially and aggregating their predictions. This transforms the problem from single-frame estimation to multi-temporal-scale estimation, enabling the model to capture both local temporal coherence within clips and long-term dynamics across clips through temporal aggregation.
2Device complexity
If generic deep features are extracted from RGB frames, then feature extraction is simplified, but the fixed topology and geometric model of the human body are not exploited
Solution Approach 1:
The patent applies body-aware feature extraction that treats different regions of the human body differently, leveraging the fixed topology and geometric constraints of the SMPL model. This allows the system to incorporate domain-specific knowledge about human body structure, improving measurement precision while maintaining reasonable complexity through targeted feature extraction.
3Productivity
If per-frame prediction is performed without joint-estimation, then processing speed is improved, but temporal consistency and accuracy are compromised
Solution Approach 1:
The patent performs preliminary per-frame predictions within each video clip to obtain initial shape and pose estimates. These preliminary predictions serve as inputs for subsequent temporal aggregation across clips, allowing the system to maintain processing efficiency while improving temporal consistency through coordinated refinement across multiple frames.
Solution Approach 2:
The system maintains continuous temporal processing by aggregating predictions across multiple video clips in sequence. This continuous action ensures that temporal consistency is preserved while allowing parallel processing within each clip, balancing productivity and reliability through structured temporal aggregation.
Data Source
AI summary
Estimating temporally consistent 3D human body shape, pose, and motion from a monocular video is a challenging task due to occlusions, poor lightning conditions, complex articulated body poses, depth ambiguity, and limited availability of annotated data. Embodiments of present disclosure provide a method for temporally consistent motion estimation from monocular video. A monocular video of person(s) is captured by a weak perspective camera and spatial features of body of the persons are extracted from each frame of the video. Then, initial estimates of body shape, body pose, and features of the weak perspective camera are obtained. The spatial features and initial estimates are then aggregated to obtain spatio-temporal features by a combination of self-similarity matrices between the spatial features, pose and the camera and self-attention maps of the camera features and the spatial features. The spatio-temporal aggregated features are then used to predict shape and pose parameters of the person(s).