Pose Relation Transformer for Occluded Keypoint Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing human pose estimation methods suffer from poor performance when occlusions occur, leading to incorrect pose estimation and errors in applications like human-robot interaction due to self-occlusion, object occlusion, and being out-of-frame issues in single-view camera setups.
Innovation Solution
A pose relation transformer is employed to refine occluded keypoints by leveraging Masked Joint Modeling (MJM), which uses self-attention and graph convolution to reconstruct occluded joints based on visible joints and joint correlations, and can be integrated with existing keypoint detectors with minimal computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing keypoint detectors are used for human pose estimation, then the system can operate with simple architecture, but the detection accuracy deteriorates significantly when occlusions occur
Solution Approach 1:
The patent introduces an occlusion refinement module as an intermediary component between the keypoint detector and the pose estimation system. This module detects occluded keypoints and refines their positions using contextual information from visible body parts, thereby improving detection accuracy without fundamentally changing the overall system architecture. The intermediary module handles the occlusion problem specifically, allowing the main detector to remain simple while achieving better overall performance.
2Reliability
If a robust occlusion handling method is implemented, then the reliability improves, but the computational overhead and device complexity increase
Solution Approach 1:
The patent segments the pose estimation problem into distinct components: initial keypoint detection, occlusion detection, and occlusion refinement. By dividing the problem into separate modular stages, each component can be optimized independently. The occlusion refinement module only processes detected occluded keypoints rather than the entire pose estimation task, reducing computational overhead while improving reliability under occlusion conditions.
3Measurement precision
If a comprehensive occlusion refinement model is used, then the measurement precision improves, but the loss of information from masked keypoints becomes a challenge
Solution Approach 1:
The occlusion refinement module uses feedback from visible keypoints and their spatial relationships to infer the positions of occluded keypoints. The system leverages the known anatomical constraints and relative position relationships of human body parts to guide the refinement process. This feedback mechanism allows the system to recover information about occluded keypoints indirectly through their relationships with visible parts, reducing the effective information loss.
Data Source
AI summary
An approach for pose estimation is disclosed that can mitigate the effect of occlusions. A POse Relation Transformer (PORT) module is configured to reconstruct occluded joints given the visible joints utilizing joint correlations by capturing the implicit joint occlusions. The PORT module captures the global context of the pose using self-attention and a local context by aggregating adjacent joint features. To train the PORT module to learn joint correlations, joints are randomly masked and the PORT module learns to reconstruct the masked joints, referred to as Masked Joint Modeling (MJM). Notably, the PORT module is a model-agnostic plug-in for pose refinement under occlusion that can be plugged into any existing or future keypoint detector with substantially low computational costs.