Transformer 3D Pose Estimation with Geometry Bias
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-view 3D human pose reconstruction techniques require accurate 2D pose estimation, which can be affected by motion blur, occlusions, and self-occlusions, leading to inefficiencies and inaccuracies, especially in distributed real-time systems.
Innovation Solution
A transformer-based model is used to directly predict 3D coordinates of body joints from 2D skeletons estimated at edge nodes near cameras, efficiently transmitting these to a fusing node for processing, while incorporating geometry and confidence biases to handle corrupted inputs and varying camera configurations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 2D pose estimation is performed from multiple views using traditional techniques, then 3D pose reconstruction can be achieved, but the accuracy deteriorates due to motion blur, occlusions, and self-occlusions
Solution Approach 1:
The patent transforms the 2D pose estimation problem into a 3D pose estimation problem by directly predicting 3D joint coordinates from multi-view images using a transformer-based neural network. This dimensional transformation allows the model to leverage depth information and spatial relationships across multiple views, overcoming the limitations of 2D estimation affected by motion blur and occlusions.
Solution Approach 2:
The transformer-based model serves multiple functions simultaneously: it performs multi-view image fusion, 3D pose prediction, and robustness to occlusions and motion blur. The model processes images from multiple cameras and directly outputs 3D pose information, eliminating the need for separate 2D pose estimation and triangulation steps.
2Measurement precision
If information from all cameras is transmitted to a centralized processing unit for fusion, then pose detection accuracy is improved, but processing requirements and bandwidth consumption increase
Solution Approach 1:
The patent segments the processing task by distributing initial image processing to edge nodes near each camera, where 2D pose information is extracted locally. Only the essential 2D pose data from each view is transmitted to the centralized processing unit, rather than transmitting all raw images. This segmentation reduces bandwidth requirements while maintaining the ability to perform accurate multi-view fusion and 3D pose estimation.
3Measurement precision
If 2D pose estimation is enhanced by fusing information from multiple views, then accuracy is improved, but system complexity increases due to centralized processing requirements
Solution Approach 1:
The patent simplifies the system architecture by skipping the complex multi-view 2D pose fusion process and directly predicting 3D poses from multi-view images using a transformer-based model. This dimensional change from 2D to 3D processing eliminates the need for complex intermediate fusion steps and reduces overall system complexity.
Solution Approach 2:
The patent uses a transformer-based neural network that learns to copy and integrate relevant information from multiple views directly into 3D pose predictions. The model automatically identifies and copies useful features from different camera views, eliminating the need for manual design of fusion algorithms and reducing system complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method of determining a three-dimensional (3D) pose, the method comprising: obtaining two-dimensional (2D) pose information, the 2D pose information indicating a position of a plurality of joints of an object in each of a plurality of 2D images of the object captured from different respective viewpoints at one or more times; generating a set of input tokens, each input token indicating a position of one of the joints in one of the images, a classification of the joint and a time at which the image was captured, the position of the joint being defined as a ray passing through the 2D position of the joint in the image and a camera centre of the image; inputting the set of input tokens to an encoder of a transformer, the encoder comprising a multi-head self-attention mechanism; generating a query indicating a classification and a time for one or more joints for which a 3D position is to be predicted; and inputting the query and an output of the encoder to a decoder of the transformer to generate, as an output of the decoder, a predicted 3D position of the one or more joints of the query, the decoder comprising a multi-head cross-attention mechanism; wherein the multi-head self-attention mechanism of the encoder is biased according to at least one of a distance between the rays representing the positions of the joints of the input tokens and a confidence of the 2D position of the joint of each input token.