Transformer Pose Prediction Using Ordinal Regression

The patent merges RGB images and depth images into a combined input for the transformer model. The RGB image provides high-quality visual information with clear edges and colors, while the depth image provides three-dimensional spatial structure information. By processing both image types simultaneously through the transformer architecture, the system achieves accurate 3D pose prediction while compensating for the weaknesses of individual modalities.

The transformer model acts as an intermediary that processes and integrates features from both RGB and depth images. The model extracts features from the combined input and uses ordinal regression to predict joint point coordinates, effectively translating the noisy depth information into accurate pose predictions by leveraging the complementary strengths of both image types.

The patent replaces conventional pose estimation methods with a transformer-based deep learning approach. Instead of using traditional computer vision algorithms that struggle with noisy depth images, the system employs a transformer model with self-attention mechanisms to automatically learn and extract relevant features from combined RGB and depth inputs, achieving superior accuracy in challenging conditions.

The patent changes the fundamental parameters of the pose estimation approach by using ordinal regression to predict joint point coordinates directly from image features. Rather than estimating poses through multiple intermediate steps or assumptions, the model directly regresses 3D coordinates of joint points, improving precision while maintaining computational efficiency.