3D Keypoint Training for Vision Tracking Under Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks for vision-based tracking face challenges in accurately predicting 2D keypoints due to occlusions and environmental variables, while 3D keypoint detectors require extensive datasets for effective training.
Innovation Solution
A method that projects 2D keypoints into 3D space, incorporates keypoint depth information, and uses a 3D model with known rotational and translational information to train the keypoint detector, integrating translational and rotational loss values to enhance accuracy and adaptability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 2D keypoint detectors are trained with a single image, then 2D keypoint prediction accuracy is improved, but performance deteriorates in scenarios with occlusions or environmental variables
Solution Approach 1:
The patent transitions from 2D keypoint detection to 3D keypoint detection by adding depth information. The neural network predicts 3D keypoints (x, y, z coordinates) instead of only 2D coordinates, enabling the system to reason about depth and spatial relationships. This dimensional enhancement allows the detector to maintain accuracy under occlusions by understanding the 3D structure of objects and predicting keypoints that are not visible in the 2D image plane.
2Loss of information
If 3D keypoint detectors are trained with multiple object views, then depth information is improved, but extensive datasets are required for effective training
Solution Approach 1:
The patent introduces a depth estimation module as an intermediary that predicts depth maps from single 2D images. These depth maps serve as intermediate representations that provide depth information without requiring multiple views during training. The depth maps are generated using camera intrinsics and geometric constraints, acting as a mediator between 2D image input and 3D keypoint prediction, thereby reducing the need for extensive multi-view training datasets.
3Measurement precision
If 3D model transformation with rotational and translational information is used, then 3D keypoint prediction accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent pre-computes transformation matrices (rotation and translation) from the 3D model and stores them for efficient retrieval during inference. Instead of performing complex 3D transformations in real-time, the system prepares transformation data in advance during the training phase or model initialization. This preliminary action reduces the computational burden during actual keypoint detection while maintaining the accuracy benefits of 3D geometric transformations.
Data Source
AI summary
Methods, apparatuses and systems of training a neural network using in a vision-based tracking system are disclosed. A two-dimensional (2D) image of at least a portion of an object is received via a camera that is fixed, relative to the object. Subsequently, a keypoint detector predicts a set of keypoints on the object in the 2D image, generating predicted 2D keypoints. These predicted 2D keypoints are then projected into three-dimensional (3D) space, and keypoint depth information is added to generate predicted 3D keypoints. To enhance the training process, a 3D model of the object is utilized. Known rotational and translational information of the object in the 2D image is incorporated to known 3D model keypoints, resulting in transformed 3D model keypoints. Following this, a comparison between predicted 3D keypoints and transformed 3D model keypoints is made to calculate a loss value. The training process is further refined using an optimizer, minimizing the loss value during a training period.