3D Keypoint Detection Training With Multi-Camera Self-Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D keypoint detection models struggle with accuracy and reliability due to the incompleteness of 3D image data, particularly in occluded areas, as they primarily rely on 2D image-based inputs, limiting their predictive capabilities.
Innovation Solution
A system and method utilizing multiple camera devices to capture a 3D entity from different angles, generating self-labeled datasets through a pre-trained model to fine-tune a 3D keypoint detection model, incorporating 3D convolutional neural networks for enhanced keypoint detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If 2D image-based keypoint detection models are used, then the model construction is simple and fast, but the accuracy and reliability of keypoint detection in occluded areas deteriorates
Solution Approach 1:
The patent transitions from 2D image-based keypoint detection to 3D point cloud-based detection. By utilizing three-dimensional spatial information from multiple camera devices, the model can accurately detect keypoints on occluded surfaces (e.g., back of the body) that are invisible in 2D images, thereby resolving the contradiction between construction simplicity and detection accuracy in occluded areas
Solution Approach 2:
The patent divides the keypoint detection task into multiple independent detection processes from different camera angles. Each camera device captures 3D data from its own perspective, and the model processes each viewpoint separately before fusing results, enabling accurate detection of occluded keypoints by leveraging information from unoccluded viewpoints
2Ease of manufacture
If supervised learning with labeled datasets is used, then the model training is straightforward, but the difficulty in labeling training data limits predictive capabilities
Solution Approach 1:
The patent implements self-supervised learning where the model generates its own training labels by detecting keypoints on unoccluded surfaces from multiple camera viewpoints. The system automatically creates labeled training data from raw 3D point cloud inputs without requiring manual annotation, thereby maintaining training simplicity while significantly improving predictive capability through abundant self-generated labeled data
Solution Approach 2:
The patent performs preliminary keypoint detection on easily obtainable unoccluded surfaces before final keypoint determination. By first detecting keypoints on visible surfaces from multiple angles and using these as preliminary labels for training, the model builds a foundation that improves final predictive accuracy on occluded areas while keeping the training process straightforward
3Loss of information
If multiple camera devices are used to capture 3D data from different angles, then the completeness of 3D data improves, but the system complexity increases
Solution Approach 1:
The patent merges 3D data from multiple camera devices into a unified point cloud representation. By combining depth information and spatial coordinates from multiple viewpoints into a single integrated 3D dataset, the system achieves complete coverage of the target object including occluded areas, while managing complexity through data fusion rather than processing multiple separate systems
Solution Approach 2:
The patent introduces a coordinate transformation and data fusion module as an intermediary between multiple camera devices and the keypoint detection model. This intermediary aligns coordinate systems, fuses point cloud data from different viewpoints, and presents unified 3D input to the model, thereby achieving complete 3D data coverage while abstracting away the complexity of multi-camera coordination
Data Source
AI summary
A method for training a 3D keypoint detection model is provided. The method includes the step of using a labeled dataset to train a pre-trained model. The method further includes the step of obtaining multiple sets of 3D data associated with a 3D entity from multiple camera devices. The method further includes the step of inputting the multiple sets of 3D data into the pre-trained model to obtain multiple sets of predicted keypoint coordinates output by the pre-trained model. The method further includes the step of generating a self-labeled dataset based on the multiple sets of 3D data and the multiple sets of predicted keypoint coordinates. The method further includes the step of using the self-labeled dataset to train the pre-trained model to create a fine-tuned model.