Text-Image Fusion for 3D Object Detection of Unseen Classes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D object detection technologies struggle to accurately identify objects of new classes not included in training data, and there is a need to improve the accuracy of 3D bounding boxes in object detection systems.
Innovation Solution
A method and apparatus that fuse image and text features to enhance 3D object detection, using a transformer-based model to process data from cameras and LiDAR sensors, incorporating positional embeddings and text-image fusion to detect objects in point clouds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional 3D object detection methods are used, then detection speed is maintained, but the ability to detect objects of new classes not included in training data is poor
Solution Approach 1:
The patent introduces text as an intermediary element that bridges the gap between known training data and unknown object classes. Text descriptions serve as a mediator that can represent new object classes without requiring retraining, allowing the system to adapt to new classes while maintaining detection accuracy through the transformer model's ability to process and understand text-image relationships.
Solution Approach 2:
The patent adds a text dimension to the traditional image-based detection framework. By incorporating text features alongside image features in a multi-modal transformer architecture, the system gains an additional dimension for representing and reasoning about object classes, enabling it to detect new classes that were not present in the training data.
2Adaptability or versatility
If text-image fusion features are used to detect new object classes, then adaptability to new classes improves, but computational complexity increases
Solution Approach 1:
The patent segments the feature processing into distinct image feature extraction and text feature extraction pathways, which are then fused in the transformer model. This segmentation allows for efficient parallel processing of different modalities while maintaining the ability to capture complex relationships between text and image features for detecting new object classes.
3Measurement precision
If 3D bounding box accuracy is improved using object region detection, then detection precision increases, but processing time increases
Solution Approach 1:
The patent performs preliminary 2D object region detection in the image plane before proceeding to 3D bounding box estimation. This preliminary action identifies candidate object regions and their 2D bounding boxes, which are then used to guide the 3D detection process, reducing the search space and improving both accuracy and efficiency of the final 3D bounding box estimation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and apparatus with object detection are provided. A method of detecting an object is performed by one or more processors and the method comprises: obtaining a feature of an object region of interest (ROI) of an object in an image captured by a first sensor, the feature obtained based on a text-image fusion feature that is a fusion of an image feature of the image and of a text feature of an input text, where the text corresponds to the image; obtaining a query corresponding to the object ROI, based on the feature of the object ROI; and obtaining, from a transformer-based object detection model, based on the query corresponding to the object ROI, object detection information in a point cloud that is captured by a second sensor and that corresponds to the image.