3D Object Detection with Text-Image Fusion for Unseen Classes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D object detection technologies struggle to accurately identify objects of classes not included in training data, particularly when using a combination of RGB images and LiDAR data.
Innovation Solution
A method and apparatus that fuse image and text features to enhance 3D object detection by using a transformer-based model, incorporating positional embedding and ROI pooling to improve the detection of new object classes in point cloud data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional 3D object detection methods are used, then detection speed is maintained, but accuracy for new object classes deteriorates
Solution Approach 1:
The patent introduces text prompts as an intermediary element that bridges the gap between existing training data and new object classes. The text prompt encoder converts natural language descriptions into feature representations that guide the detection model to identify objects of new classes without requiring retraining, thereby improving both accuracy and adaptability for unseen object categories.
Solution Approach 2:
The patent creates a composite feature representation by fusing multiple data modalities: image features from the RGB image, LiDAR features from point cloud data, and text features from the prompt. This multi-modal fusion allows the model to leverage complementary information from different sources, significantly improving detection accuracy for new object classes while maintaining robustness.
2Measurement precision
If multi-modal feature fusion is implemented, then detection accuracy for new classes improves, but computational complexity increases
Solution Approach 1:
The patent segments the complex detection task into distinct processing streams: an image backbone for visual features, a LiDAR backbone for 3D point cloud features, and a text prompt encoder for language guidance. Each stream processes its modality independently before fusion, reducing the computational burden of processing all data simultaneously while maintaining the benefits of multi-modal fusion for improved accuracy.
3Manufacturing precision
If 3D bounding box accuracy is improved through positional embedding, then localization precision increases, but processing time increases
Solution Approach 1:
The patent performs preliminary 2D object detection and region of interest (ROI) identification in the image domain before processing the full 3D point cloud data. By first localizing objects in the 2D image using the image backbone and text prompts, the system can then focus computational resources on generating 3D bounding boxes only for detected regions, significantly reducing processing time while maintaining high localization accuracy through positional embedding.
Data Source
AI summary
A method and apparatus with object detection are disclosed. A method of detecting an object is performed by one or more processors and the method includes: obtaining a feature of an object region of interest (ROI) of an object in an image captured by a first sensor, the feature obtained based on a text-image fusion feature that is a fusion of an image feature of the image and of a text feature of a text, where the text corresponds to the image; obtaining a query corresponding to the object ROI, based on the feature of the object ROI; and obtaining, based on the query corresponding to the object ROI, from a transformer-based object detection model, object detection information in a point cloud that is captured by a second sensor and that corresponds to the image.