Open-Vocabulary 3D Target Retrieval via 2D Segmentation Projection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing open-vocabulary 3D target retrieval models exhibit low accuracy in retrieving and localizing 3D objects due to their direct application in 3D scenes.
Innovation Solution
A retrieval method utilizing a large language model (LLM), open-vocabulary 2D detection and segmentation model, object filtering module, and 3D projection module to enhance and project 2D segmentation masks into 3D scenes, incorporating tracking and image screening modules to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If existing open-vocabulary 3D target retrieval models are directly applied to 3D scenes, then the device complexity is reduced, but the accuracy of retrieval results deteriorates
Solution Approach 1:
The patent segments the 3D target retrieval task into multiple stages: 2D detection in image sequences, tracking to form tracking sequences, and 3D projection. This segmentation allows each module to specialize in its subtask, improving overall accuracy while managing complexity through modular architecture.
Solution Approach 2:
The patent transitions from 2D image space to 3D scene space through the 3D projection module. By projecting 2D segmentation masks onto 3D point clouds and performing corrections in 3D space, the system achieves accurate 3D target localization that cannot be obtained through 2D methods alone.
2Measurement precision
If multiple processing modules are integrated to improve retrieval accuracy, then the measurement precision improves, but the device complexity increases
Solution Approach 1:
The patent employs multi-modal models that perform multiple functions: the open-vocabulary 2D detection model both detects targets and generates segmentation masks, the tracking module both tracks targets across frames and forms tracking sequences, and the 3D projection module both projects 2D masks to 3D space and corrects projection errors. This multi-functionality reduces the need for separate specialized modules.
Solution Approach 2:
The patent introduces intermediate representations to bridge different processing stages: 2D segmentation masks serve as intermediaries between detection and 3D projection, tracking sequences serve as intermediaries between detection and filtering, and the multi-modal model outputs serve as intermediaries between 2D analysis and 3D localization. These intermediaries enable modular processing while maintaining high accuracy.
Data Source
AI summary
A retrieval method for an open-vocabulary 3D target, a device and a storage medium are provided. The method includes: inputting text description information of a target object and an image sequence of a real scene into an open-vocabulary 3D target retrieval model. The retrieval model includes an LLM, an open-vocabulary 2D detection and segmentation model, an object filtering module and a 3D projection module. The LLM is used to enhance the text description information to obtain names of a plurality of candidate objects of the target object. The open-vocabulary 2D detection and segmentation model is used to detect and segment according to the names of the candidate objects, to obtain 2D segmentation masks of the plurality of candidate objects. The object filtering module is used to determine the target object from the candidate objects according to the text description information and a 2D segmentation mask of each candidate object.