3D Scene Processing With Text-Embedded Open-Vocabulary Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D scene processing models struggle to effectively detect objects belonging to novel classes beyond a limited close vocabulary due to the lack of annotated training data, leading to poor performance and over-reliance on human annotation, which is costly and impractical.
Innovation Solution
A method involving a detecting model that associates 3D data with open classes using a text-embedded semantic classifier and binary calibration module, leveraging image captions to learn from unannotated novel classes through a 3D data-text association, enabling open vocabulary detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a processing model is trained with limited annotated training data containing only base classes, then the model can effectively detect objects within the close vocabulary, but it cannot detect objects belonging to novel classes beyond the close vocabulary
Solution Approach 1:
The patent introduces image captions as an intermediary to bridge the gap between limited 3D annotated data and novel object classes. The text-embedded semantic classifier uses these captions to infer and detect novel classes without requiring direct 3D annotations for each novel class, thus expanding detection capability while minimizing additional annotation requirements.
Solution Approach 2:
The detecting model is designed to handle multiple functions: it can detect both base classes (from annotated data) and novel classes (from unannotated data with text embedding). This multi-functionality allows a single model to operate effectively across the open vocabulary without requiring separate models for each class type.
2Measurement precision
If more annotated training data for novel classes is collected to improve detection accuracy, then the model's ability to detect novel classes improves, but the cost and time required for human annotation increases significantly
Solution Approach 1:
The system enables self-service detection for novel classes by automatically generating text embeddings from image captions and using the text-embedded semantic classifier to identify novel objects without human intervention. This eliminates the need for manual annotation of each novel class while maintaining detection accuracy.
Solution Approach 2:
The patent performs preliminary actions by pre-training the semantic classifier with text embeddings and pre-processing image captions during the training phase. This preliminary preparation allows the model to quickly adapt to novel classes during inference without requiring time-consuming manual annotation at deployment time.
3Reliability
If the model relies heavily on annotated training data for each class, then it achieves good detection performance for known classes, but it becomes over-reliant on human annotation and cannot generalize to unseen classes
Solution Approach 1:
The patent changes the parameter representation from direct class labels to text-embedded semantic representations. By transforming the classification problem into a text-matching problem using embeddings, the model maintains reliability for base classes while gaining adaptability to novel classes through semantic similarity matching rather than exact label matching.
Data Source
AI summary
A method is proposed for detecting an object in a 3D scene, including obtaining a detecting model that describes an association relationship between a plurality of base classes of a plurality of objects and 3D data of the plurality of objects. A plurality of open classes of a plurality of candidate objects to be detected in a 3D scene are received, the plurality of open classes comprise the plurality of base classes and at least one novel class not in the plurality of base classes. A 3D portion is detected in 3D data of the 3D scene based on the detecting model and the plurality of open classes, the 3D portion corresponds to a target candidate object in the plurality of candidate objects. With this method, objects that belong to a novel class, not annotated in training data of the detecting model, may be detected from the 3D data.