Unified Object Foundation Model for Zero-Shot Perception Tasks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer vision models struggle with single-task learning frameworks that limit their applicability to object-level tasks, and open-vocabulary detection models face challenges in simultaneously excelling in localization and recognition due to language model biases.
Innovation Solution
A unified object processing model comprising an image encoder, text encoder, visual prompt encoder, and object decoder is trained using a diverse dataset with labeled object perception information, enabling simultaneous solution of multiple object perception tasks through a unified input and output paradigm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single-task learning framework is used, then the model structure is simple, but the applicability to object-level tasks is limited
Solution Approach 1:
The patent implements a unified object processing model that can handle multiple object perception tasks (detection, segmentation, tracking) within a single framework. The model uses shared components including a visual encoder, language encoder, and cross-attention mechanism that work across different tasks, eliminating the need for separate single-task models while maintaining task-specific performance.
2Measurement precision
If open-vocabulary detection models are used, then the recognition capability is improved, but the localization accuracy deteriorates due to language model biases
Solution Approach 1:
The patent introduces a cross-attention mechanism as an intermediary component that processes both visual features and language prompts simultaneously. This cross-attention layer acts as a mediator between the visual encoder and language encoder, allowing the model to attend to specific visual regions based on language descriptions while maintaining accurate localization by grounding language concepts in precise visual features rather than relying solely on language model biases.
3Reliability
If a unified model for multiple tasks is trained, then the generalization capability is improved, but the training data requirements increase
Solution Approach 1:
The patent merges multiple object perception tasks (detection, segmentation, tracking) into a single unified training framework. By combining task-specific datasets and training objectives within one model, the system leverages shared representations and patterns across tasks, reducing the overall data requirement compared to training separate specialized models while maintaining strong generalization capability across all tasks.
Data Source
AI summary
Embodiments of the present disclosure provide a method, device, and medium for training a large scale object foundation model. The method comprises obtaining a training dataset comprising a plurality of subsets for a plurality of object perception tasks, wherein a sample in the training dataset comprises an image with an object, a prompt indicating the object, and labeled object perception information of the image. The method further comprises generating, by the image encoder, an image feature based on the image. The method further comprises generating, by the text encoder or the visual prompt encoder, a prompt embedding based on the prompt. The method further comprises generating, by the object decoder, object perception information of the object based on the image feature and the prompt embedding. In addition, the method further comprises training the object processing model based on the generated object perception information and the labeled object perception information.