Natural Language Object Selection in Images Using Relative Position
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face difficulties in identifying and locating objects depicted in images or videos, such as products shown on TV or in movies, due to the lack of metadata and challenges in keyword-based searches, and existing systems struggle to accurately interpret natural language queries to perform relevant actions on those objects.
Innovation Solution
A machine learning-based system that combines automatic speech recognition, natural language understanding, and object detection to identify objects in images using natural language inputs, enabling visual searches and interactions with graphical user interfaces or augmented reality environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If keyword-based search is used to find objects in images, then search capability is provided, but search accuracy is poor and relevant objects cannot be reliably identified
Solution Approach 1:
The patent introduces embedding vectors as an intermediary between natural language queries and image objects. The query embedding and object embeddings serve as mediators that enable accurate matching through vector space comparison, resolving the contradiction by providing both ease of operation (natural language input) and high accuracy (embedding-based matching)
Solution Approach 2:
The patent transforms the search problem from keyword matching to embedding vector comparison. By changing the parameter space from discrete keywords to continuous embedding vectors, the system achieves both user-friendly operation and precise object identification through cosine similarity or other distance metrics
2Ease of operation
If natural language processing is implemented to interpret user queries, then user convenience is improved, but system complexity increases
Solution Approach 1:
The patent replaces complex rule-based natural language processing with a neural network-based embedding system. This substitution maintains user convenience (natural language input) while reducing system complexity by using end-to-end learned representations instead of hand-crafted rules and multiple processing stages
3Measurement precision
If embedding-based object selection is used, then object selection accuracy is improved, but computational requirements increase
Solution Approach 1:
The patent performs preliminary computation by pre-calculating and storing object embeddings during an offline phase. This preliminary action separates the heavy computational workload from real-time operation, allowing accurate embedding-based matching during inference while minimizing real-time energy consumption
Data Source
AI summary
Devices and techniques are generally described for selection of objects in image data using natural language input. In various examples, first image data representing at least a first object and first natural language data may be received. In some examples, first embedding data representing the first natural language data may be generated. Second embedding data representing the first image data may be generated. Relative location data indicating a location of the first object in the first image data relative to at least one other object may be generated. The first embedding data, the second embedding data, and the relative location data may be input into a multi-modal transformer model. The multi-modal transformer model may determine that the first natural language data relates to the first object.