Multimodal Asset Retrieval via Embedding Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search and retrieval mechanisms are inadequate for efficiently searching and retrieving multimodal content, which includes multiple types of content such as text, images, GIFs, icons, illustrations, etc., within a single document. These systems struggle to handle the complexity of multimodal content and often require manual annotation and model retraining, making them inefficient and costly.
Innovation Solution
A data processing system that uses a trained representation ML model to encode search queries into query embedding representations, which are then compared to multimodal tensor representations of candidate assets. This system allows for the identification and retrieval of multimodal assets based on the comparison, providing a more efficient and accurate search mechanism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current search mechanisms rely on text queries, then users can search for content, but the system cannot accurately retrieve multimodal content with multiple content types
Solution Approach 1:
The patent transforms the search query from text-based parameters to embedding-based parameters. By converting both the query and multimodal content into embedding representations through a trained ML model, the system changes the parameter space to enable accurate retrieval of content with multiple content types (text, images, GIFs, icons, illustrations) simultaneously.
Solution Approach 2:
The patent introduces an intermediary representation layer (embeddings) between the text query and the multimodal content. The trained ML model acts as a mediator that converts the text query into embedding representations that can be compared with embeddings from various content types, enabling accurate retrieval without requiring the user to formulate precise text queries for each content type.
2Reliability
If manual annotation is used for training ML models, then the model can be trained accurately, but the process becomes too expensive and time-consuming
Solution Approach 1:
The patent implements self-service through automatic embedding generation. The trained ML model automatically generates embedding representations for both the query and the multimodal content without requiring manual annotation. This self-service approach eliminates the need for human annotators while maintaining accurate retrieval performance.
Solution Approach 2:
The patent performs preliminary training of the ML model using a training data set, after which the model can generate embeddings for new content without further manual intervention. The preliminary training action enables subsequent automatic processing of large volumes of content efficiently, reducing both time and cost.
3Quantity of substance
If the number of content items is large, then more content is available for search, but manually annotating the data set becomes too expensive
Solution Approach 1:
The patent enables the system to self-process large volumes of content by automatically generating embeddings for all content items. This eliminates the need for manual annotation of each content item, making it feasible to work with large content volumes at low cost.
Solution Approach 2:
The patent replaces the mechanical process of manual annotation with an automated ML-based embedding generation system. This substitution allows the system to handle large quantities of content efficiently, scaling from manual processing to automated processing as content volume increases.
4Measurement precision
If trained ML models are used for search, then retrieval accuracy improves, but the models become too large to deploy and operate
Solution Approach 1:
The patent segments the large ML model into a query encoding component and a content embedding component. The query is encoded into embeddings using the trained model, then compared with pre-computed content embeddings. This segmentation allows the system to maintain high retrieval accuracy while reducing the computational burden during deployment.
Solution Approach 2:
The patent performs preliminary computation of content embeddings and stores them in advance. This preliminary action shifts the computational burden from real-time processing during search to offline processing during content indexing, making the deployed system more efficient and manageable.
Data Source
AI summary
A system and method and for retrieving one or more one or more multimodal assets includes receiving a search query for searching for one or more multimodal assets from among a plurality of candidate multimodal assets, encoding the search query into one or more query embedding representations via a trained query representation machine-learning (ML) model, comparing, via a matching unit, the one or more query embedding representations to a plurality of multimodal tensor representations, each of the plurality of multimodal tensor representations being a representation of one of the plurality of candidate multimodal assets, and identifying, based on the comparison, at least one of the plurality of the candidate multimodal assets as a search result for the search query, and providing the at least one of the plurality of the candidate multimodal assets for display as the search result.