Multimodal Retrieval Index Using Universal Embeddings and Throttling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vector-based retrieval systems often exhibit poor performance due to mode-specific bias and resource overutilization, especially when handling queries expressed in multiple input modes.
Innovation Solution
A computing system that uses an encoder system to generate input-agnostic target item embeddings and a retrieval system to match queries across various input modes, while throttling processing based on platform capabilities, using fixed and non-fixed weights to reduce bias and resource overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If vector-based retrieval systems use traditional encoding methods, then they can handle queries in multiple input modes, but they exhibit mode-specific bias and poor retrieval performance
Solution Approach 1:
The encoder is divided into separate input-embedding subsystems for different input modes (text, image, audio, video), each specialized in processing its specific mode. This segmentation allows each subsystem to be optimized for its input type while the unified architecture eliminates mode-specific bias in the retrieval process.
Solution Approach 2:
The system creates a universal retrieval index that can accommodate queries from any input mode by converting all modes into a common embedding space. The retrieval system universally handles text, image, audio, and video queries through a single unified process, eliminating the need for mode-specific retrieval mechanisms.
2Productivity
If the encoder system processes all input items without throttling, then retrieval completeness is maintained, but execution platform resources may be overwhelmed
Solution Approach 1:
The encoder system dynamically adjusts its processing behavior based on platform capabilities. It throttles the number of processing operations performed by the encoder according to the assessed processing capability of the execution platform, allowing the system to adapt between high-throughput and resource-conserving modes.
Solution Approach 2:
The system changes the parameter of processing operations based on platform capabilities. By assessing the execution platform's processing capability, the system adjusts the quantity and intensity of encoding operations accordingly, optimizing the balance between retrieval productivity and resource consumption.
3Adaptability or versatility
If the system trains all encoder components from scratch, then adaptation to new input modes is achieved, but training time and computational resources increase
Solution Approach 1:
The encoder is segmented into separate input-embedding subsystems for different input modes. When a new input mode is introduced, only the corresponding subsystem needs to be trained, while other subsystems retain their pre-trained weights. This modular approach enables efficient adaptation to new modes without retraining the entire encoder.
Solution Approach 2:
The system performs preliminary training of individual input-embedding subsystems in isolation before integrating them into the unified encoder. This preliminary action allows each subsystem to be optimized for its specific input mode without affecting other modes, reducing overall training time and computational resources.
Data Source
AI summary
A technique uses an encoder system to produce an index of target item embeddings. Each target item embedding is input-agnostic and universal in the sense that different expressions of a target concept, produced using different combinations of input modes, map to the same target item embedding in the index. The encoder system throttles the amount of computations it performs based on the assessed capabilities of an execution platform. A retrieval system processes a multimodal input query by first generating a candidate set of target item embeddings in the index that match the input query, and then using a filtering operation to identify those target item embeddings that are most likely to match the input query. The encoder system and the retrieval system rely on language-based components having weights that are held constant during a training operation. Other weights of these systems are updated during the training operation.