Multi-Modal Query Processing With Gated Cross-Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks struggle to effectively process multi-modal query inputs comprising both text and data items of different modalities, such as images or audio, without requiring extensive training data and computational resources.
Innovation Solution
A query processing system utilizing a modality network with a pre-trained encoder and a data-item-token processing model, including token processing layers and gated cross-attention layers, to generate a response to a query input that combines text and data items, reducing the need for extensive training data and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If current neural networks process multi-mode query inputs combining text and other modalities, then they can handle diverse input types, but they struggle to generate accurate responses efficiently
Solution Approach 1:
The system segments the processing of multi-mode inputs by separating text token processing from other modality processing. Text inputs are processed through a language model component, while other modalities (images, sound) are processed through dedicated modality networks that generate compressed representations. This segmentation allows each component to specialize in its modality type, improving overall accuracy while maintaining versatility.
Solution Approach 2:
The patent introduces gated cross-attention layers as intermediary components that bridge the language model and modality-specific processors. These layers receive compressed representations from modality networks and integrate them with text token processing, enabling accurate response generation by mediating between different input modalities and the final output generation.
2Reliability
If neural networks process multi-mode query inputs with extensive training data, then accuracy improves, but computational resources increase
Solution Approach 1:
The system extracts and processes only the essential features from each modality type. Modality networks generate compressed representations that capture the most important information from images, sound, or other non-text inputs. This extraction approach maintains high processing accuracy while significantly reducing the computational resources needed compared to processing raw multi-mode data through a single large network.
Solution Approach 2:
The patent changes the parameter representation of multi-mode inputs by converting them into a unified token format that the language model can process. Other modalities are transformed into compressed representations with optimized dimensional parameters, allowing efficient processing through the existing language model infrastructure without requiring extensive retraining or additional computational resources.
3Adaptability or versatility
If the system processes zero-shot and few-shot queries, then versatility improves, but accuracy deteriorates
Solution Approach 1:
The system performs preliminary processing of each modality type through dedicated networks that generate compressed representations before passing them to the language model. This preliminary action prepares the data in an optimized format that enables accurate processing even when encountering new or unseen query types, allowing the system to handle zero-shot and few-shot queries effectively without requiring extensive prior training on specific query formats.
Data Source
AI summary
A query processing system is described which receives a query input comprising an input token string and also at least one data item having a second, different modality, and generates a corresponding output token string.