Multimodal Prompt Processing Through Textual Tokens and Text-Only LLMs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional large language models (LLMs) are monomodal and can only process and generate textual inputs and outputs, limiting their ability to handle multimodal inputs such as audio and image data effectively.
Innovation Solution
A multimodal system comprising a multimodal encoder, a vector quantization model, and a text-only LLM is used to convert multimodal inputs into a sequence of textual tokens, enabling the LLM to generate corresponding textual outputs by training the system on multimodal training samples and fine-tuning the LLM's parameters while keeping pre-trained parameters frozen.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional LLMs are used, then the system structure remains simple, but the ability to handle multimodal inputs is limited
Solution Approach 1:
The system segments the multimodal processing task into distinct components: a multimodal encoder for extracting features from audio/image inputs, a vector quantization model for converting features to textual tokens, and a text-only LLM for generating final outputs. This segmentation allows each component to specialize in one modality type while maintaining overall system manageability.
Solution Approach 2:
The patent introduces textual tokens as an intermediary representation between multimodal inputs and the text-only LLM. The vector quantization model converts multimodal features into discrete textual tokens that the LLM can process, effectively mediating between different modalities without requiring the LLM itself to be multimodal.
2Adaptability or versatility
If a multimodal system is built to process audio and image data, then the processing capability is enhanced, but the system complexity increases
Solution Approach 1:
The text-only LLM serves multiple functions: it processes the converted textual tokens from various modalities (audio, image), generates the final output text, and maintains its pre-trained language understanding capabilities. This universality allows one model to handle multiple input types without requiring separate specialized models for each modality.
Solution Approach 2:
The multimodal encoder and vector quantization model perform preliminary processing of audio and image inputs, converting them into textual token sequences before they reach the LLM. This preliminary action prepares the data in a format suitable for the text-only LLM, reducing the complexity burden on the main language model.
3Measurement precision
If the LLM is fine-tuned on multimodal training samples, then the processing accuracy is improved, but the training time and computational resources increase
Solution Approach 1:
The patent applies partial fine-tuning by updating only a subset of the LLM's parameters (e.g., adapter layers or specific modules) rather than all parameters. This approach achieves sufficient adaptation to multimodal data while significantly reducing training time and computational resources compared to full fine-tuning.
Solution Approach 2:
The system changes the state of model parameters during fine-tuning, transitioning from frozen pre-trained parameters to partially updated parameters that are adapted to multimodal inputs. This parameter transformation enables the text-only LLM to process multimodal data effectively without requiring complete retraining.
Data Source
AI summary
A multimodal system includes a multimodal encoder, a vector quantization model, and a text-only large language model (LLM). The multimodal encoder is configured to receive, as input, a multimodal input for a prompt, and generate, based on the multimodal input, a sequence of embeddings. The vector quantization model is configured to receive, as input, the sequence of embeddings generated by the multimodal encoder, and generate, based on the sequence of embeddings, a sequence of textual tokens. The text-only LLM is configured to receive, as input text, a natural language command for the prompt and the sequence of textual tokens generated by the vector quantization model, and generate, based on the natural language command and the sequence of textual tokens, a corresponding textual output.