Multi-Modal Language Model Reconstruction Feedback Loop
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-modal language models (MMLMs) face issues such as hallucinations, contradictions, and suggestibility when interpreting media inputs like images or sound signals, leading to inconsistencies and inaccuracies in their text responses.
Innovation Solution
A system is implemented that generates a multi-modal language model (MMLM) using a neural network architecture that includes a decoder network and a first media encoder neural network. The system processes input data elements containing media inputs and text tokens, generating token outputs and reconstructed media tokens. Through repeated modifications during a training phase, the model reduces the discrepancy between reconstructed and original media tokens, improving its grounding on media inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a multi-modal language model is trained to generate text responses from media inputs, then the model can perform multi-modal tasks, but the model produces hallucinations, contradictions, and suggestibility leading to inconsistencies
Solution Approach 1:
The patent implements a feedback mechanism where the decoder network generates reconstructed media tokens from text tokens, which are then compared against original media tokens encoded by the media encoder. This closed-loop feedback system provides a reconstruction loss signal that guides the training of the multi-modal language model, forcing it to maintain consistency between text responses and original media inputs, thereby reducing hallucinations and contradictions
Solution Approach 2:
The patent introduces two intermediary components: (1) a media encoder that transforms media inputs into media tokens as an intermediate representation, and (2) a decoder network that transforms text tokens back into reconstructed media tokens. These intermediaries create a bridge between media inputs and text responses, enabling the model to ground its text generation in the original media content while reducing suggestibility and hallucinations
2Reliability
If the model is trained to improve grounding on media inputs, then hallucinations and suggestibility are reduced, but the training complexity and computational resources increase
Solution Approach 1:
The patent segments the training system into distinct modular components: a media encoder for encoding media inputs into tokens, a multi-modal language model for generating text responses, and a decoder network for reconstructuring media tokens from text tokens. This segmentation allows each component to be trained and optimized independently while maintaining clear functional boundaries, reducing overall training complexity despite the added reliability requirements
3Measurement precision
If a decoder network is added to reconstruct media tokens, then the discrepancy between text responses and media inputs is reduced, but the system complexity increases
Solution Approach 1:
The patent merges the encoding and decoding functions into a unified training framework where the media encoder and decoder network work together as an integrated system. The decoder network reconstructs media tokens from text tokens generated by the multi-modal language model, and this reconstruction process is combined with the original encoding process to create a unified loss function that measures alignment between text responses and media inputs, thereby improving measurement precision while managing architecture complexity through functional integration
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method is proposed for generating a multi-modal language model (MMLM) neural network trained to perform a multi-modal task on an input data element comprising at least one media input (an image or sound signal) to generate a token output which is a text response to the input data element. The method employs a decoder network trained to use the token output to generate reconstructed media tokens. Repeated modifications are made to the MMLM to reduce a discrepancy between the reconstructed media tokens and media tokens generated from the media input(s) by a media encoder neural network.