Single-Tower Multimodal Token Generation to Reduce Negative Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face issues of 'negative transfer' when trained on multiple modalities, adversely affecting performance on individual modalities, such as images and text.
Innovation Solution
A system utilizing a token generation neural network and an image generation subsystem, including a block encoder, processes multimodal data by generating images conditioned on features representing the current output sequence, with the block encoder mapping image pixels to tokens independently and using a deterministic encoding scheme, addressing negative transfer and improving consistency across generated images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a model is trained on multiple modalities simultaneously, then the model can generate data across different modalities, but performance on individual modalities deteriorates due to negative transfer
Solution Approach 1:
The system segments the multimodal generation task into separate specialized components: a text generation model for textual tokens and an image generation model for visual tokens. This segmentation allows each component to specialize in its modality, avoiding negative transfer while maintaining overall multimodal capability through coordinated operation of the segmented components.
Solution Approach 2:
The system introduces a unified token space as an intermediary representation that bridges text and image modalities. Both text tokens and image tokens are represented in a common token vocabulary, allowing the models to operate in a unified framework without direct competition, thus preventing negative transfer while enabling multimodal generation.
2Reliability
If images are generated conditioned on token sequences, then consistent image generation across modalities is achieved, but computational complexity increases
Solution Approach 1:
The system employs universal diffusion models that can handle both text-to-image and image-to-image generation tasks through a unified architecture. The same image generation model processes conditioned on token sequences whether the input is textual description or another image, achieving consistency across modalities while avoiding the need for separate specialized systems.
3Reliability
If a unified token space is used for both text and images, then negative transfer is reduced, but the complexity of token encoding and processing increases
Solution Approach 1:
The system uses self-service encoding where text is tokenized using standard text tokenization methods and images are converted to tokens through visual tokenization, with each modality serving its own encoding needs independently. The unified token space automatically handles the integration without requiring complex cross-modality transformation logic, reducing overall processing complexity.
Data Source
AI summary
A computer-implemented method of generating multimodal data. The method comprises using a token generation neural network to generate, autoregressively, an output sequence of multimodal tokens, and in response to a next multimodal token being a start-of-image token, generating an image using an image generation subsystem conditioned on features representing the current sequence of multimodal tokens obtained from the token generation neural network. The method further comprises processing the image to convert pixels of the image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the image that maps a set of values of the pixels to a respective image token, and appending the sequence of image tokens to the current output sequence of multimodal tokens as the next multimodal tokens in the output sequence of multimodal tokens.