Single-Tower Multimodal Generation With Block-Encoded Image Tokens
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face issues of 'negative transfer' when trained on multiple modalities, leading to degraded performance on individual modalities, such as images and text.
Innovation Solution
A system utilizing a token generation neural network and a separate image generation subsystem, like a diffusion model, with a block encoder that maps image pixels to independent tokens, allowing for conditioned image generation based on current output sequences, addressing negative transfer by maintaining image detail and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single model is trained on multiple modalities (images and text), then the model can perform multimodal tasks, but the performance on individual modalities degrades due to negative transfer
Solution Approach 1:
The system segments the multimodal processing into two independent components: a language model that handles text tokens and an image generation model that handles image tokens. This segmentation prevents negative transfer between modalities while maintaining multimodal capability through the unified token interface.
Solution Approach 2:
The patent introduces a token-based intermediary layer that mediates between the language model and image generation model. By converting images into tokens that can be processed by the language model, this intermediary enables seamless multimodal interaction without direct coupling between modalities, thus avoiding negative transfer.
2Manufacturing precision
If images are processed as raw pixels, then image detail is preserved, but the complexity of processing increases significantly
Solution Approach 1:
Instead of processing raw pixel data directly, the system creates a token-based copy or representation of image data. These image tokens capture essential visual information in a compressed format that is easier for the language model to process while still preserving sufficient detail for image generation tasks.
Solution Approach 2:
The patent transforms the parameter representation of images from raw pixel values to token embeddings. This parameter change reduces the dimensionality and complexity of image data while maintaining the essential features needed for processing, enabling the language model to handle images efficiently.
Data Source
AI summary
A computer-implemented method of generating multimodal data. The method comprises using a token generation neural network to generate, autoregressively, an output sequence of multimodal tokens, and in response to a next multimodal token being a start-of-image token, generating an image using an image generation subsystem conditioned on features representing the current sequence of multimodal tokens obtained from the token generation neural network. The method further comprises processing the image to convert pixels of the image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the image that maps a set of values of the pixels to a respective image token, and appending the sequence of image tokens to the current output sequence of multimodal tokens as the next multimodal tokens in the output sequence of multimodal tokens.