Multi-Modal Model Training via Pixel Block Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for training multi-modal models require significant computational resources and large amounts of training data, relying on visual tokenizers that are accuracy-dependent, making them inefficient and complex.
Innovation Solution
The method involves dividing images into pixel blocks, masking some blocks to generate a masked sequence, and using the corresponding text to generate target blocks, training the model on differences between original and target blocks, thereby reducing data requirements and avoiding the need for additional visual tokenizers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional methods use visual tokenizers to process images for multi-modal model training, then the model can process multi-modal data, but the computational resources required and data requirements become significant and complex
Solution Approach 1:
The patent extracts and removes the visual tokenizer component from the conventional training system. Instead of using complex visual tokenizers that require significant computational resources, the method directly operates on pixel blocks, eliminating the need for this intermediate processing layer and thereby reducing system complexity while maintaining multi-modal processing capability
Solution Approach 2:
The patent segments the image into pixel blocks as the basic processing unit, replacing the conventional approach of using visual tokenizers. This segmentation allows direct manipulation of image data at the pixel block level, simplifying the training process and reducing computational requirements while enabling effective multi-modal data processing
2Adaptability or versatility
If conventional methods use visual tokenizers for image processing, then multi-modal models can be trained, but the accuracy becomes dependent on the tokenizer performance
Solution Approach 1:
The patent removes the visual tokenizer from the pipeline, eliminating the source of accuracy dependency. By directly processing pixel blocks without relying on tokenizer-generated representations, the training accuracy becomes independent of tokenizer performance, thereby improving reliability
Solution Approach 2:
Instead of converting images to tokens and then processing (conventional approach), the patent inverts the process by directly processing pixel blocks without tokenization. This inversion eliminates the accuracy dependency on visual tokenizers while maintaining the ability to train multi-modal models effectively
3Adaptability or versatility
If large-scale unlabeled data is used for training multi-modal models, then the models can be quickly adapted to various tasks, but the data processing becomes more complex and resource-intensive
Solution Approach 1:
The patent segments images into pixel blocks as the basic processing unit, which enables efficient processing of large-scale unlabeled data. This segmentation approach maintains the ability to quickly adapt models to various tasks while improving data processing efficiency by avoiding the computational overhead of visual tokenization
Solution Approach 2:
The patent changes the fundamental parameter of data representation from token-based to pixel block-based. This parameter change enables direct processing of large-scale unlabeled data more efficiently, maintaining rapid task adaptation while reducing the computational resources and complexity required for data processing
Data Source
AI summary
Embodiments of the present disclosure relate to methods, apparatuses, electronic devices, and media for processing multi-modal data. The method comprises dividing a source image into a set of source pixel blocks and generating a sequence of masked source pixel blocks by masking one or more source pixel blocks in the set of source pixel blocks. The method further comprises generating one or more target pixel blocks corresponding to the one or more source pixel blocks based on the sequence of masked source pixel blocks and a source text corresponding to the source image. The method further comprises generating a multi-modal model based on the one or more source pixel blocks and the one or more target pixel blocks. The method can effectively reduce the training cost, reduce the complexity of training the multi-modal model, and improve the stability and accuracy of the model.