Multimodal Image Description With Sparse MoE Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multimodal large language models face challenges in scaling up and maintaining training stability, particularly when handling high-resolution inputs, leading to increased training and inference costs.
Innovation Solution
Incorporating Top-K sparsely gated Mixture-of-Experts (MoE) blocks into each sub-model of a multimodal machine learning model, utilizing a three-stage training process with auxiliary losses to stabilize training and balance expert loading, and employing multi-resolution pyramid images to manage high-resolution inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multimodal models process high-resolution inputs, then image description quality is improved, but training and inference costs increase
Solution Approach 1:
The patent applies segmentation by dividing the image processing task into multiple resolution levels (pyramid structure). Instead of processing the entire high-resolution image uniformly, the model segments the input into different resolution layers, processing only critical regions at high resolution while using lower resolutions for broader context, thereby reducing overall computational cost while maintaining description quality.
Solution Approach 2:
The patent implements local quality by applying different processing qualities to different regions of the image. High-resolution processing is applied selectively to important local regions that contribute most to accurate image description, while less critical regions are processed at lower resolutions, optimizing the trade-off between quality and computational cost.
2Measurement precision
If model size is increased to improve performance, then benchmark accuracy is improved, but training stability deteriorates
Solution Approach 1:
The patent applies preliminary action through progressive training stages. Before training the full multimodal model at large scale, the methodology first pre-trains component models separately and uses a gradual training approach, preparing the model architecture and parameters in advance to ensure stability when scaling up to larger model sizes for improved benchmark accuracy.
3Adaptability or versatility
If more experts are activated in MoE blocks, then model capability is improved, but computational cost increases
Solution Approach 1:
The patent applies partial action by activating only a subset (top-K) of the available experts in the Mixture of Experts blocks for each processing task. Instead of activating all experts which would maximize capability but also maximize cost, the methodology selectively activates only the necessary number of experts based on the specific input requirements, achieving adequate model capability while controlling computational cost.
Data Source
AI summary
The present disclosure describes techniques for generating image descriptions using a machine learning model. Mixture of Experts (MoE) blocks are incorporated into a plurality of sub-models of the machine learning model. The first sub-model of the machine learning model comprises at least one first MoE block including a first plurality of experts. A second sub-model of the machine learning model comprises at least one second MoE block including a second plurality of experts. Only a subset of the first plurality of experts is activated to generate visual tokens based on an input image. Only a subset of the second plurality of experts is activated to project the visual tokens into an input space of the third sub-model. A text description of the input image is output by the third sub-model of the machine learning model.