Shared Transformer Backbones for Compact Vision-Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformers used for processing data with different modalities, such as language and images, have a large number of parameters that make them impractical for deployment on resource-limited real-world platforms like mobile devices.
Innovation Solution
A method of training a model by configuring a first transformer for visual learning and a second transformer for textual learning, adjusting the weights to minimize differences between them, replacing the first transformer's weights with the adjusted second transformer's weights, and fine-tuning the first transformer based on these adjusted weights, allowing for partial or full weight sharing across two transformer backbones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If transformers are used for processing data with different modalities, then processing capability is improved, but model size and parameter count increase making deployment on mobile devices impractical
Solution Approach 1:
The patent merges the weight parameters of the vision transformer and text transformer by minimizing the weight difference between them during training. This allows one transformer to share weights with the other, effectively reducing the total number of parameters while maintaining both visual and textual processing capabilities.
Solution Approach 2:
The patent creates a universal transformer backbone that can process both visual and textual data by enforcing weight similarity between vision and text transformers. This single backbone serves multiple modalities, reducing the need for separate large models for each modality.
2Reliability
If separate transformers are configured for visual and textual learning, then task-specific performance is improved, but device complexity increases
Solution Approach 1:
The patent combines the architecture of vision and text transformers into a unified structure with shared weights. By minimizing the weight difference between the two transformers during training, the system maintains task-specific performance while reducing architectural complexity to a single shared backbone.
3Quantity of substance
If full weight sharing is implemented between vision and text transformers, then model size is reduced, but training difficulty increases
Solution Approach 1:
The patent implements partial weight sharing by minimizing the weight difference between vision and text transformers rather than forcing complete weight equality. This gradual approach with a learning rate schedule makes the training process more manageable while still achieving significant parameter reduction.
Solution Approach 2:
The patent changes the weight parameters of the transformers by minimizing the weight difference between vision and text transformers during training. This parameter adjustment process, combined with a reduced learning rate in later stages, enables successful weight sharing while managing training complexity.
Data Source
AI summary
A method of training a model includes configuring a first transformer for visual learning with a first set of weights, configuring a second transformer for textual learning with a second set of weights, adjusting at least the second set of weights based on minimizing a weight difference between the first set of weights and the second set of weights, replacing the first set of weights for the first transformer with the adjusted second set of weights, and updating the first transformer based on the adjusted second set of weights.