Transformer Attention Head Adaptation for New Task Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer-based machine learning models, particularly visual-language models, suffer from image understanding errors when adapting to new domains, leading to performance degradation due to reliance on pre-trained and frozen visual encoders.
Innovation Solution
Adapt transformer-based models by selectively updating only the parameters of relevant attention heads using a loss function and applying low-rank updates, while freezing other parameters, to maintain knowledge from pre-training and improve performance on new tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If the visual encoder is pre-trained and frozen to maintain stability, then the model structure remains stable and pre-trained knowledge is preserved, but the model cannot adapt to new domains and exhibits image understanding errors
Solution Approach 1:
The model parameters are segmented into two distinct groups: frozen pre-trained parameters and trainable adaptation parameters. This segmentation allows the stable pre-trained visual encoder to maintain its knowledge while specific parameter subsets can be adapted to new domains through selective training, resolving the contradiction between stability and adaptability.
Solution Approach 2:
Instead of uniformly freezing or training all parameters, the invention applies different treatment to different parameter locations. Pre-trained parameters remain frozen to preserve stability, while specifically selected parameters (such as those in adapter modules or certain transformer layers) are made trainable to enable domain adaptation, implementing local quality differentiation.
2Adaptability or versatility
If all model parameters are updated to adapt to a new task, then the model achieves better performance on the new task, but knowledge from pre-training is lost
Solution Approach 1:
The invention extracts and isolates only the necessary parameters for task adaptation while leaving the majority of pre-trained parameters frozen. By taking out only the essential adaptable parameters for training, the model achieves task adaptability without losing the bulk of pre-trained knowledge, directly addressing the contradiction between adaptation and knowledge retention.
Solution Approach 2:
The invention selectively changes parameters rather than uniformly updating all parameters. By identifying and modifying only specific parameter subsets that are critical for new task performance while keeping other parameters fixed, the model adapts to new tasks effectively while preserving pre-trained knowledge, resolving the contradiction between adaptability and knowledge retention.
3Reliability
If the entire model is fine-tuned for a new task, then the model achieves optimal performance, but the computational requirements and training time increase significantly
Solution Approach 1:
Instead of performing full model fine-tuning, the invention applies partial action by training only a subset of parameters. This partial training approach achieves sufficient model performance for the new task while dramatically reducing computational requirements and training time, effectively resolving the contradiction between performance and training efficiency.
Solution Approach 2:
The parameter space is segmented into trainable and frozen portions, allowing the training process to focus computational resources only on the necessary subset of parameters. This segmentation enables the model to achieve good performance on new tasks with significantly reduced computational cost and training time compared to full fine-tuning.
Data Source
Figure 1~2
Figure 3~4
AI summary
A computer-implemented method for adapting a machine learning model to a new task is provided. The method comprises steps of: (S10) providing a model that is pre-trained for a base task, wherein the model is a transformer-based machine learning model comprising a plurality of attention heads; (S20) selecting one or more relevant attention heads among the plurality of attention heads, wherein the relevant attention heads are relevant for performing a new task; and (S30) adapting the model to the new task by updating, among the parameters of the model, only parameters of the relevant attention heads.