Frozen Base Model Adaptation for Lower-Cost Task Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training base models on specific tasks are either too costly due to full fine-tuning or suffer performance degradation when partial layers are frozen, and they struggle to effectively adapt general knowledge to specific tasks using small datasets.
Innovation Solution
A custom model is trained using a pre-trained base model with feature extraction blocks, adaptation blocks, and residual units, where only the task head, residual units, and adaptation blocks are updated through backpropagation, while feature extraction blocks remain frozen, and features are fused and converted to enhance inference performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full fine-tuning is performed on the base model, then training performance is improved, but training cost becomes astronomically expensive
Solution Approach 1:
The model is segmented into two parts: frozen base model layers and trainable adaptation layers. The base model (first through fifth layers) is frozen to avoid retraining expensive general knowledge, while only the adaptation layers (sixth layer and later) are trained for the specific task, resolving the contradiction between performance and cost
Solution Approach 2:
Instead of training the entire base model (excessive action), only the necessary adaptation layers are trained (partial action). This partial fine-tuning approach trains only the portions of the model needed for the specific task while keeping the expensive general knowledge portions frozen
2Use of energy by moving object
If partial layers are frozen to reduce training cost, then training cost is reduced, but inference performance degrades
Solution Approach 1:
The model is divided into frozen base layers and trainable adaptation layers. The frozen base model maintains general knowledge for efficient training, while the adaptation layers learn task-specific features to maintain inference performance, resolving the contradiction between cost reduction and performance maintenance
Solution Approach 2:
The base model is pre-trained on general knowledge before being frozen and used for the specific task. This preliminary action ensures that the frozen base model contains useful general features that can be leveraged for the specific task without requiring retraining, thus maintaining performance while reducing cost
3Reliability
If all base model layers are trained, then task-specific performance is improved, but memory requirements increase
Solution Approach 1:
The model is segmented into frozen base layers and trainable adaptation layers. Only the adaptation layers require memory for training and backpropagation, significantly reducing memory requirements compared to training all base model layers while maintaining task performance through the frozen general knowledge base
4Adaptability or versatility
If adaptation layers are added to frozen base model, then task adaptability is improved, but model complexity increases
Solution Approach 1:
The model is segmented into a frozen base model and trainable adaptation layers. This segmentation allows the base model to maintain its general knowledge while the adaptation layers provide task-specific functionality, achieving task adaptability without requiring complete model retraining or excessive complexity
Solution Approach 2:
The frozen base model serves as a universal foundation that can be adapted to multiple specific tasks through different adaptation layers. This multi-functionality approach allows the same base model to serve various tasks by training different adaptation layers, improving task adaptability without proportionally increasing overall model complexity
Data Source
AI summary
The present disclosure provides a method for training a custom model based on a pre-trained base model. The custom model is configured to include a base model having at least one pre-trained feature extraction block, at least one adaptation block, at least one residual unit, and a task head block, and update at least part of the task head block, the at least one residual unit and the at least one adaptation block through a backpropagation using a task loss, while not updating parameters of the base model. Further, at least one residual feature (generated through the at least one residual unit) and at least one converted feature (generated by converting the at least one intermediate feature through the at least one adaptation block) are fused to thereby allow a learning device to achieve an advanced inference.