Bi-level Finetuning for Low-Resource LLM Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training large language models in low-resource settings is challenging due to susceptibility to overfitting and limited generalization abilities, with existing methods either restricting model expressiveness or relying on external knowledge sources.
Innovation Solution
The proposed solution involves constructing a task-dependent similarity structure using bi-level optimization, where model weights and similarity matrix parameters are learned separately to enhance model performance without external knowledge, and dimensionality reduction techniques are applied to optimize computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If parameter restriction is applied during finetuning to avoid overfitting, then overfitting is reduced, but model expressiveness deteriorates
Solution Approach 1:
The model parameters are segmented into two groups: restricted parameters (embedding layers) and trainable parameters (classification head). This segmentation allows the model to avoid overfitting by limiting training to specific parameter subsets while preserving the expressive power of the pretrained embedding layers that are not updated during finetuning.
Solution Approach 2:
Different quality constraints are applied to different parts of the model. The embedding layers maintain their pretrained high-quality representations with frozen parameters, while only the classification head undergoes local adaptation through training. This local quality approach ensures that the most critical representational knowledge is preserved while allowing task-specific adaptation where needed.
2Quantity of substance
If data augmentation is performed using external lexical resources or pretrained language models, then training data size increases, but task-dependency deteriorates
Solution Approach 1:
The method uses the model's own pretrained embedding layers to generate augmented training data through operations like synonym replacement and paraphrase generation. Instead of relying on external lexical resources or separate pretrained language models, the system performs data augmentation self-service using its internal knowledge representations, ensuring task-independence while maintaining data quantity.
Solution Approach 2:
Data augmentation is achieved by changing parameters in the embedding space rather than relying on external resources. The system modifies word representations through operations in the vector space (e.g., adding noise, interpolating between embeddings) to generate varied training examples that maintain semantic relationships while increasing data diversity without external dependencies.
Data Source
AI summary
There is included a method and apparatus comprising computer code configured to cause a processor or processors to constructing similarity scores between words, initializing a similarity structure which is task-dependent and based on the similarity scores, and machine learning a task-dependency of the similarity structure by implementing bi-level optimization including a search phase comprising learning model weights by estimating a parameter of a model, respective to a first entry of the similarity structure, and learning a parameter of a second entry of the similarity structure by using the parameter on the second entry, and a fine-tuning phase comprising updating the parameter of the model while holding the similarity structure fixed.