Gradient Norm Average Weighting for Multi-Task Learning Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-task learning (MTL) techniques face challenges in balancing task weights during training, leading to one-sided dominance where some tasks overwhelm others, especially when using pre-trained models for fine-tuning, due to coarse-grained updating strategies that do not adequately consider gradient magnitudes.
Innovation Solution
The Gradient Norm Average (GNA) weighting scheme adjusts task weights based on both loss change rates and gradient magnitudes within a designated window, using a custom interval and decaying coefficients to fine-tune weights at a more granular level, ensuring even training across tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If coarse-grained updating strategies are used for task weights in multi-task learning, then the training process is simpler and faster, but task weight balance deteriorates leading to one-sided dominance
Solution Approach 1:
The patent implements dynamic task weight adjustment by continuously monitoring gradient magnitudes and loss change rates during training. The weighting scheme transitions from static to dynamic, allowing task weights to adapt automatically based on real-time training dynamics, thereby preventing one-sided dominance while maintaining training efficiency.
Solution Approach 2:
The patent introduces a feedback mechanism where the system monitors gradient magnitudes and loss change rates, then uses this information to adjust task weights. This closed-loop control ensures that task weights remain balanced by continuously responding to training dynamics, resolving the contradiction between simple updating and weight balance.
2Stability of the object's composition
If fine-grained updating strategies are used for task weights considering gradient magnitudes, then task weight balance improves, but computational complexity increases
Solution Approach 1:
The patent segments the weight adjustment process into distinct computational components: gradient magnitude calculation, loss change rate computation, and weight update logic. This modular segmentation allows the system to implement fine-grained control while managing computational complexity through structured organization of operations.
Solution Approach 2:
The patent changes key parameters (gradient magnitudes, loss change rates) to drive weight adjustments. By monitoring and responding to parameter changes in the training process, the system achieves fine-grained weight balance without requiring overly complex computational mechanisms, as it leverages naturally occurring training dynamics.
3Loss of time
If pre-trained models are used for fine-tuning in multi-task learning, then training convergence is faster, but task dominance issues worsen due to unequal gradient magnitudes
Solution Approach 1:
The patent applies counterweight mechanisms by adjusting task weights inversely proportional to gradient magnitudes. Tasks with larger gradient magnitudes (which would otherwise dominate) receive smaller weights, while tasks with smaller gradients receive larger weights, thereby counterbalancing the inherent inequalities in pre-trained model fine-tuning scenarios.
Solution Approach 2:
The patent implements dynamic weight adjustment that adapts to the specific convergence patterns of pre-trained models. As training progresses and tasks converge at different rates, the weighting scheme dynamically adjusts to maintain balance, allowing fast convergence to be preserved while preventing task dominance through continuous adaptation.
Data Source
AI summary
The disclosure provides an apparatus, method, device, and medium for loss balancing in MTL. The apparatus includes interface circuitry and processor circuitry. The processor circuitry is configured to initialize parameters of shared layers of a deep neural network for MTL using a pre-trained neural network; determine a custom interval consisting of a designated number of mini-batch training steps and a designated window of N custom intervals (N>2); for each task, calculate a loss change rate between each pair of N−1 pairs of neighboring custom intervals within a designated window prior to a present custom interval and a gradient magnitude with respect to selected shared weights within the designated window prior to the present custom interval, and adjust, a weight of the task, based on the calculated loss change rate and gradient magnitude with respect to selected shared weights.