Multi-Task Neural Network Training with Normalized Task Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training multi-task neural networks face challenges in balancing task weights effectively, leading to issues such as different loss scalings across tasks, resource imbalances, and computational inefficiencies, particularly in scenarios with highly imbalanced losses.
Innovation Solution
The method involves computing task weights by scaling individual losses to a constant using a normalization function, such as softmax, and applying a stop gradient operator to stabilize the weights, thereby normalizing them to ensure balanced learning across tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If Uncertainty Weighting is used to dynamically adjust task weights, then task-specific optimal weights are achieved, but update inertia occurs and overfitting is prone
Solution Approach 1:
The patent applies dynamics by making task weights adaptive rather than fixed. The uncertainty weights are dynamically adjusted during training based on the observed loss values, allowing the system to respond to changing task difficulties and data distributions. This is implemented through the uncertainty weighting mechanism that computes weights proportional to the inverse of loss variance, enabling automatic adaptation without manual intervention.
Solution Approach 2:
The patent changes the parameter of task weights from fixed or manually set values to dynamically computed uncertainty-based weights. By transforming the weight computation from static assignment to dynamic calculation based on loss statistics, the system achieves better weight accuracy while the normalization and smoothing techniques prevent instability and overfitting.
2Measurement precision
If Scalarization with grid search is used to find optimal fixed weights, then superior performance is achieved, but computational cost becomes prohibitive
Solution Approach 1:
The patent applies self-service by enabling the system to automatically determine optimal task weights without external intervention or exhaustive search. The uncertainty weighting mechanism autonomously computes appropriate weights based on the observed loss patterns during training, eliminating the need for manual grid search while achieving comparable or superior performance.
Solution Approach 2:
The patent replaces the mechanical grid search approach with a gradient-based uncertainty weighting mechanism. Instead of exhaustively searching through weight combinations, the system uses analytical solutions based on loss statistics to directly compute optimal weights, dramatically reducing computational time while maintaining accuracy.
3Ease of manufacture
If Equal Weighting is used for all tasks, then implementation is simple, but drastically different loss scalings across tasks occur
Solution Approach 1:
The patent changes the task weight parameter from uniform equal values to uncertainty-based computed values. This transformation maintains the simplicity of automatic weight assignment while correcting the loss scaling imbalances by adjusting weights according to observed task performance and loss characteristics.
Solution Approach 2:
The patent introduces dynamics by making weights adaptive rather than static. The uncertainty weighting mechanism continuously adjusts task weights based on observed loss values during training, allowing the system to automatically balance loss scalings across tasks with different magnitudes and difficulties.
Data Source
Figure 1a~2
Figure 3a~3b
Figure 4a
AI summary
Some embodiments are directed to training a multi-task neural network. The multi-task neural network is configured to receive an input and to produce multiple outputs. During training task weights are computed to weight individual losses. A normalization function is applied to normalize the task weights.