Task-Specific Queues for Multi-Task Model Parameter Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-task learning models face challenges in balancing the training of multiple sub-tasks due to decreased accuracy and performance during distributed training, where updating model parameters across all tasks leads to inefficiencies.
Innovation Solution
A method and apparatus for updating parameters of a multi-task model by sequencing samples into task-specific queues, training shared and target sub-network layers based on data requirements, and updating parameters in a parameter server using generated gradients, thereby ensuring local updates and improving model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed training is used to meet data capacity and performance requirements, then the model can be trained on large-scale data, but the accuracy and performance of the multi-task model decrease due to parameter updates across all tasks
Solution Approach 1:
The patent segments the parameter update process by task type. Instead of updating all parameters for all tasks simultaneously in distributed training, the system divides parameters into task-specific parameters and shared parameters. Only task-specific parameters are updated during distributed training based on task-specific loss gradients, while shared parameters remain unchanged. This segmentation prevents negative interference between different task updates and maintains model accuracy while enabling large-scale distributed training.
2Adaptability or versatility
If model parameters are updated for all tasks during distributed training, then comprehensive model improvement is achieved, but training efficiency decreases and task bias increases
Solution Approach 1:
The patent extracts and isolates task-specific parameters from the overall model parameters. During distributed training, only the task-specific parameters are updated using gradients computed from task-specific loss functions, while shared parameters are excluded from updates. This extraction approach eliminates unnecessary computational overhead from updating unrelated parameters, significantly improving training efficiency while maintaining task-specific adaptability through targeted parameter updates.
3Adaptability or versatility
If all parameters are updated simultaneously across multiple tasks, then the model learns from all tasks, but task interference occurs and performance degrades
Solution Approach 1:
The patent implements local quality by applying different update strategies to different parameter subsets. Task-specific parameters are updated with task-specific loss gradients to maintain task performance stability, while shared parameters are kept unchanged during distributed training to prevent negative interference. This localized update approach ensures that each task's learning process is not corrupted by gradients from unrelated tasks, maintaining both multi-task learning capability and task performance stability.
Data Source
AI summary
The invention discloses a method and an apparatus for updating parameters of a multi-task model. The method includes: obtaining a training sample set, in which the training sample set comprises a plurality of samples and a task to which each sample belongs; putting each sample into a corresponding sample queue sequentially according to the task to which each sample belongs; training a shared network layer in the multi-task model and a target sub-network layer of tasks associated with the sample queue with samples in the sample queue in case that the number of the samples in the sample queue reaches a training data requirement, so as to generate a model parameter update gradient corresponding to the tasks associated with the sample queue; and updating parameters of the shared network layer and the target sub-network layer in a parameter server according to the model parameter update gradient.