Distributed Gradient Communication Priority for Faster Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deep learning distributed training, the transmission of gradient values for parameters can be affected by sharing communication links with other tasks, leading to reduced training speed and prolonged training periods.
Innovation Solution
A method and apparatus for updating model parameters in a distributed training system that prioritizes and adjusts the communication of gradient values based on a sequence and group sequence of tasks, ensuring preferential execution of lower-ranked tasks during high batch training periods to minimize interference from other tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple training tasks share the same communication link to send gradient values, then the system resource utilization is improved, but the training speed is reduced and training period is prolonged
Solution Approach 1:
The patent changes the parameter of communication priority by introducing a sequence number based on batch training period. Tasks are assigned different priorities dynamically based on their sequence number, transforming the static communication scenario into a dynamic one where parameters (priority levels) change over time to resolve conflicts and improve overall training speed while maintaining resource utilization
Solution Approach 2:
The patent introduces dynamic priority adjustment mechanisms where the communication priority of training tasks is not fixed but changes based on their sequence number and batch training period. This dynamic approach allows the system to adapt to varying workloads and communication conditions, resolving the contradiction between resource utilization and training speed
2Productivity
If gradient values of parameters of a certain training task are transmitted continuously, then the communication efficiency for that task is improved, but the transmission of gradient values of other training tasks is affected
Solution Approach 1:
The patent segments the communication resources by introducing sequence numbers and priority levels for different training tasks. Instead of allowing one task to monopolize the communication link continuously, the system divides communication opportunities into segmented time slots with different priorities, ensuring that multiple tasks can complete their gradient transmissions reliably while maintaining overall communication efficiency
Solution Approach 2:
The patent introduces a communication scheduling mechanism as an intermediary that manages and coordinates gradient value transmissions from multiple training tasks. This intermediary assigns sequence numbers and priorities, mediating between competing tasks to ensure fair and efficient resource allocation, thereby maintaining both communication efficiency and task completion reliability
Data Source
Figure 1
Figure 2
Figure 3~5
AI summary
The disclosure provides a method for updating a parameter of a model, a distributed training system, and relates to a field of deep learning technologies. The method includes: obtaining a batch training period of batch training data to be trained for a model; increasing priorities of tasks ranked at the bottom in a sequence of gradient communication tasks for parameters of the model when the batch training period is greater than or equal to a preset period threshold; and performing a communication of gradients of the parameters and updating the parameters based on priorities of the gradient communication tasks for the parameters in the model.