Distributed Model Parameter Aggregation for Faster Node Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In existing model training systems, the transfer of large quantities of model parameters between nodes in a computing node cluster results in low data transmission efficiency and subsequent low model training efficiency due to the occupation of DDR bandwidth and increased data transmission delays.
Innovation Solution
A model training system where parameters are distributed across multiple nodes with different memory types, allowing for combining parameters from separate memories before transmission, reducing the need for DDR bandwidth and minimizing data transmission delays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model parameters are transferred one by one between nodes, then data can be transmitted accurately, but data transmission efficiency is low and model training efficiency is low
Solution Approach 1:
The patent combines multiple parameter transmission operations into a single aggregated transmission. The first node collects parameters from multiple memory locations (first memory and second memory) and transmits them together in one operation, reducing the total number of transmission packets and improving data transmission efficiency.
2Adaptability or versatility
If parameters are stored in separate memories with different protocols, then storage flexibility is improved, but data combination and transmission complexity increases
Solution Approach 1:
The first node acts as an intermediary that receives requests from the second node, retrieves parameters from both the first memory and second memory using different protocols, combines them, and transmits the aggregated data. This mediator approach manages the complexity of multi-protocol access while maintaining storage flexibility.
Data Source
AI summary
In one example model training system, a second node is configured to send a plurality of parameter obtaining requests to a first node. The first node is configured to determine, based on the plurality of parameter obtaining requests, that parameters obtained based on the plurality of parameter obtaining requests are respectively stored in a first memory and a second memory, combine a parameter obtained from the first memory and a parameter obtained from the second memory to obtain a combined parameter, and send the combined parameter to the second node. The second node is further configured to receive the combined parameter, and train a model based on the combined parameter. The parameter obtained from the first memory and the parameter obtained from the second memory are combined and then sent.