Packet Aggregation by Message Index for Distributed DNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed deep neural network (DNN) training methods face communication bottlenecks due to excessive data transmission, leading to network bandwidth limitations and communication delays, especially in scenarios with a large number of workers.
Innovation Solution
A communication method that aggregates packets based on shared message indices and sequence numbers to reduce data redundancy and delay, ensuring accurate identification and efficient forwarding of packets, thereby improving DNN model training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If distributed DNN training methods are used with multiple workers exchanging parameters, then training capability is improved, but communication data amount increases significantly causing network bandwidth limitations
Solution Approach 1:
The patent merges multiple packets carrying parameter data of the same DNN model layer into a single aggregated packet. Instead of transmitting separate packets from each worker, the system combines them into one packet that contains all necessary parameter updates, significantly reducing the total data transmission volume while maintaining the distributed training capability
Solution Approach 2:
The patent creates an aggregated packet that serves as a consolidated copy of multiple parameter data packets. Rather than transmitting original separate packets, the system generates a single aggregated representation that contains the essential information from all worker packets, reducing network traffic while preserving training functionality
2Productivity
If ring-all-reduce processing is performed with many workers, then distributed training is enabled, but each worker must transmit nearly twice the DNN model parameter data amount
Solution Approach 1:
The patent applies merging by combining multiple parameter data packets into a single aggregated packet at the network device. This eliminates the need for each worker to transmit and receive multiple separate packets in ring-all-reduce operations, reducing communication rounds and overall transmission time while maintaining distributed training capability
3Quantity of substance
If packet aggregation is performed based on message indices, then data redundancy is reduced, but packet identification and routing complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-assigning message indices to packets before transmission. Each packet is tagged with a message index that identifies its destination and content type in advance. This allows the network device to perform aggregation efficiently using simple index matching without complex real-time analysis, reducing both redundancy and identification complexity
Data Source
AI summary
A communication method and apparatus relating to the field of communication technologies. The method includes a network device receives at least two packets, and aggregates the at least two packets to obtain aggregated packets. Messages to which the at least two packets separately belong have a same message index. The message index is used to index the messages to which the at least two packets separately belong. In the communication method provided in this application, the network device is used to implement aggregation processing, to reduce a communication delay and data redundancy in a communication network.