Network Device Reducing Data Exchange in Parallel Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning model training processes require frequent data exchanges among distributed computing resources, leading to network resource occupation and delayed training due to the large amount of data exchange, especially when using GPUs for parallel computing.
Innovation Solution
A method involving a network device that receives processing results from computing devices, executes a reduction operation (such as AllReduce), and transmits the result back to the computing devices, reducing the amount of data exchanged and network latency by using pre-processing and post-processing techniques to adapt results for floating-point calculations and broadcasting the reduction result.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallel computing with distributed computing resources is used to accelerate training, then training speed is improved, but network resource occupation increases and training is delayed due to frequent data exchange
Solution Approach 1:
The network device performs pre-processing on processing results before transmission. Specifically, it converts floating-point numbers to integers and performs packing operations in advance, so that computing devices receive pre-processed data that requires less computation, thereby reducing overall training time despite the added pre-processing step
Solution Approach 2:
The network device acts as an intermediary between computing devices. It receives processing results from multiple computing devices, performs reduction operations (summing, averaging, etc.), pre-processing (floating-point to integer conversion, packing), and transmits the processed results back to computing devices, thereby reducing the communication and computation burden on individual computing devices
2Power
If distributed computing resources are used for parallel processing, then computing capacity is improved, but the amount of data exchange increases
Solution Approach 1:
The network device merges processing results from multiple computing devices by performing reduction operations (such as summing or averaging). This consolidation reduces the total volume of data that needs to be transmitted and processed, as instead of each computing device handling full-sized datasets, they work with aggregated results
Solution Approach 2:
The network device changes the parameter representation of data by converting floating-point numbers to integers and performing packing operations. This parameter transformation reduces the bit-width and volume of data being transmitted across the network, directly reducing data exchange volume while preserving the essential information needed for training
Data Source
AI summary
Embodiments of the present disclosure provide methods, devices, and computer program products for processing a task. A method of processing a task comprises: receiving, at a network device and from a set of computing devices, a set of processing results derived from processing the task by the set of computing devices; in response to receiving the set of processing results, executing a reduction operation on the set of processing results; and transmitting a result of the reduction operation to the set of computing devices. In this way, embodiments of the present disclosure can significantly reduce an amount of data exchanged among a plurality of devices processing a task in parallel, and thus reduce network latency caused by data exchange.