Distributed Deep Learning Gradient Aggregation via Hardware Interconnect
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed deep learning systems face inefficiencies in communication processing as the number of nodes increases, leading to increased overhead and reduced learning efficiency due to software-based communication and data transmission processes.
Innovation Solution
A distributed deep learning system that employs a computing interconnect device connected to multiple learning nodes via a communication network, where gradient calculations and updates are processed in parallel using hardware, allowing for simultaneous high-speed transmission and reception of communication packets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If software-based communication processing is used in distributed deep learning systems, then system implementation is simpler, but communication overhead increases and learning efficiency decreases as the number of nodes increases
Solution Approach 1:
The patent replaces software-based communication processing with a dedicated hardware computing interconnect device. This hardware device performs communication operations (data transmission, reception, and processing) at the hardware level, eliminating the need for software-based processing and thereby reducing communication overhead while maintaining system implementation simplicity.
2Speed
If the number of learning nodes is increased to process large amounts of data in parallel, then learning speed increases, but communication overhead and processing time increase
Solution Approach 1:
The patent introduces a dedicated hardware computing interconnect device as an intermediary between learning nodes. This device专门 handles communication processing tasks, enabling efficient data exchange between multiple learning nodes without increasing communication overhead. The hardware intermediary processes communication operations at a lower level, reducing processing time even as the number of nodes increases.
3Productivity
If parallel processing is performed with multiple learning nodes, then calculation resources are utilized more effectively, but communication processing becomes a bottleneck
Solution Approach 1:
The patent segments the system into learning nodes for calculation processing and a separate computing interconnect device for communication processing. This segmentation allows calculation operations to be performed in parallel across multiple learning nodes while communication processing is handled independently by the dedicated hardware device, preventing communication from becoming a bottleneck.
Data Source
AI summary
Each of learning nodes calculates gradients of a loss function from an output result obtained by inputting learning data to a learning target neural network, converts a calculation result into a packet, and transmits the packet to a computing interconnect device. The computing interconnect device receives the packet transmitted from each of the learning nodes, acquires a value of the gradients stored in the packet, calculates a sum of the gradients, converts a calculation result into a packet, and transmits the packet to each of the learning nodes. Each of the learning nodes receives the packet transmitted from the computing interconnect device and updates a constituent parameter of a neural network based on a value stored in the packet.