Distributed Processing Nodes for Neural Network Weight Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed deep learning systems, the increased number of nodes leads to higher integration and distribution communication times, resulting in decreased processing performance due to increased data processing loads on aggregation and distributed processing nodes, which hampers the acceleration of deep learning processes.
Innovation Solution
The system employs an aggregation processing node and distributed processing nodes that packetize and transmit data in order of weight numbers, allowing for simultaneous transmission and reception of data, with each node monitoring unreceived data amounts to control transmission and prevent transient data concentration, thereby optimizing communication and reducing processing loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of distributed processing nodes is increased to accelerate gradient calculation processing, then the processing capacity increases, but the integration communication time and distribution communication time increase, resulting in decreased processing performance
Solution Approach 1:
The patent segments the gradient calculation processing into multiple independent stages that can be executed in parallel across distributed nodes. Each node processes a portion of the sample data independently, and the segmented results are aggregated through optimized communication protocols, thereby maintaining processing capacity while reducing communication overhead time.
Solution Approach 2:
The patent implements preliminary actions by pre-processing sample data into manageable chunks before distribution, and pre-establishing communication channels between nodes. This preparation work is done in advance to minimize the actual communication time during the main processing phase, allowing faster aggregation and distribution operations.
2Measurement precision
If the number of distributed processing nodes is increased to improve inference accuracy, then more sample data can be processed, but the data processing loads on aggregation and distributed processing nodes increase, hampering acceleration
Solution Approach 1:
The patent divides the processing workload into segmented tasks distributed across multiple nodes, with each node handling a specific portion of the computation. This segmentation reduces the processing load on individual nodes while maintaining the ability to process large volumes of sample data collectively, thereby improving inference accuracy without overwhelming single nodes.
Solution Approach 2:
The patent transitions from a centralized processing model to a multi-dimensional distributed architecture where processing occurs across spatial dimensions (multiple nodes) and temporal dimensions (parallel execution stages). This dimensional expansion allows the system to handle increased data volumes and complexity without proportionally increasing the load on any single node.
Data Source
AI summary
Individual distributed processing nodes packetize distributed data for each weight of a neural network of a learning object in an order of a number of the weight, transmit the distributed data to an aggregation processing node, acquire aggregation data transmitted from the node in order, and update the weight of the neural network. The node acquires the transmitted distributed data, packetizes the aggregation data for which the distributed data of all the distributed processing nodes is aggregated for each weight, and transmits the aggregation data to the individual nodes. The individual nodes monitor an unreceived data amount which is a difference between data amounts of the transmitted distributed data and the acquired aggregation data, and when the unreceived data amount becomes equal to or larger than a threshold Ma, stops transmission of the distributed data until the unreceived data amount becomes equal to or smaller than a threshold Mb (Mb<Ma).