Distributed Neural Network Training via Heterogeneous Gradient Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed neural network training faces limitations due to increased communication time and bounded batch size, which affects the accuracy and efficiency of the training process.
Innovation Solution
A neural network processing system utilizing heterogeneous computation units (HCUs) computes and aggregates gradients from multiple samples to generate local gradient updates, allowing for parallelization of computation and communication, thereby improving the efficiency of weight updates and reducing the statistical quality bound.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If large batch size is used to speed up training, then computation time is reduced, but communication time increases and accuracy deteriorates
Solution Approach 1:
The system segments the training computation across multiple HCUs, where each HCU processes a subset of samples independently. This allows parallel computation while enabling more frequent, smaller gradient aggregations rather than waiting for a single large batch, thereby reducing communication time without sacrificing training speed.
Solution Approach 2:
The system dynamically adjusts the timing of weight updates based on computation progress. Instead of synchronous updates after complete batch processing, the system enables asynchronous or dynamic updates where HCUs can update weights at different times, reducing communication overhead while maintaining training effectiveness.
2Productivity
If batch size is increased beyond a limit, then computation time is reduced, but final accuracy of the converged neural network suffers
Solution Approach 1:
The system divides the large batch into smaller sub-batches processed by individual HCUs. Each HCU computes gradients for its local samples, then results are aggregated. This segmentation allows the system to benefit from large effective batch size for efficiency while maintaining smaller local batch sizes that preserve accuracy through more frequent updates.
Solution Approach 2:
The system changes the parameter of update frequency and aggregation timing. By enabling more frequent gradient aggregations and weight updates across multiple HCUs, the system effectively reduces the statistical quality bound and maintains accuracy while achieving the computational efficiency of large batch processing.
3Productivity
If training is distributed to a large number of machines, then computation time is reduced, but communication time increases
Solution Approach 1:
The system segments computation across multiple HCUs with local samples, enabling independent processing. This reduces the need for frequent full-system synchronization and communication, as each HCU can process and aggregate locally before communicating results, thereby reducing overall communication time while maintaining high computation speed.
Solution Approach 2:
The system enables continuous computation and partial aggregation without waiting for complete communication cycles. HCUs can continuously process samples and perform local aggregations while communication is ongoing, maintaining useful action continuity and reducing idle communication time in the distributed system.
Data Source
AI summary
The present disclosure discloses a system and method for distributed neural network training. The method includes: computing, by a plurality of heterogeneous computation units (HCUs) in a neural network processing system, a first plurality of gradients from a first plurality of samples; aggregating the first plurality of gradients to generate an aggregated gradient; computing, by the plurality of HCUs, a second plurality of gradients from a second plurality of samples; aggregating, at each of the plurality of HCUs, the aggregated gradient with a corresponding gradient of the second plurality of gradients to generate a local gradient update; and updating, at each of the plurality of HCUs, a local copy of a neural network with the local gradient update.