Residual Parameter Compression for Distributed Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed deep-learning environments, the large-scale deep-learning models require efficient methods for compressing and transmitting learning parameters across multiple computers, as existing methods face challenges with network latency and storage due to the sheer volume of parameters, especially with an increase in the number of distributed computers.
Innovation Solution
The method involves generating and transmitting information for updating neural network parameters based on the residual difference between the trained and pre-training parameter states, using predictive coding for parameter compression, and reconstructing parameters using received coded residuals and previous values, which reduces data transmission and storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of distributed computers is increased to process large-scale deep-learning models, then the processing capability is improved, but the network latency and storage requirements increase due to the large volume of parameters that need to be exchanged
Solution Approach 1:
The patent extracts and transmits only the residual differences between parameter updates rather than complete parameter sets. Each distributed computer calculates the difference (residual) between its local parameters and the global parameters, then only transmits this residual information. This extraction of essential update information reduces transmission volume and network latency while maintaining training accuracy.
Solution Approach 2:
The parameter update process is segmented into residual calculation and reconstruction phases. The parameter transmission is divided into transmitting only the residual differences rather than complete parameter sets. This segmentation allows distributed computers to process and exchange smaller, more manageable data portions, reducing overall network latency.
2Productivity
If the number of distributed computers is increased to process large-scale deep-learning models, then the processing capability is improved, but the storage requirements increase due to the large volume of parameters that need to be exchanged
Solution Approach 1:
The patent extracts only the residual difference information needed for parameter updates rather than storing and transmitting complete parameter sets. By calculating and storing only the residual (difference) between local and global parameters, the storage requirements at each distributed computer are significantly reduced while maintaining the ability to reconstruct accurate parameter updates.
3Measurement precision
If complete parameter information is transmitted between distributed computers, then the accuracy of model convergence is maintained, but the data transmission volume increases causing network latency
Solution Approach 1:
The patent extracts only the essential residual information needed for accurate parameter updates. By transmitting only the difference (residual) between parameter states rather than complete parameter information, the data transmission volume is reduced, thereby decreasing network latency while maintaining model convergence accuracy through precise residual-based updates.
4Measurement precision
If complete parameter information is transmitted between distributed computers, then the accuracy of model convergence is maintained, but the data transmission volume increases
Solution Approach 1:
The patent extracts and transmits only the residual difference information rather than complete parameter sets. This extraction approach maintains model convergence accuracy by preserving the essential update information while significantly reducing the data transmission volume exchanged between distributed computers.
Data Source
AI summary
Disclosed herein are a method and apparatus for compressing learning parameters for training of a deep-learning model and transmitting the compressed parameters in a distributed processing environment. Multiple electronic devices in the distributed processing system perform training of a neural network. By performing training, parameters are updated. The electronic device may share the updated parameter thereof with additional electronic devices. In order to efficiently share the parameter, the residual of the parameter is provided to the additional electronic devices. When the residual of the parameter is provided, the additional electronic devices update the parameter using the residual of the parameter.