GPU Matrix Decomposition for Low-Bandwidth Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale parallel data training in deep learning is bottlenecked by low-speed network transmission efficiency, particularly with low-bandwidth connections like PCIe, which limits GPU calculation speed.
Innovation Solution
A GPU communication method that decomposes matrices into sub-matrices and compressed matrices, performs reduce, compress, allgather, and decompress operations to reduce communication complexity and data transmission, ensuring convergence precision while deleting smaller feature values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If matrix decomposition and compression operations are performed, then communication complexity and data transmission volume are reduced, but calculation complexity and operation time increase
Solution Approach 1:
The patent divides the original large matrix into multiple sub-matrices and further decomposes them into compressed matrices, enabling selective transmission of only essential data components. This segmentation reduces the overall data transmission volume while distributing the decomposition workload across multiple processing stages.
Solution Approach 2:
The patent transforms the matrix data from its original format into a compressed representation by changing the data parameters and structure. This parameter transformation reduces the amount of data that needs to be transmitted over the network while maintaining the essential information needed for gradient computation.
2Productivity
If data compression is applied to reduce transmission, then transmission efficiency improves, but convergence precision may be affected
Solution Approach 1:
The patent extracts and transmits only the most significant components of the gradient matrix through compression, separating essential information from redundant data. This extraction process reduces transmission volume while preserving the critical elements necessary for model convergence.
Solution Approach 2:
The patent creates compressed copies of the original matrix data that retain the essential characteristics needed for accurate gradient computation. These compressed representations serve as sufficient replicas for the purpose of distributed training without requiring transmission of the complete original data.
Data Source
AI summary
Provided is a GPU communication method, including: decomposing a matrix to be transmitted on each GPU into sub-matrices and a compressed matrix, wherein the compressed matrix obtained by decomposing each matrix to be transmitted is the same; causing each GPU to perform a reduce operation for respective sub-matrices, such that each GPU obtains an intermediate matrix; performing an allgather operation on each GPU, such that each GPU respectively sends the intermediate matrix of the GPU itself to all other GPUs; and respectively multiplying, by the compressed matrix, one or more intermediate matrices received by each GPU and the intermediate matrix of the GPU itself, so as to obtain a final matrix. Also provided are a computer device and a readable storage medium. By means of the solution, the complexity of communication is greatly reduced by decomposing the matrix. On the premise of ensuring the convergence precision, a part of smaller feature values may be deleted, thereby further reducing data transmission.