Collective Communication Compression for Sparse Chip Data Exchange
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current collective communication technologies for sparse data in recommendation algorithms, such as those used in deep learning, suffer from inefficiencies due to excessive transmission of zero-value data, leading to reduced communication efficiency.
Innovation Solution
A method and system for efficient collective communication that compresses data before transmission between computing chips, using techniques like row compression, coordinate list compression, and compressed sparse row format to reduce the amount of data transmitted, thereby improving the efficiency of operations like all-reduce and all-gather.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing collective communication technology is used for sparse data, then data can be transmitted between computing chips, but a large amount of invalid zero-value data is transmitted, reducing communication efficiency
Solution Approach 1:
The patent extracts and removes the zero-value elements from sparse data before transmission. By identifying and eliminating redundant zero elements, the system transmits only meaningful non-zero data between computing chips, thereby reducing invalid data transmission and improving communication efficiency.
Solution Approach 2:
The patent changes the representation parameters of sparse data by using compressed formats that encode only non-zero elements and their positions. This parameter transformation converts dense data structures into compact representations, significantly reducing the volume of transmitted data and improving communication efficiency.
2Productivity
If all data including zero values is transmitted, then complete data integrity is maintained, but transmission volume increases and communication efficiency decreases
Solution Approach 1:
The patent extracts only the essential non-zero elements and their position information from the original data, discarding redundant zero-value elements. This extraction process reduces data transmission volume while preserving all necessary information for computation, thereby improving communication efficiency.
Solution Approach 2:
The patent creates a compressed copy of the original data that contains only non-zero elements and their positions. This compressed representation serves as an efficient substitute for transmitting the complete original data, reducing transmission volume while maintaining data integrity and computational accuracy.
Data Source
AI summary
This application discloses a collective communication method and system, and a computer device. In one example, a collective communication system includes at least a first computing chip and a second computing chip. The first computing chip compresses first data and sends compressed first data to the second computing chip through a communication channel. The second computing chip performs an operation based on the compressed first data.