Graph Attention Network Compression with Adjacency Submatrix Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph attention networks become very large, requiring extensive computational resources and memory bandwidth due to the large number of calculations involved in processing, especially when implemented in hardware logic.
Innovation Solution
The method involves rearranging the rows and columns of the adjacency matrix mask to form adjacency sub-matrix masks, which are used to compress the graph attention network, reducing the number of calculations and memory requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If graph attention network is implemented with full adjacency matrix mask, then network performance is maintained, but computational expense and memory bandwidth requirements increase significantly
Solution Approach 1:
The adjacency matrix mask is segmented into multiple adjacency sub-matrix masks by rearranging rows and columns. This segmentation groups connected graph nodes together, creating a block-diagonal structure that reduces the number of calculations required while preserving the network's ability to process graph data effectively.
Solution Approach 2:
The patent changes the structural parameters of the adjacency matrix by applying specific row and column rearrangements. This transforms the dense adjacency matrix into a more sparse block-diagonal form, reducing computational complexity from O(N²) to approximately O(kN) where k is the average degree of connected components.
2Reliability
If graph attention network is implemented with full adjacency matrix mask, then network performance is maintained, but memory bandwidth requirements increase
Solution Approach 1:
By segmenting the adjacency matrix into sub-matrices, the patent reduces the total amount of data that needs to be transferred between memory and processing units. The block-diagonal structure allows for more efficient memory access patterns and reduces redundant data transmission.
3Productivity
If graph attention network is compressed using adjacency sub-matrix masks, then computational expense is reduced, but device complexity increases
Solution Approach 1:
The patent applies preliminary row and column rearrangements to the adjacency matrix before processing. This pre-processing step creates the block-diagonal structure once, and then the compressed graph attention network can efficiently process data using the pre-organized sub-matrices, avoiding repeated complex operations.
Data Source
AI summary
A graph attention network including a graph attention network layer arranged to perform an operation in dependence on an adjacency matrix mask having a plurality of elements representative of connected graph nodes is compressed by rearranging the rows and/or columns of the adjacency matrix mask so as to gather the plurality of elements representative of connected graph nodes into one or more adjacency sub-matrix masks, the one or more adjacency sub-matrix masks having a greater number of elements representative of connected graph nodes per total number of elements of the one or more adjacency sub-matrix masks than the number of elements representative of connected graph nodes per total number of elements of the adjacency matrix mask. A compressed graph attention network comprising a compressed graph attention network layer arranged to perform a compressed operation in dependence on the one or more adjacency sub-matrix masks is outputted.