Compressed Attention Networks for Lower Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks, particularly attention-based neural networks, require significant memory bandwidth and computational resources due to the large amount of data needed to implement their operations, leading to high latency and power consumption.
Innovation Solution
A method and hardware logic for implementing a compressed attention-based neural network by rearranging and partitioning elements of an embedded tensor into sub-matrices, applying weight matrices to these sub-matrices, and concatenating the results to determine Key, Query, and Value matrices, thereby reducing the data required for computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If attention-based neural networks are implemented with full Key, Query, and Value matrices, then accurate attention function computation is achieved, but memory bandwidth consumption and computational complexity increase significantly
Solution Approach 1:
The patent segments the computation process by partitioning the embedded tensor into multiple sub-matrices and applying weight matrices in a staged manner. Instead of computing all Key, Query, and Value matrices simultaneously, the system divides the computation into smaller steps, processing subsets of the data sequentially. This reduces the peak memory bandwidth requirements and computational complexity while maintaining the accuracy of the attention function through proper concatenation of intermediate results.
2Measurement precision
If large weight matrices are used for attention computation, then model accuracy is maintained, but memory bandwidth requirements and power consumption increase
Solution Approach 1:
The computation is segmented into multiple smaller operations that process portions of the embedded tensor separately. By partitioning the weight matrix application into staged computations on sub-matrices, the system reduces the amount of data that needs to be moved through memory at any given time, thereby reducing power consumption associated with memory bandwidth while preserving model accuracy through proper reconstruction of the full computation.
Solution Approach 2:
The system performs preliminary partitioning of the embedded tensor into sub-matrices before the main weight matrix application. This preliminary organization allows for more efficient computation by pre-positioning data in a format that minimizes subsequent memory access requirements, reducing power consumption while maintaining computational accuracy.
3Measurement precision
If complete Key, Query, and Value matrices are computed, then accurate attention weights are obtained, but latency increases due to large data movement
Solution Approach 1:
The patent applies segmentation by dividing the embedded tensor into sub-matrices and computing Key, Query, and Value matrices in staged operations. This reduces the amount of data that must be moved through memory at any one time, thereby reducing computation latency. The accuracy of attention weights is preserved by properly concatenating the intermediate results from each computational stage to reconstruct the full attention matrices.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A method of, and hardware logic for, implementing a compressed attention-based neural network are described herein. The compressed attention-based neural network comprises a compressed attention layer arranged to implement an attention function. An embedded tensor is received at the compressed attention layer. The compressed attention layer rearranges and partitions elements of the embedded tensor to form one or more embedded sub-matrices. The compressed attention layer applies a set of one or more Key weight sub-matrices to the respective one or more embedded sub-matrices, and concatenates the results of applying the one or more Key weight sub-matrices to the respective one or more embedded sub-matrices, to determine a Key matrix. The compressed attention layer applies a set of one or more Query weight sub-matrices to the respective one or more embedded sub-matrices, and concatenates the results of applying the one or more Query weight sub-matrices to the respective one or more embedded sub-matrices, to determine a Query matrix. The compressed attention layer applies a set of one or more Value weight sub-matrices to the respective one or more embedded sub-matrices, and concatenates the results of applying the one or more Value weight sub-matrices to the respective one or more embedded sub-matrices, to determine a Value matrix. The compressed attention layer implements the attention function using the determined Key matrix, the determined Query matrix and the determined Value matrix.