Compressed Attention Neural Networks for Lower Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks, particularly attention-based ones, require significant memory bandwidth and computational resources for data transfer and operations, leading to high latency and power consumption when implemented on hardware logic.
Innovation Solution
Implement a compressed attention-based neural network by rearranging and partitioning embedded tensors into sub-matrices, applying weight matrices to these sub-matrices to determine Key, Query, and Value matrices, and using these matrices to perform attention functions efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If attention-based neural networks are implemented on hardware logic, then computational capability is improved, but memory bandwidth requirements and power consumption increase
Solution Approach 1:
The patent segments the attention mechanism computation into distinct stages: computing attention scores, applying softmax normalization, and computing weighted sums. This segmentation allows for optimized hardware implementation where each stage can be processed efficiently, reducing overall power consumption while maintaining computational capability.
Solution Approach 2:
The patent performs preliminary computation of attention scores and normalization factors before the final weighted sum calculation. By pre-computing these values and storing them in intermediate buffers, the system reduces the computational complexity of the final aggregation step, thereby lowering power consumption during inference.
2Productivity
If attention-based neural networks are implemented on hardware logic, then computational capability is improved, but latency increases
Solution Approach 1:
The attention mechanism is divided into parallelizable segments that can be processed simultaneously. The patent computes attention scores for different query-key pairs in parallel, then applies softmax normalization to each segment independently, finally computing weighted sums in parallel. This segmented parallel processing reduces overall latency while maintaining high computational capability.
Solution Approach 2:
The patent implements a pipelined architecture where different stages of the attention computation (score computation, normalization, weighted sum) operate in periodic cycles. While one stage is processing data for time step t, another stage processes data for time step t-1, enabling continuous throughput and reduced latency.
3Productivity
If attention-based neural networks are implemented on hardware logic, then computational capability is improved, but memory bandwidth requirements increase
Solution Approach 1:
The patent pre-computes and stores normalization factors (softmax denominators) in intermediate buffers before the weighted sum computation. This preliminary action eliminates the need to re-access the original key and value matrices during the aggregation phase, significantly reducing memory bandwidth requirements while maintaining computational accuracy.
Solution Approach 2:
The patent introduces intermediate buffers that store computed attention scores and normalization factors. These intermediaries act as mediators between the input matrices and the final output, allowing the system to perform multiple computational passes using the same input data without requiring repeated high-bandwidth memory access, thus reducing overall memory bandwidth requirements.
Data Source
AI summary
A compressed attention-based neural network comprises a compressed attention layer implementing an attention function. The compressed attention layer rearranges and partitions an embedded tensor to form embedded sub-matrices. The compressed attention layer applies Key weight sub-matrices to the embedded sub-matrices, and concatenates the results to the respective embedded sub-matrices to determine a Key matrix. The compressed attention layer applies Query weight sub-matrices to the embedded sub-matrices and concatenates the results to determine a Query matrix. The compressed attention layer applies a set of one or more Value weight sub-matrices to the respective one or more embedded sub-matrices, and concatenates the results of applying the one or more Value weight sub-matrices to the respective one or more embedded sub-matrices, to determine a Value matrix. The compressed attention layer implements the attention function using the determined Key matrix, the determined Query matrix and the determined Value matrix.