Attention-Based Neural Network Compression Through Dense Weight Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks, particularly attention-based neural networks, require significant memory bandwidth and computational resources due to the large amount of data needed to implement their operations, leading to high latency and power consumption.
Innovation Solution
A method for compressing attention-based neural networks by rearranging and partitioning weight matrices into sub-matrices with higher density of non-zero values, reducing the amount of data required for implementation and optimizing operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If attention-based neural networks are implemented with full precision weight matrices, then model accuracy is maintained, but memory bandwidth requirements and computational resources increase significantly
Solution Approach 1:
The patent segments the attention mechanism by separating the computation of attention weights from the value retrieval process. The attention layer is divided into distinct components: query-key attention weight computation and value matrix application. This segmentation allows for optimized memory access patterns where the value matrix can be compressed or stored in a more efficient format since it is only read after attention weights are computed, reducing overall memory bandwidth requirements while maintaining computational accuracy
Solution Approach 2:
The patent applies preliminary action by pre-computing and caching the value matrix transformations. The value matrix is transformed and stored in an optimized format before the forward pass, allowing the attention mechanism to retrieve pre-processed data rather than performing full matrix multiplications during inference. This preliminary preparation reduces the computational burden and memory bandwidth requirements during actual network execution while preserving model accuracy
2Measurement precision
If attention-based neural networks process all input data through full attention mechanisms, then comprehensive feature relationships are captured, but operational latency increases
Solution Approach 1:
The patent extracts and isolates the computationally intensive attention weight computation from the value retrieval process. By separating these operations and allowing independent optimization, the system can compute attention weights only when necessary and retrieve values more efficiently. This extraction enables the use of approximations or optimizations in the value retrieval stage without compromising the quality of attention-weighted feature relationships, thereby reducing latency while maintaining comprehensive feature analysis
Solution Approach 2:
The patent implements partial action by allowing the attention mechanism to operate in different modes: full precision attention weight computation when accuracy is critical, and optimized/approximated value retrieval when speed is prioritized. The system can selectively apply full attention computation only to critical portions of the input data or use simplified retrieval methods for less critical operations, achieving a balance between comprehensive feature relationship capture and operational latency reduction
3Reliability
If attention-based neural networks use standard matrix operations, then computational correctness is ensured, but power consumption increases
Solution Approach 1:
The patent segments the computational pipeline into distinct stages: attention weight computation and value application. This segmentation allows for targeted optimization where power-intensive operations are identified and optimized separately. The value matrix operations can be performed using lower-precision arithmetic or specialized hardware units since they occur after the critical attention weight computation, reducing overall power consumption while maintaining computational correctness through the structured separation of concerns
Solution Approach 2:
The patent applies parameter changes by allowing different precision levels for different components of the attention mechanism. The attention weights can be computed with full precision to ensure correctness, while the value matrix operations can use reduced precision or quantized representations. This parameter variation across different computational stages maintains the reliability of the attention mechanism's decision-making while significantly reducing the power consumption of the overall operation
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A computer implemented method of, and a processing system for, compressing an attention-based neural network are described herein. An attention-based neural network comprises an attention layer. The attention layer is arranged to implement an attention function in dependence on a Key matrix, a Query matrix and a Value matrix. The attention layer is configured to use a Key weight matrix to determine the Key matrix, use a Query weight matrix to determine the Query matrix, and use a Value weight matrix to determine the Value matrix. A combined matrix is determined which is the same size and shape as the Key weight matrix, the Query weight matrix and the Value weight matrix. For each element of the combined matrix: (i) that element of the combined matrix is determined to be zero if corresponding elements of the Key weight matrix, the Query weight matrix and the Value weight matrix are zero, and (ii) that element of the combined matrix is determined to be non-zero if at least one of the corresponding elements of the Key weight matrix, the Query weight matrix and the Value weight matrix is non-zero. A rearrangement of the rows and columns of the combined matrix is determined that would gather the plurality of non-zero values of the combined matrix into a set of one or more combined sub-matrices, the set of one or more combined sub-matrices having a greater average density of non-zero values than the combined matrix. The rows and columns of the Key weight matrix, the Query weight matrix and the Value weight matrix are rearranged to match the determined rearrangement of the rows and columns of the combined matrix so as to determine a set of one or more Key weight sub-matrices, a set of one or more Query weight sub-matrices and a set of one or more Value weight sub-matrices corresponding to the set of one or more combined sub-matrices. A compressed attention-based neural network is outputted which comprises a compressed attention layer arranged to implement the attention function by performing a compressed operation in dependence on: (i) the set of one or more Key weight sub-matrices, (ii) the set of one or more Query weight sub-matrices, and (iii) the set of one or more Value weight sub-matrices.