Transformer Attention Weight Merging for Lower Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high computational burden and memory access requirements of transformer-based neural networks pose challenges for real-time applications on resource-constrained devices, particularly in natural language processing tasks.
Innovation Solution
Implementing weight merging techniques, including QK merging, VO merging, and low-rank decompositions, to pre-compute and store merged weights in memory, reducing the number of operations and parameters, and optimizing computations across attention and MLP layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard transformer attention mechanism is used, then attention score computation accuracy is maintained, but computational burden and memory access amount increase
Solution Approach 1:
The patent merges the query weight matrix and key weight matrix into a single first merged weight matrix, and merges the value weight matrix and output score weight matrix into a single second merged weight matrix. This merging reduces the number of separate weight matrices from four to two, thereby reducing computational operations and memory access requirements while maintaining the functional equivalence of the attention mechanism
Solution Approach 2:
The patent pre-computes the first merged weight matrix (combining query and key weights) and the second merged weight matrix (combining value and output score weights) before the forward propagation process. By performing these computations in advance, the patent reduces the computational burden during real-time inference, as the merged weights are stored and directly applied without requiring separate weight matrix multiplications during the attention score computation
2Measurement precision
If standard transformer attention mechanism is used, then attention score computation accuracy is maintained, but memory access amount increases
Solution Approach 1:
The patent combines multiple weight matrices into fewer merged weight matrices, reducing the total number of parameters that need to be stored in memory. The first merged weight matrix replaces separate query and key weight matrices, and the second merged weight matrix replaces separate value and output score weight matrices, thereby reducing memory access requirements during forward propagation
Solution Approach 2:
The merged weight matrices are pre-computed and stored in memory before the forward propagation process. This preliminary computation allows the system to store compact merged weight representations instead of multiple separate weight matrices, reducing the memory access amount during real-time inference while maintaining computational accuracy
3Productivity
If weight merging and low-rank decomposition are applied, then computational load is reduced, but model parameter complexity increases
Solution Approach 1:
The patent merges weight matrices to reduce the number of separate parameters, and applies low-rank decomposition to further compress the merged weight matrices. The low-rank decomposition represents each merged weight matrix as a product of two smaller matrices with lower dimensions, significantly reducing the number of parameters while maintaining the computational functionality and accuracy of the attention mechanism
Data Source
AI summary
A method and an electronic apparatus for computation on an attention layer of a transformer-based neural network are proposed. The method includes to receive a query input, a key input, and a value input from a previous layer of the attention layer, to obtain a first merged weight which is pre-computed based on a weight matrix for queries and a weight matrix for keys, to obtain a second merged weight which is pre-computed based on a weight matrix for values and a weight matrix for output scores, and to perform computation based on the query input, the key input, the value input, the first merged weight, and the second merged weight to generate an attention score of the attention layer.