Quadratic Attention With Group Normalization for Recurrent Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) trained with transformers have high training efficiency and scaling performance but incur linearly increasing inference costs, while RNNs offer fixed inference costs but lack training efficiency, necessitating a hybrid model that combines the benefits of both.
Innovation Solution
Modify a pre-trained transformer's architecture to replace dot-product softmax attention with a linear kernel squared dot product attention layer utilizing Group Normalization, converting it into an RNN for efficient inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If transformers are used for LLM training, then training efficiency and scaling performance are improved, but inference cost increases linearly with the number of tokens
Solution Approach 1:
The patent transforms the attention mechanism by changing the normalization parameter from softmax to Group Normalization, and modifies the attention score computation from dot-product to squared dot-product with linear kernels. This parameter change enables the model to achieve RNN-like fixed inference cost while maintaining transformer training efficiency
Solution Approach 2:
The patent segments the attention computation into distinct components: linear projection layers (WQ, WK, WV), squared dot-product scoring, and Group Normalization. This segmentation allows each component to be optimized independently, with the linear projections enabling efficient computation that reduces inference cost from linear to fixed complexity
2Use of energy by moving object
If RNNs are used for sequence modeling, then inference cost is fixed, but training efficiency and scaling performance deteriorate
Solution Approach 1:
The patent merges the architectural benefits of transformers (parallel training efficiency, scaling performance) with the inference advantages of RNNs (fixed cost). By combining transformer-style attention with linear projections and Group Normalization, the model achieves both training efficiency and fixed inference cost in a unified architecture
Solution Approach 2:
The patent introduces dynamic linear projections that adapt the attention computation based on input characteristics. The linear kernels and Group Normalization dynamically adjust the computation path, enabling the model to maintain high training efficiency while achieving fixed inference cost characteristics
3Device complexity
If dot-product softmax attention is used, then attention computation is simple, but memory and computation costs increase
Solution Approach 1:
The patent substitutes the traditional softmax attention mechanism with a linear kernel-based squared dot-product attention. This replacement eliminates the computationally expensive softmax operation while maintaining attention functionality through linear projections and Group Normalization, reducing computation cost from linear to fixed complexity
Data Source
AI summary
A method may include receiving parameters associated with a pre-trained transformer trained on first training data, modifying an architecture of the pre-trained transformer to generate a modified transformer, the modified transformer replacing a dot-product softmax attention layer with a linear kernel squared dot product attention layer utilizing Group Normalization, receiving second training data, and training the modified transformer based on the training data.