Transformer Model Quantization via Self-Attention Map Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transformer models, such as BERT, face challenges with large model sizes requiring significant memory, and existing quantization methods suffer from insufficient accuracy and poor performance with small data, especially when data augmentation is not used, and fail to properly compress self-attention operations.
Innovation Solution
A two-step quantization learning method is employed, where a self-attention map of a teacher model is inserted into a student model during the first step, and then the student model's self-attention map is adjusted to follow the teacher model's in a second step, using knowledge distillation and Kullback-Leibler Divergence to enhance accuracy and compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Weight of stationary object
If quantization is applied to compress the transformer model, then the model size is reduced, but the accuracy becomes insufficient
Solution Approach 1:
A self-attention map recovery module is introduced as an intermediary component between the quantized student model and the output. This module recovers high-precision self-attention maps from the quantized representations, allowing the model to maintain accurate attention computations even with reduced precision weights, thereby resolving the accuracy loss from quantization
Solution Approach 2:
The patent applies different quantization bit depths to different parameter types within the transformer model. Critical parameters such as self-attention weights are preserved at higher precision or recovered through the self-attention map module, while less critical parameters are aggressively quantized, optimizing the trade-off between model size and accuracy
2Quantity of substance
If data augmentation is not used, then the data requirement is reduced, but the accuracy after quantization becomes very poor
Solution Approach 1:
The self-attention map recovery module is pre-configured with the architectural knowledge of attention mechanisms before quantization occurs. This preliminary structure allows the model to recover accurate attention patterns during inference without requiring extensive training data, enabling effective quantization with limited data
Solution Approach 2:
The patent copies the self-attention map structure and computational pattern from the full-precision teacher model to guide the quantized student model. By replicating the attention map generation process and recovering it through the dedicated module, the student model achieves similar accuracy without needing augmented training data
3Device complexity
If self-attention operation is compressed, then the model complexity is reduced, but the core operation cannot be properly performed
Solution Approach 1:
The transformer model is segmented into distinct functional modules: quantization layers, self-attention modules, and a self-attention map recovery module. This segmentation allows the core self-attention operation to be preserved and recovered with high fidelity while other parts of the model can be more aggressively compressed
Solution Approach 2:
The self-attention map recovery module acts as an intermediary that receives quantized attention representations and reconstructs high-precision self-attention maps. This ensures the core self-attention operation maintains its correctness and reliability even when the surrounding model structure is heavily compressed
Data Source
AI summary
A method for quantization learning by a model quantizer that is operating in a computer system and compressing a transformer model. The method may include generating a student model through quantization of the transformer model, performing a first quantization learning by inserting a self-attention map of a teacher model into a self-attention map of the student model, and performing a second quantization learning using a knowledge distillation method so that the self-attention map of the student model follows the self-attention map of the teacher model.