Self-Attention Feature Map Sparsification for Long-Sequence Compute
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer deep-learning models, particularly those utilizing multi-head self-attention mechanisms, face significant computational challenges with longer input sequences, as they are quadratic with respect to sequence length, leading to inefficient processing.
Innovation Solution
The method involves adding a feature-map regularization term to the loss function during training to induce sparsity, removing low-magnitude features during inference, and quantizing weights using 8-bit quantization and Exponential-Golomb coding to compress feature and weight maps, thereby reducing computational requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multi-head self-attention mechanisms are used in transformer models, then model accuracy and performance are improved, but computational complexity and processing time increase quadratically with sequence length
Solution Approach 1:
The patent extracts and removes low-magnitude features from feature maps during inference. By identifying and eliminating features with magnitudes below a predetermined threshold, the system reduces the number of computations required in self-attention mechanisms while preserving the most significant features that contribute to model accuracy.
Solution Approach 2:
The patent applies different processing strategies to different parts of the feature maps based on their magnitude. High-magnitude features are preserved and processed in full precision, while low-magnitude features are removed or quantized. This local differentiation allows the system to maintain accuracy for important features while reducing computational load for less significant ones.
2Reliability
If full precision weights and feature maps are maintained throughout the model, then model accuracy is preserved, but memory usage and computational load increase significantly
Solution Approach 1:
The patent changes the precision parameter of weights and feature maps dynamically. During training, full precision is maintained to ensure accurate gradient computation and model convergence. During inference, the precision is reduced through quantization and sparse feature selection, significantly reducing memory usage while maintaining acceptable accuracy through the preservation of high-magnitude features.
Solution Approach 2:
Instead of applying uniform quantization to all features, the patent applies partial action by selectively removing only low-magnitude features below a threshold. This partial removal strategy reduces memory requirements sufficiently while preserving the majority of information contained in significant features, achieving a balance between compression and accuracy.
3Stability of the object's composition
If no feature sparsity is applied during training, then model training stability is maintained, but inference computation and memory requirements remain excessively high
Solution Approach 1:
The patent performs preliminary action by adding a feature-map regularization term to the loss function during training. This regularization encourages the model to learn sparser feature representations, causing many feature magnitudes to naturally approach zero during training. As a result, the model is pre-conditioned for sparsity, enabling efficient inference with minimal additional computation while maintaining training stability through the gradual introduction of the regularization term.
Data Source
AI summary
A method is disclosed to reduce computation in a self-attention deep-learning model. A feature-map regularization term is added to a loss function while training the self-attention model. At least one low-magnitude feature is removed from at least one feature map of the self-attention model during inference. Weights of the self-attention model are quantized after the self-attention model has been trained. Adding the feature-map regularization term reduces activation values of feature maps, and removing the at least one low-magnitude feature from at least one feature map may be performed by setting the low-magnitude feature to be equal to zero based on the low-magnitude feature having a value that is less than a predetermined threshold. Feature maps of the self-attention model quantized and compressed.