Self-Attention Feature Map Sparsification for Lower Compute
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer deep-learning models, particularly those utilizing multi-head self-attention mechanisms, face significant computational challenges due to their quadratic complexity with respect to input sequence length, leading to inefficient processing of longer sequences.
Innovation Solution
Implementing L1 regularization during training to induce sparsity in key, query, value, and context signals, adding activation layers to zero out small activations, and quantizing and encoding output results to reduce computation, while using parallelization and sparse coding techniques to optimize feature maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multi-head self-attention mechanisms are used in transformer models, then model accuracy and performance are improved, but computational complexity increases quadratically with input sequence length
Solution Approach 1:
The patent extracts and eliminates redundant or less important features from the feature maps through sparsification techniques. By identifying and removing unnecessary computational elements in the self-attention mechanism, the model maintains accuracy on essential features while reducing overall computational complexity quadratic to linear or sub-quadratic scaling.
Solution Approach 2:
The patent applies different processing strategies to different parts of the feature maps based on their importance. Sparsification is applied selectively to less critical regions while preserving full precision in important areas, allowing the model to maintain accuracy where needed while reducing computation in less critical regions.
2Reliability
If full precision feature maps are processed in self-attention mechanisms, then model accuracy is maintained, but computational requirements and processing time increase
Solution Approach 1:
The patent applies partial precision processing to feature maps by sparsifying them to retain only the most significant features. This partial action approach processes only the essential portions of the data at full precision while using reduced precision for less critical features, significantly reducing processing time while maintaining adequate model accuracy.
3Productivity
If L1 regularization is applied to induce sparsity in feature maps, then computational requirements are reduced, but model accuracy may be affected
Solution Approach 1:
The patent changes the precision parameter of feature maps through sparsification, transforming full-precision representations into sparse representations with selective precision levels. This parameter change reduces computational requirements by eliminating redundant precision in less critical features while maintaining sufficient accuracy for model performance.
Data Source
Figure 1A~1C
Figure 2
AI summary
A method is disclosed to reduce computation in a self-attention deep-learning model. A feature-map regularization term is added to a loss function while training the self-attention model. At least one low-magnitude feature is removed from at least one feature map of the self-attention model during inference. Weights of the self-attention model are quantized after the self-attention model has been trained. Adding the feature-map regularization term reduces activation values of feature maps, and removing the at least one low-magnitude feature from at least one feature map may be performed by setting the low-magnitude feature to be equal to zero based on the low-magnitude feature having a value that is less than a predetermined threshold. Feature maps of the self-attention model quantized and compressed.