Transformer Attention Outlier Attenuation via Clipped Softmax
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer neural networks face increased computational complexity due to the generation of significant outliers, which requires large and complex data types, leading to high computational expense and power consumption.
Innovation Solution
The implementation of a clipped softmax function or gated attention block within the transformer neural network to attenuate outlier values, allowing data to be processed using smaller and simpler data types, thereby reducing computational complexity and power usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If transformer neural networks process data without outlier attenuation, then inference performance may be maintained, but computational complexity and power consumption increase significantly
Solution Approach 1:
The patent extracts and removes outlier values from the attention output before they propagate through the network. By identifying and attenuating these extreme values through clipping operations, the system eliminates the harmful effect of outliers that would otherwise require complex data types and excessive computational resources to handle, thereby reducing power consumption while maintaining inference performance.
Solution Approach 2:
The patent changes the parameter distribution of the attention output by applying clipping thresholds. This transforms the extreme parameter values (outliers) into bounded values within a specific range, allowing the use of simpler data types and reducing the computational complexity required for subsequent processing operations.
2Measurement precision
If transformer neural networks use large and complex data types to handle outliers, then data accuracy is maintained, but computational expense increases
Solution Approach 1:
The patent extracts outlier values from the normal data distribution and applies a separate clipping operation to them. This separation allows the main data stream to use simpler data types while the outliers are handled through bounded clipping operations, reducing overall computational complexity without sacrificing data accuracy for the majority of values.
Solution Approach 2:
The patent applies different processing quality to different parts of the data: normal values are processed with standard precision, while outlier values are processed with clipping to bounded ranges. This local differentiation in quality control maintains accuracy where needed while reducing complexity where extreme values would otherwise dominate resource requirements.
3Measurement precision
If transformer neural networks process outlier values with high precision, then inference accuracy is maintained, but power consumption increases
Solution Approach 1:
The patent extracts outlier values from the attention output and applies clipping to bound them within specific ranges. This extraction and bounding operation prevents outliers from consuming excessive computational resources and power, while maintaining inference accuracy by ensuring that clipped values remain within meaningful ranges that preserve the decision-making capability of the network.
4Device complexity
If transformer neural networks attenuate outlier values, then computational resources are reduced, but data dynamic range is decreased
Solution Approach 1:
The patent extracts only the extreme outlier values for clipping, leaving the normal data distribution intact. This selective extraction maintains the dynamic range of the majority of data while reducing the impact of extreme values that would otherwise require the system to allocate resources for the full extreme dynamic range.
Solution Approach 2:
The patent applies parameter clipping to bound outlier values within specific ranges, changing their parameter distribution from unbounded extremes to bounded values. This local parameter change reduces the overall dynamic range requirements for the system while preserving the dynamic range of normal operational values.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for processing data using a transformer neural network. The method generally includes receiving an input for processing using a transformer neural network. An attention output is generated in the transformer neural network. Generally, the attention output may be generated such that outlier values for the attention output are attenuated in the transformer neural network. An output of the transformer neural network is generated based on the generated attention output.