Attention Scoring Circuit for Division-Free Transformer Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational load of the attention scoring function in self-attention transformer models, particularly in multi-head self-attention applications, is significant due to extensive normalized function computations, including inner products and divisions, which are time-consuming and energy-intensive.
Innovation Solution
An attention scoring device comprising a pre-processing circuit, post-processing circuit, summing circuit, and arithmetic circuit is used to perform attention scoring pre-processing, post-processing, summation processing, and arithmetic processing, eliminating the need for division computations by converting them into multiplication operations after post-processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If normalized function computation is performed using conventional division operations, then the attention scoring function can be calculated, but the computational load and execution time increase significantly
Solution Approach 1:
The patent transforms the normalized function computation from division operations to multiplication operations by applying mathematical equivalence. Specifically, the division operation a/b is converted to multiplication a*(1/b), where the denominator is pre-computed and inverted once, then multiplied with all numerator elements. This parameter transformation maintains computational accuracy while dramatically reducing execution time, especially for batch processing in multi-head self-attention where the same denominator is reused across multiple computations.
2Measurement precision
If division computations are performed for each exponential function value in the normalized function, then the attention scoring is accurate, but the computational complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-computing the denominator of the normalized function before performing the actual normalization for each element. The denominator (sum of exponential values) is calculated once, then inverted and stored for reuse. This preliminary computation eliminates the need to perform division operations repeatedly for each element in batch processing, significantly reducing computational complexity while maintaining normalization accuracy across all attention scoring calculations.
3Measurement precision
If extensive normalized function computations are performed in multi-head self-attention, then comprehensive attention scoring is achieved, but energy consumption increases
Solution Approach 1:
The patent reduces energy consumption by changing the operational parameter from division to multiplication in the normalized function computation. Multiplication operations are fundamentally less energy-intensive than division operations in digital hardware, especially when performed in parallel across multiple heads and sequence lengths. By pre-computing the inverted denominator and reusing it across all multiplication operations, the system achieves comprehensive attention scoring while minimizing energy expenditure compared to performing repeated division operations.
Data Source
AI summary
An attention scoring device and an operating method of the attention scoring device are provided. The attention scoring device includes a pre-processing circuit, a post-processing circuit, a summing circuit, and an arithmetic circuit. The pre-processing circuit performs attention scoring pre-processing by using an input vector to generate an exponential function value and a value vector. The post-processing circuit performs attention scoring post-processing by using the exponential function value and the value vector to generate a linear combination vector. The summing circuit performs summation processing by using the exponential function value to generate a combined value. The arithmetic circuit performs arithmetic processing by using the linear combination vector and the combined value to generate an attention scoring vector corresponding to the input vector.