Lookup-Table Gelu Activation Computing for Accurate Low-Area ASICs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computation of the Gaussian error linear unit (Gelu) function in self-attention models is time-consuming and requires significant hardware resources without improving output accuracy, and current ASIC implementations are inefficient.
Innovation Solution
An activation function computing device utilizing lookup tables and a controller to store correspondences between mantissa values and coefficients, employing an approximation function with a Sigmoid function to generate output values, reducing energy consumption and improving accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the Gelu function is implemented through current ASICs, then the computation can be performed, but the hardware area required is large and the output accuracy is not improved
Solution Approach 1:
The Gelu function computation is segmented into multiple components: exponent classification, mantissa processing, lookup table selection, and approximation calculation. This segmentation allows the system to process different parts of the floating-point number separately, reducing the overall hardware area while maintaining accuracy through coordinated processing of each segment
Solution Approach 2:
The system performs preliminary classification of the input exponent and pre-loads the appropriate lookup table based on the exponent value before performing the actual Gelu computation. This preliminary action enables the system to quickly select the optimal approximation parameters, improving both accuracy and reducing the computational hardware area needed
2Measurement precision
If the Gelu function is computed with high precision, then the output accuracy improves, but the computation time increases
Solution Approach 1:
The system pre-calculates and stores approximation coefficients in lookup tables based on exponent values before runtime. During actual computation, the system only needs to perform simple table lookup and basic arithmetic operations, achieving high precision Gelu function results without the time-consuming complex calculations
Solution Approach 2:
The system changes the computation parameters by using different approximation functions and coefficients based on the input exponent value. This allows the system to adapt to different input ranges and maintain high accuracy while minimizing computation time through optimized parameter selection
3Measurement precision
If the Gelu function is computed using complex equations, then the computation can be performed, but the energy consumption increases and accuracy does not improve
Solution Approach 1:
The system extracts the computationally intensive parts of the Gelu function and replaces them with pre-computed lookup tables and simplified approximation formulas. This extraction removes the energy-consuming complex calculations while preserving the essential accuracy through the use of pre-calculated coefficients
Solution Approach 2:
The system uses simple, energy-efficient arithmetic operations and lookup table accesses instead of complex computational circuits. These simpler operations consume significantly less energy while achieving the same computational goal through a different, more efficient approach
Data Source
AI summary
An activation function computing device and a computing method thereof are provided. The activation function computing device computes an input value conforming to a floating-point number format to generate an output value. The activation function computing device includes a plurality of lookup tables and a controller. The plurality of lookup tables respectively store correspondences between a plurality of mantissa values and a plurality of coefficients. The controller selects a selected coefficient from the coefficients according to an input exponent part and an input mantissa part of the input value. The controller computes the selected coefficient and the input value according to an approximation function including a Sigmoid function to generate the output value which conforms to the floating-point number format and has high accuracy.