Softmax Approximation via Leaky ReLU and Polynomial Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The conventional softmax function in neural network classifiers is computationally intensive and energy-consuming due to the numerous polynomial orders and float32 format input values, leading to time-consuming and energy-intensive calculations.
Innovation Solution
An approximation method for the softmax function using a Leaky Rectified Linear Unit (Leaky ReLU) and polynomial function computations of second-order to fifth-order, combined with a Clamp function and a protection value, to reduce the complexity of exponential function calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the conventional softmax function is used with float32 format input values and multiple polynomial orders, then the calculation accuracy is maintained, but the calculation time and energy consumption increase significantly
Solution Approach 1:
The patent changes the input parameter format from float32 to integer type, and transforms the exponential function calculation into polynomial operations of limited orders (second-order to fifth-order). This parameter transformation maintains calculation accuracy while significantly reducing computational complexity and execution time.
Solution Approach 2:
The softmax function calculation is segmented into distinct stages: integer input processing, exponential approximation through polynomial operations (second-order to fifth-order), and normalization. This segmentation allows each stage to be optimized independently, reducing overall calculation time while maintaining accuracy.
2Measurement precision
If the conventional softmax function is used with float32 format input values and multiple polynomial orders, then the calculation accuracy is maintained, but the energy consumption increases
Solution Approach 1:
The patent changes the input parameter format from float32 to integer type, and transforms the exponential function calculation into polynomial operations of limited orders (second-order to fifth-order). This parameter transformation maintains calculation accuracy while significantly reducing computational complexity and execution time.
Solution Approach 2:
The patent uses simpler, computationally cheaper operations (integer arithmetic and low-order polynomials) instead of expensive float32 exponential calculations. These simplified operations consume less energy while providing sufficient accuracy for the softmax function.
3Device complexity
If the exponential function approximation is performed using polynomial functions of second-order to fifth-order, then the calculation complexity is reduced, but the approximation error increases
Solution Approach 1:
The softmax function calculation is segmented into distinct stages: integer input processing, exponential approximation through polynomial operations (second-order to fifth-order), and normalization. This segmentation allows each stage to be optimized independently, reducing overall calculation time while maintaining accuracy.
Solution Approach 2:
The patent uses polynomial operations up to fifth-order, which provides sufficient approximation accuracy for practical applications. This partial action approach avoids the need for extremely high-order polynomials or exact exponential calculations, achieving an optimal balance between complexity and precision.
4Speed
If the input values are converted to integer type, then the calculation speed increases, but the precision of representation decreases
Solution Approach 1:
The patent changes the input parameter format from float32 to integer type, and transforms the exponential function calculation into polynomial operations of limited orders (second-order to fifth-order). This parameter transformation maintains calculation accuracy while significantly reducing computational complexity and execution time.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
An approximation method of softmax function which converts an input value of k-dimensional vector into an output value of m-dimensional vector, comprises: an exponential function approximation computing step performing a Leaky ReLU computation on one input value of k-dimensional vector to obtain a Leaky ReLU computation value and performing a polynomial function computation of a certain order based on the Leaky ReLU computation value to obtain an exponential approximation value, the exponential function approximation computing step repeated for another input value of k-dimensional vector to obtain another exponential approximation value; an addition computing step adding the exponential approximation values to obtain a sum value; and a division computing step dividing at least one of the exponential approximation values obtained in the exponential function approximation computing step by the sum value to obtain an output value of m-dimensional vector.