Softmax Function Piecewise Approximation Integer Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The SoftMax function in neural networks requires exponential operations, which are computationally expensive and energy-intensive, posing challenges for efficient computation, especially in resource-constrained processors like microcontrollers and digital signal processors.
Innovation Solution
The method employs piecewise approximation and integer-based operations to compute the SoftMax function, avoiding exponential calculations by converting them into exponentiations of two using the natural logarithm of two, and utilizing binary shifts and integer arithmetic for efficient computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exponential operations are used to compute the SoftMax function, then accuracy is maintained, but computational time and energy consumption increase significantly
Solution Approach 1:
The patent transforms the SoftMax computation from using natural exponential operations to using base-2 exponential operations. This parameter change in the mathematical basis allows the computation to be performed using efficient binary shift operations and lookup tables, dramatically reducing computational time while maintaining accuracy through carefully designed approximation functions and correction terms
Solution Approach 2:
The patent replaces the traditional floating-point exponential computation mechanism with an integer-based computation mechanism using binary shifts, additions, and pre-computed lookup tables. This substitution eliminates the need for complex floating-point arithmetic units and exponential function implementations, reducing both computational time and hardware complexity
2Measurement precision
If exponential operations are used to compute the SoftMax function, then correct probability distribution is obtained, but energy consumption increases
Solution Approach 1:
The patent changes the computational parameter from natural exponential (e^x) to base-2 exponential (2^x), enabling the use of efficient bit-shift operations and integer arithmetic. This parameter transformation maintains the mathematical correctness of the SoftMax function while dramatically reducing energy consumption by avoiding complex floating-point exponential calculations
Solution Approach 2:
The patent performs preliminary computations by pre-calculating and storing exponential values in lookup tables, and by pre-computing normalization constants. This preliminary action eliminates the need for real-time exponential calculations during inference, significantly reducing energy consumption while maintaining accuracy through the use of pre-computed reference values
3Productivity
If integer-based operations with piecewise approximation are used, then computational efficiency improves, but implementation complexity increases
Solution Approach 1:
The patent segments the SoftMax computation into distinct stages: input normalization, base-2 conversion using piecewise approximation, exponential calculation via binary shifts, and final normalization. Each segment is implemented using simple integer operations and lookup tables, making the overall complex function manageable and efficient to compute while maintaining high productivity
Data Source
AI summary
A computer-implemented method of processing a classification output of a neural network. The method includes, for each input number yi, determining, using an approximation of a SoftMax function that uses integer-based operations without using an exponential operation, a binary number qi that represents a probability distribution of the input number yi, each input number being an offset of a respective classification output of the neural network. The method includes for each input number yi, determining a normalized probability pi from the binary numbers qi.