Activation Function Approximation Using Piecewise Quadratic Fitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing activation functions in artificial neural networks, such as sigmoid and ReLU, incur high computational complexity due to exponential and division operations, leading to slow execution speeds.
Innovation Solution
Approximating nonlinear activation functions using piecewise fitting of quadratic functions and a finite element method, dividing the activation function into segments and storing corresponding parameters in a coefficient memory for efficient computation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exponential and division operations are used in activation functions, then modeling capability is improved, but computational complexity increases and execution speed decreases
Solution Approach 1:
The activation function is divided into multiple segments, with each segment approximated by a different polynomial order. This segmentation allows the system to use higher-order polynomials for complex regions requiring accurate modeling while using lower-order polynomials for simpler regions, thereby reducing overall computational complexity while maintaining modeling capability where needed.
Solution Approach 2:
The patent changes the parameters of the approximation by using variable polynomial orders for different segments rather than a fixed order. This parameter change enables adaptive approximation where the complexity of the polynomial matches the local characteristics of the activation function, optimizing the balance between accuracy and computational efficiency.
2Measurement precision
If higher-order polynomial approximation is used, then accuracy is improved, but computational complexity increases
Solution Approach 1:
The domain is segmented into multiple intervals, each handled with an appropriate polynomial order. This avoids applying high-order polynomials globally, thereby reducing the average computational complexity while maintaining high accuracy in critical regions where the activation function has complex behavior.
Solution Approach 2:
Instead of using high-order polynomials throughout the entire domain (excessive action), the patent applies higher-order approximations only where necessary (partial action), specifically in segments where the activation function exhibits complex nonlinear behavior, thus optimizing the trade-off between accuracy and complexity.
Data Source
AI summary
A device, a method and a storage medium for accelerating activation function in relation to data processing by artificial neural network provides a register for storing a storage table, a matching unit including a plurality of comparators, a logic unit, and a selection unit. The comparators compare an input variable of the activation function with the variable intervals of the activation function to obtain a comparison output result, the logic unit performs a logical operation according to the comparison output result to obtain a logic output result and determines a variable interval to be calculated according to the logic output. The selection unit queries the storage table according to the variable interval to be calculated and obtains parameters of fitted quadratic function. A calculation unit performs calculations on the input variable according to the parameters.