Neural Network Activation Function for Stable Gradients and Fast Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing activation functions in neural networks, such as Sigmoid, Tanh, and ReLU, face issues like high calculation cost, output limitations, gradient instability, and sparsity, which affect convergence speed and efficiency.
Innovation Solution
A new activation function f(x) expressed as f(x)={1-e-ax, x>0, 0, x≤0} is implemented using an analog circuit with rectifier and gain modules, and a digital system with iterative calculations to achieve a non-linear relationship between input and output signals, addressing the limitations of existing functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If Sigmoid activation function is used, then output is bounded between 0 and 1, but calculation cost increases due to exponentiation and division operations
Solution Approach 1:
The patent replaces the expensive Sigmoid function with a computationally cheaper piecewise linear function that uses simple operations (multiplication, addition, maximum) instead of exponentiation and division, achieving similar bounded output behavior with much lower computational cost
Solution Approach 2:
The patent introduces a new functional form with parameters a and b that can be adjusted to control the output range and characteristics, allowing the activation function to maintain stability properties while reducing computational complexity through algebraic operations
2Productivity
If ReLU activation function is used, then calculation speed increases, but output sparsity causes gradient disappearance
Solution Approach 1:
The patent creates an asymmetric piecewise function where the positive and negative branches have different behaviors - the positive branch provides non-zero output for gradient flow while the negative branch handles suppression, avoiding the complete gradient disappearance problem of ReLU
Solution Approach 2:
The patent combines multiple functional segments (linear positive branch, linear negative branch, and saturation region) into a composite activation function that integrates the speed benefits of piecewise functions with the gradient stability of continuous differentiable functions
3Adaptability or versatility
If complex activation functions are used, then nonlinear transformation capability improves, but convergence speed decreases
Solution Approach 1:
The patent divides the input domain into distinct segments (x < -b/a, -b/a ≤ x ≤ 0, x > 0) with different linear transformations in each segment, providing nonlinear overall behavior through piecewise linearity while maintaining fast computation in each segment
Solution Approach 2:
The patent creates a dynamic activation function that adapts its behavior based on the input range, switching between different linear transformations and saturation regions to optimize both transformation capability and convergence speed for different input conditions
Data Source
AI summary
A system for signal conversion in a neural network can include: a processing module configured to control an output signal of the processing module to be a fixed value, when an input signal of the processing module is in a first interval; the processing module being configured to control the output signal to be in a preset non-linear relationship with the input signal, when the input signal is in a second interval; and where the output signal increases nonlinearly with the increase of the input signal and finally converges.