Neural Network Activation Function with Bias Adjustment for Gradient Flow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The convergence speed and accuracy of image recognition models using universal activation functions like sigmoid and tanh are affected by gradient disappearance as the gradients approach zero at both ends, leading to slower convergence and reduced accuracy, especially in deeper neural networks.
Innovation Solution
A novel activation function is constructed by adding a preset bias adjustment function to the basic activation function, specifically using a trainable weight parameter and a quadratic term, which helps maintain a non-zero gradient and improves convergence speed and accuracy by avoiding gradient disappearance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If universal activation functions (sigmoid or tanh) are used in deep neural networks, then the network structure can be implemented, but the gradient disappears as depth increases, causing slow convergence and reduced accuracy
Solution Approach 1:
The patent modifies the activation function by adding a learnable bias term to the basic sigmoid or tanh function. This parameter change allows the activation function to adapt its output range and gradient characteristics during training, preventing gradient disappearance in deep networks while maintaining the benefits of these universal activation functions.
Solution Approach 2:
The patent creates a composite activation function by combining the basic activation function (sigmoid or tanh) with a bias adjustment function. This composite structure integrates the nonlinear transformation capability of the basic function with the gradient-preserving properties of the bias term, resolving the contradiction between network depth and gradient flow.
2Stability of the object's composition
If the gradient of activation function approaches zero at both ends, then the function provides stable output range, but the convergence speed and recognition accuracy are affected
Solution Approach 1:
The patent introduces a dynamic bias term that is learned during training, allowing the activation function to adjust its characteristics based on the input data distribution. This dynamic adjustment maintains stable output ranges while preventing gradients from vanishing, thus improving convergence speed without sacrificing stability.
Solution Approach 2:
The activation function becomes self-adjusting through the learnable bias parameter, which automatically adapts to maintain optimal gradient flow during training. This self-service mechanism allows the function to maintain stable outputs while inherently preventing gradient disappearance, improving convergence without external intervention.
Data Source
AI summary
A method of image recognition includes: acquiring an image training sample data set, where the image training sample data set includes image training sample data and tag information corresponding to the image training sample data; constructing basic activation function and preset bias adjustment function as preset activation function in an addition relationship, determining the preset activation function to be an activation function of a neural network model, to obtain an initial neural network model, where the preset bias adjustment function is a function constructed from a symbol function, a first trainable parameter and a quadratic term in a multiplication relationship; inputting the image training sample data set into the initial neural network model for training until the model converges, to obtain a trained neural network model; when an image to be recognized is acquired, outputting a recognition result corresponding to the image by using the trained neural network model.