ReELU Activation and Weight Initialization for Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models face challenges such as the Vanishing Gradient problem and the Dying ReLU problem due to inadequate activation functions and initialization methods, which hinder stable learning, especially in deep neural networks.
Innovation Solution
The use of a ReELU activation function that applies a rectified linear unit (ReLU) function in the positive number region and a linear function with different gradients in the negative number region, combined with an initialization method that sets initial weights based on a normal probability distribution, ensuring stable learning by preventing convergence to specific values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional activation functions (sigmoid, tanh) are used, then the network can learn basic patterns, but the Vanishing Gradient problem occurs and learning becomes impossible in deep networks
Solution Approach 1:
The patent changes the parameters of the activation function by introducing ReELU with different gradient values (α1 for negative region, α2 for positive region) instead of using traditional sigmoid or tanh functions. This parameter change allows the activation function to maintain gradient flow in deep networks while preserving non-linearity, thus solving the vanishing gradient problem and enabling stable learning in deep architectures.
Solution Approach 2:
The patent creates a composite activation function (ReELU) that combines characteristics of ReLU (linear activation with non-zero gradient) and ELU (exponential linear unit with negative region handling). This composite approach merges the advantages of different activation functions to achieve both stable gradient flow and effective learning in deep networks.
2Productivity
If ReLU activation function is used, then the Vanishing Gradient problem is solved, but the Dying ReLU problem occurs where neurons become inactive and output converges to 0
Solution Approach 1:
The patent modifies the ReLU function by introducing a parameter α2 for the positive region that allows the gradient to be non-zero and configurable. This parameter change prevents neurons from dying by ensuring that positive inputs continue to propagate gradients effectively, thus solving the dying ReLU problem while maintaining the computational efficiency of ReLU.
Solution Approach 2:
The patent makes the activation function dynamic by allowing different gradient parameters (α1, α2) that can be adjusted during training. This dynamic approach enables the network to adapt the activation characteristics to different layers and training stages, preventing neurons from becoming permanently inactive while maintaining gradient flow.
3Ease of manufacture
If standard weight initialization is used, then the network can start training, but weights converge to the same value and channel gains cannot be obtained
Solution Approach 1:
The patent applies local quality by initializing weights differently for different channels and layers based on their specific characteristics. Instead of using a uniform initialization scheme, the method tailors the initialization parameters (α1, α2) to each channel's requirements, ensuring that each channel maintains its unique gain characteristics throughout training and prevents weight convergence to identical values.
Solution Approach 2:
The patent performs preliminary action by carefully initializing weights with specific parameter distributions before training begins. This pre-initialization strategy ensures that neurons start with diverse weight values and appropriate gradient flows, preventing the convergence to identical weights and enabling each channel to develop its own gain characteristics from the outset.
4Stability of the object's composition
If activation function output converges to a specific value, then the computation becomes stable, but learning performance deteriorates due to lack of information flow
Solution Approach 1:
The patent changes the parameters of the activation function (α1, α2) to control the output range and gradient magnitude. By carefully selecting these parameters, the network achieves stable outputs that do not converge to fixed values, while maintaining sufficient information flow for effective learning. The parameters are tuned to balance stability and information preservation.
Solution Approach 2:
The patent incorporates feedback mechanisms through the activation function parameters that adjust the output based on the input distribution and training progress. This feedback ensures that outputs remain stable within a useful range while preventing convergence to specific values, thereby maintaining learning performance throughout training.
Data Source
AI summary
Provided is an artificial neural network learning apparatus for deep learning. The apparatus includes an input unit configured to acquire an input data or a training data, a memory configured to store the input data, the training data, and a deep learning artificial neural network model, and a processor configured to perform computation based on the artificial neural network model, in which the processor sets the initial weight depending on the number of nodes belonging to a first layer and the number of nodes belonging to a second layer of the artificial neural network model, and determines the initial weight by compensation by multiplying a standard deviation (σ) by a square root of a reciprocal of a probability of a normal probability distribution for a remaining section except for a section in which an output value of the activation function converges to a specific value.