ReLU Weight Initialization and Learning Rate Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The initialization of weights in artificial neural networks (ANNs) with rectifier linear units (ReLUs) can lead to suboptimal performance due to the high probability of negative inputs resulting in zero outputs, affecting the convergence of weight updates and the efficiency of training, especially when using multiple hidden layers.
Innovation Solution
The proposed solution involves initializing weights to limit the fraction of negative inputs to ReLU units, using a specific distribution with a mean of 1.5 and standard deviation of 1.0, and adjusting the learning rate by scaling factors for each layer to ensure that a high fraction of ReLU nodes are activated, thereby improving the convergence and accuracy of ANN training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If random initialization of weights is used in ANNs with ReLU units, then the initialization process is simple, but the fraction of activated ReLU nodes is low (about 50% initially, decreasing post-training)
Solution Approach 1:
The patent changes the parameters of the weight initialization distribution from the conventional zero-mean Gaussian distribution to a non-zero-mean distribution with mean μ=1.5 and standard deviation σ=1.0. This parameter change ensures that the initial weights produce positive inputs to ReLU units, thereby activating a high fraction of nodes (over 97%) during training, while maintaining the simplicity of the initialization process.
2Ease of operation
If conventional weight initialization is used, then the initialization method is straightforward, but the convergence of weight updates is affected due to high probability of zero outputs from negative inputs
Solution Approach 1:
The patent modifies the parameters of the weight initialization distribution to mean μ=1.5 and standard deviation σ=1.0, which ensures that inputs to ReLU units are predominantly positive. This change improves the reliability of weight update convergence by ensuring that ReLU units remain active during training, while preserving the straightforward nature of the initialization method.
Solution Approach 2:
The patent performs preliminary action by carefully initializing weights with a specific non-zero-mean distribution before training begins. This preliminary initialization ensures that ReLU units are activated during training, preventing the dead ReLU problem and ensuring reliable convergence of weight updates throughout the training process.
3Ease of operation
If standard learning rate is used for all layers, then the training process is simple, but the activation fraction of ReLU nodes is not optimized across different layers
Solution Approach 1:
The patent applies local quality by using different learning rates for different layers of the neural network. Specifically, the learning rate is scaled by a factor of 0.1 for layers closer to the output and a factor of 1.0 for layers closer to the input. This local differentiation optimizes the activation fraction of ReLU nodes in each layer, ensuring that over 97% of nodes remain active throughout training.
Data Source
AI summary
A data processing technique uses an Artificial Neural Network (ANN) with Rectifier Linear Units (ReLU) to yield improve accuracy in a runtime task, for example, in processing audio-based data acquired by a speech-enabled device. The technique includes a first aspect that relates to initialization of the ANN weights to initially yield a high fraction of positive outputs from the ReLU. These weights are then modified using an iterative procedure in which the weights are incrementally updated. A second aspect relates to controlling the size of the incremental updates (a “learning rate”) during the iterations of training according to a variance of the weights at each layer.