Depth-wise Convolutional Layer Initialization for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The efficiency and convergence of neural network training often depend on the initialization of parameters, and existing methods like Glorot initialization may not adequately handle scenarios where the number of inputs and outputs differ significantly.
Innovation Solution
A new initialization method for the parameters of a depth-wise convolutional layer, where values are drawn from a predefined probability distribution with a variance characterized by the reciprocal of the square root of the product of the number of filters at each depth, leading to faster convergence during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Glorot initialization is used for neural network parameters, then training can proceed with standard initialization, but training convergence is slow or fails when the number of inputs and outputs differ significantly
Solution Approach 1:
The patent modifies the initialization parameters by changing the variance calculation from the standard Glorot method to incorporate the upscaling factor. Specifically, the variance is set to 1/(fan_in * upscaling_factor^2) instead of the conventional 2/(fan_in + fan_out), where upscaling_factor accounts for the discrepancy between input and output dimensions. This parameter adjustment ensures that gradients remain stable even when the layer transforms data between significantly different dimensional spaces.
2Device complexity
If standard initialization is used, then the process is simple, but gradients may vanish or explode during training
Solution Approach 1:
The patent introduces an upscaling factor into the initialization parameter calculation to account for dimensional transformations. The variance is calculated as 1/(fan_in * upscaling_factor^2), where upscaling_factor = sqrt(output_channels / input_channels) for convolutional layers. This modification maintains gradient stability without significantly increasing computational complexity, as it only requires a simple additional calculation during initialization.
Data Source
AI summary
A computer-implemented method for training a neural network. The neural network is configured to determine an output signal based on an input signal. The training includes training parameters of a depth-wise convolutional layer of the neural network. The depth-wise convolutional layer is initialized based on values drawn from a predefined probability distribution. A variance of the probability distribution is characterized by a reciprocal of a square root of a number of filters applied at each depth of an input of the depth-wise convolutional layer.