Neural Network Normalization Removal for Stable Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The removal of normalization structures in deep learning models during training leads to instability and poor model quality, affecting the accuracy of information output.
Innovation Solution
Gradually remove a specified number of normalization structures in the training process based on a target probability or number of steps to maintain training stability and improve model quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If normalization structures are removed in the training process, then the delay of model prediction is shortened, but the stability of model training deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-processing the normalization structure into two separate structures before training begins. This preprocessing step converts the normalization operation into forms that can be executed without actual normalization computation during training, thereby reducing prediction delay while maintaining training stability through careful design of the split structures.
Solution Approach 2:
The patent segments the normalization structure into two distinct normalization structures, separating the normalization function into different parts of the neural network. This segmentation allows one structure to handle training stability while the other is optimized for prediction speed, resolving the contradiction between stability and delay.
2Productivity
If normalization structures are removed in the training process, then the productivity of model training is improved, but the quality of the trained model deteriorates
Solution Approach 1:
By pre-processing the normalization structure into two separate structures before training, the patent enables faster training execution while preserving model quality. The preliminary transformation ensures that the essential normalization function is maintained in a form that supports both productivity and quality objectives.
Solution Approach 2:
Segmenting the normalization structure allows different parts to serve different functions: one part maintains training stability and model quality, while the other optimizes for training productivity. This segmentation resolves the contradiction between speed and quality.
3Device complexity
If normalization structures are removed in the training process, then the device complexity is reduced, but the reliability of model output deteriorates
Solution Approach 1:
The pre-processing of the normalization structure into two separate structures simplifies the overall device architecture by eliminating the need for complex normalization computations during training and prediction. This preliminary transformation maintains reliability by preserving the essential normalization function in a simplified form.
Solution Approach 2:
Segmenting the normalization structure reduces device complexity by distributing the normalization function across two simpler structures rather than one complex structure. The segmentation maintains reliability by ensuring that the combined function of the two structures equals the original normalization function.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
A method for processing information including: acquiring to-be-processed information; and taking the to-be-processed information as an input of a processing model to acquire target information output by the model. The processing model is acquired by training a preset model including a plurality of operation modules and a plurality of normalization structures, which one of the plurality of normalization structures corresponds to one of the plurality of operation modules. The normalization structure is configured to normalize an output of the corresponding operation module. The processing model is acquired by removing a specified number of normalization structures according to a target probability or a number of steps for training the preset model in a process of training the preset model. As such, training of the preset model cannot be interfered and has high stability, the quality of the processing model can be ensured, and the accuracy of the target information is improved.