Neural Network Batch Normalization Bias Elimination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing Batch Normalization method in neural networks introduces artificial bias, which, if not eliminated, can lead to increased bit widths and difficulty in fixed-point realization, resulting in wasted information bits and expanded bit widths in the network.
Innovation Solution
A method and apparatus that process initial bias values of neural network layers to obtain equivalent bias values, determining and setting bias values between layers to zero, with the equivalent bias value of a target layer used as the bias value, thereby reducing artificial bias and bit widths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If Batch Normalization is applied to speed up deep network training, then training speed is improved, but artificial bias is introduced which amplifies and causes difficulty in fixed-point realization
Solution Approach 1:
The patent extracts and removes the harmful artificial bias component from the Batch Normalization output. By calculating the bias value introduced by Batch Norm and subtracting it from the output, the method retains the training speed benefits while eliminating the bias amplification problem that causes fixed-point realization difficulties.
Solution Approach 2:
The patent converts the harmful artificial bias into a beneficial correction process. By calculating the bias value and deliberately subtracting it, the method transforms the problematic bias amplification into a controlled bias elimination process, improving fixed-point compatibility while maintaining Batch Norm's training acceleration benefits.
2Manufacturing precision
If output data of each layer is quantified directly, then quantization is achieved, but a lot of bit width is wasted on invalid data bias
Solution Approach 1:
The patent extracts the invalid bias component from the quantization process. By removing the artificial bias before quantization, the method ensures that bit width is allocated only to represent valid data information, eliminating the waste of bits on bias representation and improving quantization efficiency.
3Device complexity
If bias values are not eliminated, then network processing is simple, but bias is gradually amplified leading to difficulty in fixed-point realization
Solution Approach 1:
The patent applies preliminary action by eliminating bias before it can be amplified through subsequent network layers. By subtracting the calculated bias value at each Batch Normalization layer, the method prevents bias accumulation and amplification, ensuring fixed-point realization feasibility without significantly increasing processing complexity.
Data Source
AI summary
A neural network batch normalization optimization method includes: setting a first network layer in a neural network as a starting layer; sequentially obtaining initial bias values of different network layers backwards starting from the starting layer; calculating equivalent bias values of the different network layers; determining whether there is a target network layer, wherein a ratio of the equivalent bias value corresponding to a previous layer of a target network layer to the equivalent bias value corresponding to the target network layer is no less than a pre-set threshold value; and if the target network layer is present, setting the bias values of the different network layers between the starting layer and the previous layer of the target network layer to zero, and taking the equivalent bias value of the target network layer as a bias value of the target network layer.