Neural Network Batch Normalization Fine-Tuning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks using reduced precision formats, such as quantized models, often experience accuracy loss when transitioning from floating-point trained weights, and retraining with stochastic gradient descent is complex and time-consuming, especially when fine-tuning batch normalized layers without access to original training data.
Innovation Solution
The method involves dynamically updating the average and variance during forward propagation in neural network layers and applying transformations using fixed scale and shift parameters, allowing for improved accuracy without full retraining, specifically by fine-tuning batch normalization parameters in quantized models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If quantized models with reduced precision formats are used, then computational efficiency is improved, but accuracy is lost
Solution Approach 1:
The patent changes the parameters of batch normalization by dynamically updating the average and variance during forward propagation, and applying transformations using fixed scale and shift parameters. This allows the quantized model to adapt its normalization parameters to maintain accuracy while using reduced precision formats for computation.
2Measurement precision
If full retraining with stochastic gradient descent is performed to fine-tune batch normalized layers, then accuracy is improved, but time and complexity increase
Solution Approach 1:
The patent performs preliminary updates to the batch normalization parameters during the forward propagation pass itself, rather than requiring a separate retraining phase. By dynamically updating the average and variance and applying transformations with fixed scale and shift parameters, the model achieves fine-tuning as part of the normal inference process, eliminating the need for time-consuming stochastic gradient descent retraining.
3Measurement precision
If full retraining is performed to fine-tune batch normalized layers, then accuracy is improved, but device complexity and computational resources increase
Solution Approach 1:
The patent extracts the batch normalization parameter updates from the complex retraining process and performs them independently during forward propagation. By separating the dynamic updating of average and variance from the fixed scale and shift parameter transformations, the method simplifies the overall process and eliminates the need for complex stochastic gradient descent retraining infrastructure.
Data Source
AI summary
Processors and methods for neural network processing are provided. A method includes receiving a subset of data corresponding to a layer of a neural network for processing using the processor. The method further includes during a forward propagation pass: (1) normalizing the subset of the data corresponding to the layer of the neural network based on an average associated with the subset of the data and a variance associated with the subset of the data, where the normalizing the subset of the data comprises dynamically updating the average and dynamically updating the variance, to generate normalized data and (2) applying a transformation to the normalized data using a fixed scale parameter corresponding to the subset of the data and a fixed shift parameter corresponding to the subset of the data such that during the forward propagation pass neither the fixed scale parameter nor the fixed shift parameter is updated.