Pruning Batch Normalization Layers in Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks with batch normalization layers face challenges in deployment on resource-limited devices due to high computational intensity and memory requirements, making it difficult to implement on devices like mobile phones and embedded systems.
Innovation Solution
A method to prune batch normalization layers from pre-trained deep neural networks by removing them and adjusting corresponding linear operation layer weights, ensuring inference accuracy remains unchanged.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If batch normalization layers are included in deep neural networks to improve training convergence, then training performance is improved, but inference time and computational complexity increase
Solution Approach 1:
The patent extracts and removes batch normalization layers from the deep neural network during the inference stage. By taking out these layers that are no longer needed for inference, the model achieves faster processing speed while maintaining the training benefits that were already incorporated into the weights during the training phase.
Solution Approach 2:
The patent performs weight adjustment in advance during the training phase to compensate for the future removal of batch normalization layers. By preliminarily adjusting the weights of adjacent layers to account for the absence of batch normalization, the model is prepared for efficient inference without needing batch normalization layers during deployment.
2Measurement precision
If batch normalization layers are added to deep neural networks to improve training performance, then model accuracy is improved, but model size and memory requirements increase
Solution Approach 1:
The patent removes batch normalization layers from the final deployed model, extracting only the essential computational components. This reduction in model components directly decreases model size and memory requirements while maintaining inference accuracy through preliminary weight adjustments.
Solution Approach 2:
The patent performs weight adjustment during training to preliminarily incorporate the effect of batch normalization into the adjacent layer weights. This preliminary action allows the model to achieve the same accuracy without retaining the batch normalization layers, thereby reducing model size.
3Stability of the object's composition
If batch normalization layers are included in deep neural networks to stabilize training, then training stability is improved, but computational intensity and processing time increase
Solution Approach 1:
The patent extracts batch normalization layers from the inference pipeline, removing the computational overhead associated with these layers. By taking out these layers that provide stability during training but add latency during inference, the model achieves faster processing while maintaining training stability benefits in the weights.
Solution Approach 2:
The patent performs weight adjustment in advance during training to preliminarily compensate for the removal of batch normalization layers. This preliminary weight adjustment embeds the stabilizing effect of batch normalization into the layer weights, allowing inference to proceed without the actual batch normalization computation.
Data Source
AI summary
A method of pruning a batch normalization layer from a pre-trained deep neural network model is proposed. The pre-trained deep neural network model is inputted as a candidate model. The candidate model is pruned by removing the at least one batch normalization layer from the candidate model to form a pruned candidate model only when the at least one batch normalization layer is connected to and adjacent to a corresponding linear operation layer. The corresponding linear operation layer may be at least one of a convolution layer, a dense layer, a depthwise convolution layer, and a group convolution layer. Weights of the corresponding linear operation layer are adjusted to compensate for the removal of the at least one batch normalization. The pruned candidate model is then output and utilized for inference.