Compressed CNN Models via Quantization and Layer Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) face inefficiencies due to floating point operations, which limit their deployment on computing platforms like FPGAs lacking dedicated floating point modules and limited memory bandwidth, making them power and memory-intensive.
Innovation Solution
The solution involves quantizing input data and weights from floating point to fixed point formats and fusing convolutional layers with batch normalization layers, removing division operations and simplifying processing, thereby reducing memory requirements and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating point operations are used in CNNs, then computational precision is maintained, but power consumption and memory usage increase significantly
Solution Approach 1:
The patent changes the numerical representation parameter from floating point to fixed point format. This parameter change allows the system to maintain sufficient computational precision for CNN operations while dramatically reducing power consumption and memory requirements, making the model suitable for deployment on resource-constrained devices like FPGAs
Solution Approach 2:
The patent employs lower-precision fixed point arithmetic instead of high-precision floating point operations. This approach uses computationally cheaper operations that consume less power and require less memory bandwidth, accepting a trade-off in precision that is acceptable for many CNN applications
2Measurement precision
If floating point operations are used in CNNs, then computational accuracy is preserved, but memory bandwidth requirements increase
Solution Approach 1:
The patent changes the data format parameter from 32-bit floating point to 8-bit fixed point representation. This parameter change reduces the memory bandwidth requirement by a factor of 4, allowing the CNN model to be deployed on platforms with limited memory bandwidth while preserving sufficient computational accuracy
3Ease of manufacture
If convolutional layers are separated from batch normalization layers, then modular design is maintained, but processing complexity and power consumption increase
Solution Approach 1:
The patent merges the convolutional layer and batch normalization layer into a single fused operation. This merging eliminates the need for separate division operations and intermediate memory transfers, reducing processing complexity and power consumption while maintaining the functional equivalence of the original modular design
4Measurement precision
If division operations are included in batch normalization, then normalization accuracy is improved, but processing time and power consumption increase
Solution Approach 1:
The patent extracts the division operation from the batch normalization process by pre-computing the inverse standard deviation during training. This extraction eliminates the computationally expensive division operation during inference, reducing processing time and power consumption while maintaining normalization accuracy through alternative computational approaches
Data Source
AI summary
Systems and processes for training and compressing a convolutional neural network model include the use of quantization and layer fusion. Quantized training data is passed through a convolutional layer of a neural network model to generate convolutional results during a first iteration of training the neural network model. The convolutional results are passed through a batch normalization layer of the neural network model to update normalization parameters of the batch normalization layer. The convolutional layer is fused with the batch normalization layer to generate a first fused layer and the fused parameters of the fused layer are quantized. The quantized training data is passed through the fused layer using the quantized fused parameters to generate output data, which may be quantized for a subsequent layer in the training iteration.