CNN Quantization and Pruning for Low-Overhead GPU Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graphics processing units (GPUs) face inefficiencies in processing convolutional neural networks (CNNs) due to the overhead of branches with zero or one weight values, and the need for optimizing convolution operations to improve performance and reduce resource usage.
Innovation Solution
Implementing an optimization mechanism that includes pruning branches with zero weight values, quantization and de-quantization of convolution operations, and eliminating insignificant convolution windows to create a hardware-optimized CNN executable.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional branch-based processing is used in CNNs, then general processing capability is maintained, but computational overhead increases and processing efficiency decreases
Solution Approach 1:
The patent extracts and removes branches with zero weight values from the computational graph, eliminating unnecessary computations. This is achieved by analyzing the computational graph to identify branches where weight values are zero and systematically removing them from the execution path, thereby reducing computational overhead while maintaining processing capability for non-zero branches.
Solution Approach 2:
The patent changes the parameter representation by using quantization to convert weight values into discrete levels. This transformation allows the system to identify and eliminate branches with specific quantized values (such as zero), optimizing the computational path by parameter transformation rather than processing all original precision values.
2Measurement precision
If full precision convolution operations are performed, then computational accuracy is maintained, but resource usage and processing time increase
Solution Approach 1:
The patent applies quantization to change the precision parameter of weight values, converting them from full precision to discrete quantized levels. This parameter transformation reduces the computational complexity of convolution operations while maintaining sufficient accuracy for the application, thereby reducing processing time without completely sacrificing precision.
Solution Approach 2:
The patent performs quantization on weight values rather than on all data throughout the entire computation pipeline. This partial application of precision reduction focuses the optimization on the most impactful parameter (weights) while maintaining higher precision where necessary, achieving a balance between speed and accuracy.
3Reliability
If all convolution windows are processed, then complete feature extraction is achieved, but computational complexity and resource consumption increase
Solution Approach 1:
The patent extracts and eliminates insignificant convolution windows from the computational graph based on analysis of their contribution to the final output. By identifying windows that have minimal or redundant impact on feature extraction, the system removes them from processing, reducing computational complexity while preserving the reliability of the essential feature extraction through the remaining windows.
4Productivity
If standard CNN execution is used, then implementation simplicity is maintained, but performance optimization is limited
Solution Approach 1:
The patent performs preliminary analysis and optimization of the computational graph before actual CNN execution. By pre-identifying zero-weight branches, insignificant convolution windows, and optimization opportunities in the computational structure, the system prepares an optimized execution path in advance, achieving performance improvement without adding complexity to the runtime execution.
Solution Approach 2:
The patent replaces the traditional uniform branch execution mechanism with a selective execution mechanism that skips zero-weight branches. This substitution changes the control flow from a mechanical sequential execution of all branches to an intelligent selective execution that bypasses unnecessary computations, thereby improving performance with manageable complexity.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Herein, an apparatus for optimization of a convolutional neural network (CNN) model after the CNN model has been trained and is ready to be deployed, is provided. The apparatus comprises: a graphics processing unit (GPU) including one or more graphics processors. The GPU implements an optimization mechanism including primitives to implement quantization and de-quantization operations. The GPU receives the CNN model, the CNN model having been trained using a training data set representing a problem being modeled by the CNN model; sets a quantization table to enable nonuniform quantization of data associated with the CNN model, wherein generating the quantization table includes executing a quantization primitive provided by the optimization mechanism; quantizes floating-point data associated with the CNN model from a floating-point format to an 8-bit integer format using the quantization table; and performs an inference operation utilizing the CNN model with the quantized data in the 8-bit integer format.