Neural Network Inference Acceleration Via Quantized Operation Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods to simplify complex operations in neural network models, such as softmax and swish, result in reduced computational costs but often lead to low model accuracy.
Innovation Solution
Convert complex operations in neural network models to simpler operations like lookup tables and perform quantization aware training with a supported bit width to simulate operational logic, ensuring accuracy is maintained during the conversion process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If complex operations like softmax and sigmoid are simplified using lookup tables and arithmetic operations, then computational costs are reduced and inference speed is improved, but model accuracy deteriorates
Solution Approach 1:
The patent applies preliminary action by performing quantization aware training before the actual inference. During training, the model is exposed to quantized data and operations to learn the correct behavior under limited precision constraints. This preliminary preparation ensures that when the simplified operations are used during inference, the model maintains high accuracy while benefiting from reduced computational complexity.
Solution Approach 2:
The patent changes the parameter of data precision by converting floating-point operations to fixed-point operations with specific bit widths (e.g., 8-bit, 16-bit). By adjusting the quantization parameters and bit width, the model achieves a balance between computational efficiency and accuracy. The quantization parameters are optimized during training to preserve the essential information needed for accurate inference.
2Device complexity
If operations are simplified to reduce computational complexity, then device complexity is reduced, but measurement precision (model accuracy) deteriorates
Solution Approach 1:
The patent introduces quantization aware training as an intermediary process between the original complex operations and the simplified operations. This intermediary training phase acts as a bridge, adapting the model to work correctly with simplified operations while maintaining accuracy. The training process serves as a mediator that reconciles the conflict between simplicity and precision.
Solution Approach 2:
The patent replaces complex mechanical computation systems with simpler fixed-point arithmetic systems. By substituting floating-point operations with fixed-point operations and using lookup tables, the computational mechanism is simplified. The quantization aware training ensures that this substitution does not compromise the measurement precision of the model.
Data Source
AI summary
Disclosed are a method and apparatus for accelerating inference of a neural network model, an electronic device, and a medium. The method includes: acquiring image training data, text training data, or speech training data; determining a first neural network model to be accelerated; converting a preset operation on a preset network layer in the first neural network model to a first operation for simulating operational logic of a target operation to obtain a second neural network model; performing, based on the image training data, the text training data, or the speech training data, quantization aware training on the second neural network model by a preset bit width to obtain a third neural network model which is quantized; and converting the first operation of the third neural network model to the target operation, to obtain a target neural network model, which is accelerated, corresponding to the first neural network model.