Generative Model Quantization With Uniform-Precision Operator Inputs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing quantization methods for generative models result in long calculation times and erroneous results due to hybrid precision calculations, particularly in operators with input data of varying precisions.
Innovation Solution
A model quantization method that involves inserting or deleting fake-quantization nodes to ensure uniform data precision for input data within target operators, thereby obtaining a second quantized network structure through training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If quantization is performed on generative models to improve inference speed, then data processing speed is improved, but calculation time of operators increases and calculation errors occur due to hybrid precision
Solution Approach 1:
The patent changes the precision parameter of input data by inserting fake-quantization nodes that convert float16/float32 data to int8 data before operators process them. This ensures all input data to an operator has uniform precision (int8), eliminating hybrid precision calculation and its associated time penalties and errors.
Solution Approach 2:
The fake-quantization node acts as an intermediary component between the data source and the operator. It mediates the precision mismatch by converting data to a uniform precision format, allowing the operator to receive consistent precision input without requiring complex hybrid precision handling.
2Productivity
If quantization is performed to improve data processing speed, then data processing speed is improved, but calculation errors occur due to hybrid precision
Solution Approach 1:
The patent uniformly changes the precision parameter of all input data to int8 through fake-quantization nodes, eliminating the hybrid precision condition that causes calculation errors. This ensures consistent precision throughout the operator inputs, thereby improving calculation accuracy while maintaining speed benefits.
3Reliability
If fake-quantization nodes are inserted to ensure uniform precision, then calculation accuracy is improved, but device complexity increases
Solution Approach 1:
The patent segments the quantization process by placing fake-quantization nodes at specific locations (input ends or between operators) rather than attempting uniform quantization throughout the entire network. This segmented approach ensures uniform precision where needed while minimizing the addition of complex components elsewhere in the network.
Data Source
AI summary
This application discloses a model quantization method and apparatus, and a device and a medium. The method includes a model quantization method performed by a model quantization device, and the method comprising determining a first quantized network structure from a generative model, the first quantized network structure being a quantized structure of a partial network structure in the generative model, a target operator in the first quantized network structure corresponding to a plurality of pieces of input data having different data precisions; obtaining a second quantized network structure by inserting or deleting a fake-quantization node based on the first quantized network structure, data precisions of the plurality of pieces of input data of the target operator in the second quantized network structure being the same, and the fake-quantization node being a node for quantizing the input data; and training the generative model comprising the second quantized network structure.