Quantization-Aware Training for Neural Network Processing Units
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for quantization-aware training of neural networks do not provide a universal solution to improve post-quantization accuracy, particularly for Neural Network Processing Units (NPUs) that operate with reduced precision, leading to decreased accuracy compared to reference models.
Innovation Solution
The method involves obtaining tensors from NPU inference and either replacing corresponding tensors during training with these inferred tensors or providing noise tensors based on the difference between inferred and training framework tensors, allowing the network to learn quantization noise and improve accuracy without modifying the backward pass.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If quantization is applied to reduce precision for NPU operation, then hardware efficiency and speed are improved, but accuracy deteriorates compared to reference models
Solution Approach 1:
The patent applies preliminary action by performing quantization-aware training before deployment. The method simulates quantization effects during the training phase by replacing floating-point tensors with quantized tensors, allowing the network to learn and adapt to quantization noise in advance. This preliminary adaptation enables the model to maintain higher accuracy after quantization is applied for fast NPU inference.
2Measurement precision
If quantization-aware training is applied to improve post-quantization accuracy, then accuracy is improved, but device complexity and training complexity increase
Solution Approach 1:
The patent uses copying by creating a copy of the original neural network model for training purposes. The method copies the network architecture and initializes weights, then applies quantization to this copy while keeping the original floating-point model intact. This copying approach allows quantization-aware training to be performed without modifying the original model, simplifying the training process and reducing complexity.
Solution Approach 2:
The patent introduces an intermediary mechanism by using a quantization simulator that acts as a bridge between the floating-point training framework and the quantized NPU inference. The simulator replaces floating-point tensors with quantized tensors during training, mediating the transition without requiring changes to the underlying training framework or NPU hardware, thus reducing training complexity.
3Adaptability or versatility
If universal quantization method is developed for different NPUs, then adaptability is improved, but manufacturing precision and optimization difficulty increase
Solution Approach 1:
The patent achieves universality by developing a quantization-aware training method that can be applied to different NPU architectures without requiring architecture-specific modifications. The method uses a general quantization simulator that works with standard neural network frameworks and can adapt to various NPU quantization schemes (INT8, INT4, etc.), making it universally applicable across different hardware platforms while maintaining quantization precision.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure relates to a method for quantization-aware training of a neural network including a neural network processing unit, NPU, the method comprising the following steps: obtaining a plurality of tensors of the neural network from an NPU inference; replacing the corresponding tensors during the training of the neural network with the plurality of tensors of the neural network from the NPU inference.