Quantized Neural Network Design Flow for Hardware Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models, particularly deep neural networks, are computationally expensive and challenging to implement in real-time due to their complexity, especially when deployed on general-purpose processors, and direct quantization of floating-point weights does not provide adequate performance with limited bit assignments.
Innovation Solution
A method for designing and implementing quantized neural networks by converting normal-precision floating-point models to quantized precision formats, adjusting hyperparameters and quantization parameters, and retraining the models to improve accuracy and performance on hardware accelerators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If direct quantization of floating-point weights is applied to reduce computational complexity, then hardware implementation complexity is lowered, but performance deteriorates when the number of bits assigned is small
Solution Approach 1:
The patent applies parameter changes by systematically varying the number of bits assigned to weights and signals in the quantized neural network. Instead of using a fixed bit width, the method explores different quantization precision levels (e.g., 4-bit, 8-bit, 16-bit) to find the optimal balance between hardware complexity and performance. This allows the system to adapt the quantization parameters to match the specific hardware constraints while maintaining acceptable accuracy.
Solution Approach 2:
The patent introduces dynamic quantization where the number of bits assigned to different parts of the neural network can vary. Rather than uniformly quantizing all weights and activations to the same precision, the system can apply different bit widths to different layers or tensors based on their importance and sensitivity to quantization errors. This dynamic approach allows critical parts of the network to maintain higher precision while less critical parts use lower precision to save hardware resources.
2Use of energy by moving object
If quantization is applied to reduce computational resource consumption, then resource efficiency is improved, but accuracy of the neural network model deteriorates
Solution Approach 1:
The patent systematically adjusts quantization parameters such as bit width, scaling factors, and offset values to optimize the trade-off between resource consumption and accuracy. By changing these parameters, the system can achieve significant reductions in computational resource usage (e.g., reducing from 32-bit floating-point to 8-bit integer operations) while maintaining acceptable accuracy through careful parameter selection and optimization.
Solution Approach 2:
The patent employs feedback mechanisms where the quantized model's performance is evaluated and used to adjust quantization parameters iteratively. The system measures the accuracy loss from quantization and uses this feedback to refine the quantization strategy, such as adjusting bit allocation across different layers or applying post-quantization fine-tuning to recover accuracy while maintaining resource efficiency.
3Measurement precision
If retraining of quantized networks is performed to improve accuracy, then model accuracy is improved, but training time and computational overhead increase
Solution Approach 1:
The patent applies preliminary action by performing quantization-aware training or fine-tuning on a smaller subset of training data or for fewer epochs before full deployment. The system pre-adapts the quantized model to the quantized domain using a reduced training regimen, which recovers much of the accuracy loss from quantization without requiring exhaustive retraining. This preliminary adaptation step significantly reduces the time investment compared to complete retraining while still achieving good accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods and apparatus are disclosed supporting a design flow for developing quantized neural networks. In one example of the disclosed technology, a method includes quantizing a normal-precision floating-point neural network model into a quantized format. For example, the quantized format can be a block floating-point format, where two or more elements of tensors in the neural network share a common exponent. A set of test input is applied to a normal-precision flooding point model and the corresponding quantized model and the respective output tensors are compared. Based on this comparison, hyperparameters or other attributes of the neural networks can be adjusted. Further, quantization parameters determining the widths of data and selection of shared exponents for the block floating-point format can be selected. An adjusted, quantized neural network is retrained and programmed into a hardware accelerator.