LSTM Weight Quantization for Low-Precision Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks, particularly LSTM networks, face challenges in being deployed on low-precision hardware without significant accuracy loss, requiring modifications to the training pipeline or re-training, and existing quantization-aware training methods can be unstable and complex.
Innovation Solution
A method to quantize trained LSTM neural networks by converting floating-point weights to fixed-point integers, allowing deployment on low-precision hardware with minimal accuracy loss, without altering the training pipeline, using asymmetric or symmetric quantization techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If LSTM neural networks are deployed on low-precision hardware, then hardware efficiency and resource consumption are improved, but model accuracy deteriorates
Solution Approach 1:
The patent applies parameter changes by converting weight parameters from floating-point representation to fixed-point representation with specific bit-widths (e.g., 8-bit, 16-bit). This transformation allows the model to operate on low-precision hardware while maintaining acceptable accuracy through controlled parameter representation changes, directly resolving the contradiction between hardware efficiency and model accuracy.
2Measurement precision
If quantization-aware training is used to maintain accuracy on low-precision hardware, then model accuracy is improved, but training complexity and instability increase
Solution Approach 1:
The patent applies preliminary action by performing quantization on the weights after the model is already trained. Instead of incorporating quantization constraints during the training process (which would increase complexity), the method pre-trains the model using standard floating-point training, then applies post-training quantization to convert weights to fixed-point format. This sequential approach maintains training simplicity while achieving deployment readiness for low-precision hardware.
3Adaptability or versatility
If re-training is performed to adapt to low-precision hardware, then deployment compatibility is improved, but time consumption and computational resources increase
Solution Approach 1:
The patent applies preliminary action by completing the full training process first using floating-point arithmetic, then applying a separate post-training quantization step. This two-stage approach avoids the need to re-train the entire model from scratch for deployment compatibility. The quantization step processes already-trained weights efficiently, significantly reducing time consumption and computational resources compared to re-training, while still achieving deployment compatibility with low-precision hardware.
Data Source
AI summary
Method for quantizing a trained long short-term memory (LSTM) neural network having a plurality of weights, the method comprising: obtaining data specifying trained floating-point values for each of the weights of the trained LSTM neural network, the trained LSTM neural network comprising one or more LSTM layers, each LSTM layer having a plurality of gates and each of the plurality of gates being associated with an input weight matrix and a recurrent weight matrix; quantizing the trained LSTM neural network, comprising: for each gate, quantizing the elements of the input weight matrix to a target fixed bit-width; for each gate, quantizing the elements of the recurrent weight matrix to the target fixed bit-width; and providing data specifying a quantized LSTM neural network for use in performing quantized inference.