Static Fixed-Point Conversion for LSTM Neural Network Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in maintaining accuracy after compression, particularly due to the conversion of floating-point numbers to fixed-point numbers, which affects computation speed and resource efficiency.
Innovation Solution
A static fixed-point conversion scheme is proposed for LSTM neural networks, where weights are converted to fixed-point numbers during the fine-tuning process, allowing for forward calculations using a training database to restore accuracy levels similar to floating-point calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If floating-point numbers are converted to fixed-point numbers for compression, then computation speed and resource efficiency are improved, but accuracy is compromised
Solution Approach 1:
The patent applies preliminary action by performing fixed-point conversion on the neural network weights before the actual computation phase. The conversion is done in advance during a fine-tuning process, allowing the system to work with fixed-point representations throughout training and inference, thereby achieving both speed improvement and accuracy preservation
Solution Approach 2:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By systematically converting weight parameters, activation values, and gradient computations to fixed-point arithmetic, the system achieves faster computation while maintaining accuracy through careful bit-width selection and fine-tuning
2Quantity of substance
If neural networks are compressed through quantization, then storage and computational resources are reduced, but accuracy decreases requiring retraining
Solution Approach 1:
The patent applies parameter changes by systematically quantizing neural network parameters (weights, biases, activation values) to lower precision formats. Through fixed-point conversion with carefully selected bit-widths and fine-tuning, the system reduces model size and computational requirements while recovering accuracy through adaptive retraining on the quantized representation
3Measurement precision
If fixed-point conversion is applied during fine-tuning, then accuracy loss from compression is compensated, but additional training time is required
Solution Approach 1:
The patent performs fixed-point conversion and fine-tuning as a preliminary action before deployment. By completing the accuracy recovery process in advance during the fine-tuning phase, the system establishes an optimized fixed-point model that achieves both compression benefits and accuracy preservation without requiring additional time during actual inference
Data Source
AI summary
The present disclosure proposes a fixed-point training method and apparatus based on static fixed-point conversion scheme. More specifically, the present disclosure proposes a fixed-point training method for LSTM neural network. According to this method, during the fine-tuning process of the neural network, it uses fixed-point numbers to conduct forward calculation. Accordingly, within several training cycles, the network accuracy may returned to the desired accuracy level under floating point calculation.