Dynamic Fixed-Point Conversion for LSTM Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in maintaining accuracy during compression, particularly when converting floating-point numbers to fixed-point numbers, which affects computation speed and resource efficiency.
Innovation Solution
A dynamic fixed-point conversion scheme is proposed for LSTM neural networks, where weights and intermediate calculations are converted to fixed-point numbers during the fine-tuning process, with adaptive conversion schemes determined based on data inputs to minimize accuracy loss and prevent overflow.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If floating-point numbers are converted to fixed-point numbers for compression, then computation speed and resource efficiency are improved, but accuracy is degraded
Solution Approach 1:
The patent applies dynamics by making the fixed-point conversion scheme adaptive rather than static. The system dynamically determines the number of decimal bits based on the actual data distribution and value ranges observed during training, allowing the precision to adjust to the specific requirements of different layers and operations, thereby maintaining accuracy while optimizing for fixed-point computation efficiency
Solution Approach 2:
The patent changes the parameter of decimal bit length in the fixed-point conversion scheme from a fixed value to a dynamically determined value. By adjusting this parameter based on the actual data characteristics and requirements of different neural network components, the system achieves optimal balance between precision and computation speed
2Device complexity
If a static fixed-point conversion scheme is used, then device complexity is reduced, but overflow occurs during training
Solution Approach 1:
The system transitions from a static conversion scheme to a dynamic one where the decimal bit length is determined based on the actual data characteristics. This dynamic adaptation allows the system to prevent overflow by adjusting precision to match the actual data requirements, rather than using a fixed scheme that may be either too conservative or insufficient
Solution Approach 2:
The patent performs preliminary analysis of data distribution and value ranges before finalizing the fixed-point conversion scheme. By examining the actual data characteristics in advance and determining appropriate decimal bit lengths based on observed requirements, the system prevents overflow issues before they occur during training
3Reliability
If precision is increased to prevent overflow, then reliability is improved, but computation speed decreases
Solution Approach 1:
The system optimizes the decimal bit length parameter dynamically based on actual data requirements rather than using excessive precision. By adjusting this parameter to match the specific needs of each layer and operation, the system achieves sufficient precision for reliability while avoiding the computational overhead of unnecessarily high precision
Data Source
AI summary
The present disclosure proposes a fixed-point training method and apparatus based on dynamic fixed-point conversion scheme. More specifically, the present disclosure proposes a fixed-point training method for LSTM neural network. According to this method, during the fine-tuning process of the neural network, it uses fixed-point numbers to conduct forward calculation. Accordingly, within several training cycles, the network accuracy may returned to the desired accuracy level under floating point calculation.