Neural Network Training Using Floating-Point to Fixed-Point Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The training time for learning parameters in neural networks is prolonged due to high computational and memory loads, and using fixed-point numbers results in lower accuracy due to their narrower representable value range.
Innovation Solution
A method that initially trains a neural network using floating-point numbers and then converts the parameters to fixed-point numbers, dynamically adjusting the point position based on overflow percentages to maintain accuracy while reducing data loads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If training is performed using floating-point numbers, then training accuracy is improved, but training time and computational load increase
Solution Approach 1:
The training process is divided into two distinct phases: a first training phase using floating-point numbers to achieve high accuracy, and a second training phase using fixed-point numbers to reduce computational load. This segmentation allows the system to leverage the advantages of both data types at different stages of training.
Solution Approach 2:
The data type used for training is dynamically changed from floating-point to fixed-point based on the training stage and performance requirements. The system transitions between different numerical representations to optimize the trade-off between accuracy and efficiency.
2Productivity
If training is performed using fixed-point numbers, then computational load and memory usage are reduced, but training accuracy decreases
Solution Approach 1:
The system performs preliminary training using floating-point numbers to establish accurate baseline performance and parameter ranges. This preliminary action ensures that the model learns correct patterns before transitioning to the more efficient but less precise fixed-point arithmetic.
Solution Approach 2:
The system monitors training performance and accuracy metrics to determine when to switch between floating-point and fixed-point training modes. Feedback from the training process guides the selection of appropriate data types to maintain accuracy while improving efficiency.
3Quantity of substance
If fixed-point number precision is increased to maintain accuracy, then data load increases, but if precision is decreased to reduce data load, then accuracy deteriorates
Solution Approach 1:
Different precision levels are applied to different parts of the training process. High-precision floating-point arithmetic is used during critical early training stages, while lower-precision fixed-point arithmetic is used during later stages where the model is more robust to numerical variations.
Solution Approach 2:
The numerical representation parameters are changed from floating-point format to fixed-point format with specific integer and fractional bit allocations. This parameter change optimizes the balance between data load and precision for the specific training stage being executed.
Data Source
AI summary
A non-transitory computer-readable recording medium stores therein a program for causing a computer to execute a process for, in repeatedly training a given training model, repeatedly training the training model a given number of times by using a numerical value of a floating-point number, the numerical value being a parameter of the training model or training data of the training model, or any combination thereof; and, after the training by using the numerical value of the floating-point number, repeatedly training the training model by using a numerical value of a fixed-point number corresponding to a numerical value of the floating-point number obtained by the training.