Neural Network Data Format Prediction via Gradient Simulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training artificial neural networks are inefficient due to high computational requirements and power consumption, especially for large-scale models, where there is no effective way to predict training performance in advance, leading to increased time and cost as the network size and data volume grow.
Innovation Solution
An artificial neural network performance prediction method and device that determines an optimal data format by simulating the network with both original and candidate data formats, calculating parameter gradients, and assessing performance indicators to identify suitable low-precision data formats for efficient training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If low-precision data format is applied to train artificial neural networks, then power consumption and calculation requirements are reduced, but training performance cannot be predicted in advance requiring full training to evaluate
Solution Approach 1:
The patent performs preliminary simulation and performance prediction before actual training to evaluate whether low-precision data format is suitable. By calculating parameter gradients through simulation and comparing performance indicators, the system determines data format suitability in advance, avoiding the need to complete full training evaluation and thus reducing time loss.
2Measurement precision
If full training is performed to compare performance of different data formats, then accurate performance comparison is achieved, but time and power costs rapidly increase with larger neural networks and more data
Solution Approach 1:
The patent creates a simulated copy of the training process to evaluate performance without executing full training. By simulating forward and backward propagation to obtain parameter gradients and calculating performance indicators, the system achieves performance comparison accuracy while avoiding the time and power costs of actual full training.
Solution Approach 2:
Instead of performing complete training, the patent executes partial simulation actions - running simulation for a limited number of steps to obtain parameter gradients and performance indicators. This partial action is sufficient for performance comparison and data format selection, significantly reducing evaluation time and power consumption.
3Reliability
If multiple data formats are tested through actual training to find optimal format, then best performance is achieved, but the process becomes impractical for large-scale networks due to excessive time and computational resources
Solution Approach 1:
The patent performs preliminary simulation-based performance prediction for multiple data formats before actual training. By evaluating performance indicators through simulation, the system identifies suitable data formats in advance, ensuring reliable performance selection while dramatically improving efficiency compared to testing multiple formats through full actual training.
Data Source
AI summary
Artificial neural network performance prediction method and device according to data format are proposed. The artificial neural network performance prediction method may comprise: determining a zone and an operand of an artificial neural network that uses a candidate data format; obtaining a first parameter gradient through a first simulation of the artificial neural network on input data by applying an original data format to the operand in the zone; obtaining a second parameter gradient through a second simulation of the artificial neural network on the input data by applying the candidate data format to the operand in the zone; and determining a performance indicator according to the candidate data format based on the first parameter gradient and the second parameter gradient. Therefore, it is possible to find a low-precision data format suitable for a neural network to be trained and to perform low-precision training with high performance.