Neural Network Training With Dynamic Precision and Topology Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and deploying neural networks on general-purpose processors is computationally expensive, making real-time feature extraction challenging, and existing hardware accelerators like FPGAs face inefficiencies due to lack of floating-point support and high computational demands.
Innovation Solution
Implementing neural networks using block floating-point formats, where multiple values share a common exponent, reducing precision during early training and increasing it based on performance metrics, and utilizing hardware accelerators like FPGAs and TPUs to perform operations efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If normal precision floating-point format is used for neural network training, then accuracy is maintained, but computational complexity and energy consumption increase
Solution Approach 1:
The patent applies dynamic precision adjustment by switching between different floating-point precision formats (full precision, reduced precision, and block floating-point) during different stages of neural network training. The system dynamically selects the precision level based on the training epoch and performance metrics, using lower precision during early training when accuracy requirements are less stringent, and transitioning to higher precision as training progresses and accuracy requirements increase.
Solution Approach 2:
The patent changes the precision parameter of floating-point representations throughout the training process. It uses full precision floating-point format initially, then transitions to reduced precision formats, and finally employs block floating-point format where multiple values share a common exponent. This parameter change allows the system to maintain accuracy when needed while reducing computational complexity and energy consumption during phases where lower precision is acceptable.
2Measurement precision
If full precision is used throughout training, then final accuracy is maintained, but training time and computational resources increase
Solution Approach 1:
The system dynamically adjusts precision levels during training based on the epoch number and performance metrics. Early training epochs use reduced precision formats to speed up computation, while later epochs transition to full precision to ensure final accuracy. This dynamic approach allows the system to achieve the same final accuracy as full-precision training but in less time.
Solution Approach 2:
The patent performs preliminary training using reduced precision formats before transitioning to full precision. This preliminary action allows the neural network to learn coarse features and patterns quickly, reducing the overall training time required to reach the same accuracy level as full-precision training would achieve.
3Device complexity
If reduced precision block floating-point format is used, then computational complexity is reduced, but precision and accuracy may deteriorate
Solution Approach 1:
The system dynamically switches between reduced precision block floating-point format and full precision format based on the training stage and performance metrics. When using block floating-point format with shared exponents to reduce computational complexity, the system monitors accuracy and transitions to full precision when the metric indicates that precision requirements are not met, thus balancing complexity reduction with precision maintenance.
Solution Approach 2:
The patent employs feedback mechanisms by monitoring performance metrics (such as accuracy or loss) during training. When the metric indicates that reduced precision is causing unacceptable degradation, the system feedbacks this information and adjusts by increasing precision or switching to full precision format, ensuring that computational complexity is reduced only when it does not compromise the required accuracy.
4Measurement precision
If precision is increased during training, then accuracy improves, but memory overhead and computational demand increase
Solution Approach 1:
The system dynamically adjusts the precision of stored values during training. Early in training, when memory overhead is a concern and accuracy requirements are less stringent, the system uses reduced precision block floating-point formats with shared exponents that require less memory. As training progresses and accuracy requirements increase, the system transitions to full precision formats, managing memory overhead by matching precision levels to the current training stage's requirements.
Data Source
AI summary
Apparatus and methods for training neural networks based on a performance metric, including adjusting numerical precision and topology as training progresses are disclosed. In some examples, block floating-point formats having relatively lower accuracy are used during early stages of training. Accuracy of the floating-point format can be increased as training progresses based on a determined performance metric. In some examples, values for the neural network are transformed to normal precision floating-point formats. The performance metric can be determined based on entropy of values for the neural network, accuracy of the neural network, or by other suitable techniques. Accelerator hardware can be used to implement certain implementations, including hardware having direct support for block floating-point formats.