Adaptive Neural Network Training Using Randomized Low-Order Bits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural network training methods face inefficiencies due to overtraining and the need for precise number representation, leading to high bandwidth and time consumption when new data is presented, especially in parallel graphics processing units with SIMT architectures.
Innovation Solution
Introduce randomness in floating-point numbers during neural network training by replacing less-significant low-order bits of operand and result values, combined with adaptive runtime layers and precision profiling to optimize training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional neural network training uses precise number representation, then training accuracy is improved, but bandwidth consumption and time increase
Solution Approach 1:
The patent changes the precision parameter of number representation during training by replacing less-significant low-order bits with random values. This parameter change allows the system to use lower precision (e.g., 8-bit or 16-bit floating point) instead of full precision (32-bit or 64-bit), thereby reducing bandwidth consumption and training time while maintaining sufficient training accuracy through the adaptive runtime layer that monitors and adjusts precision requirements.
2Measurement precision
If conventional neural network training uses precise number representation, then training accuracy is improved, but bandwidth consumption increases
Solution Approach 1:
The patent applies parameter changes by modifying the precision level of number representation from high precision (32-bit/64-bit floating point) to low precision (8-bit/16-bit floating point with truncated low-order bits). This reduction in precision parameter directly decreases the amount of data that needs to be transferred across the memory hierarchy, thereby reducing bandwidth consumption and energy loss while the adaptive runtime layer ensures training accuracy is maintained.
3Reliability
If neural network training introduces randomness in floating-point numbers, then overtraining is reduced, but computational complexity increases
Solution Approach 1:
The patent introduces randomness by changing the low-order bits parameter of floating-point numbers during computation. Instead of using full precision values, the system truncates or randomizes the less-significant bits, which acts as a form of regularization that reduces overtraining. The adaptive runtime layer monitors computational requirements and adjusts precision dynamically, managing the trade-off between randomness introduction and computational complexity.
4Productivity
If adaptive runtime layers with precision profiling are used, then training efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements self-service through adaptive runtime layers that automatically profile precision requirements and adjust computational precision without external intervention. The system monitors training progress, identifies which operations require high precision and which can tolerate lower precision, and dynamically adjusts the precision parameter accordingly. This self-managing approach improves training efficiency by reducing unnecessary precision computations while the system handles its own complexity management through automated profiling and adaptation.
Data Source
AI summary
A mechanism is described for facilitating efficient training of neural networks at computing devices. A method of embodiments, as described herein, includes detecting one or more inputs for training of a neural network, and introducing randomness in floating point (FP) numbers to prevent overtraining of the neural network, where introducing randomness includes replacing less-significant low-order bits of operand and result values with new low-order bits during the training of the neural network.