Error-Prone Memory for Stochastic Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and inference processes for artificial neural networks (ANNs) are resource-intensive due to the need for stochastic rounding and error-free memory, which limits efficiency and accuracy, especially in binary ANNs where deterministic rounding is used despite its lesser accuracy.
Innovation Solution
Utilizing error-prone memory, such as magnetic random access memory (MRAM), for training and inference by stochastically writing values, eliminating the need for generating stochastic parameter values and reducing the need for error correction, thereby allowing for more efficient storage and classification without significant increases in prediction errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If deterministic rounding is used to store binary ANN parameters in memory, then resource usage is reduced, but training accuracy deteriorates
Solution Approach 1:
The patent converts the harmful write errors in error-prone memory into a beneficial stochastic rounding mechanism. By operating the memory at elevated temperatures or with reduced write current, the inherent write failures generate stochastic variations in stored values that approximate stochastic rounding, thereby improving training accuracy without requiring additional computational resources for random number generation.
Solution Approach 2:
The patent changes the operational parameters of error-prone memory by operating it at elevated temperatures or with reduced write current. These parameter changes intentionally increase the write error rate to achieve the desired stochastic effect, transforming the memory's degradation characteristics into a functional advantage for neural network training.
2Measurement precision
If stochastic rounding is used to generate binary ANN parameters, then training accuracy is improved, but resource intensity increases
Solution Approach 1:
The patent enables the error-prone memory to serve itself by utilizing its inherent write errors as the stochastic mechanism. Instead of requiring external random number generation circuits or algorithms, the memory's own degradation characteristics provide the necessary stochasticity, eliminating the need for additional resource-intensive stochastic rounding processes.
3Measurement precision
If error-free memory is used to store ANN parameters, then prediction accuracy is maintained, but memory density and storage capacity are reduced
Solution Approach 1:
The patent changes the operational parameters of error-prone memory to intentionally increase write error rates during training, which then serve as beneficial stochastic variations. This parameter change allows the use of lower-density error-prone memory while maintaining training effectiveness, as the elevated error rates provide the necessary stochasticity that would otherwise require more sophisticated error-free memory systems.
Solution Approach 2:
The patent converts the harmful write errors that limit memory density into a beneficial feature that enables effective training with error-prone memory. By accepting and utilizing these errors as stochastic rounding, the system can use higher-density error-prone memory without sacrificing training accuracy, effectively transforming the density-limiting factor into a performance-enhancing mechanism.
4Reliability
If error correction mechanisms are implemented in memory, then data reliability is improved, but device complexity and overhead increase
Solution Approach 1:
The patent inverts the conventional approach to memory reliability. Instead of using error correction to eliminate write errors, the system intentionally operates memory in an error-prone state and utilizes the resulting write errors as beneficial stochastic variations for training. This inversion eliminates the need for complex error correction mechanisms while maintaining training effectiveness.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enhances the speed and efficiency of ANN training and inference by allowing for denser memory storage of ANN parameters with minimal impact on prediction accuracy, reducing the need for parameter busing and enabling more accurate training with higher activation error rates compared to error-free memory.
Implementation Method 1
some embodiments of the present disclosure use error-prone memory (e.g., memory prone to write errors) to train a binary artificial neural network (ANN). Because the write process in error-prone memory is itself stochastic
Implementation Method 2
storing trained values for the weights of the neural network, wherein the trained neural network is configured to classify second data based on the stored values
Data Source
AI summary
A computing device includes one or more processors, random access memory (RAM), and a non-transitory computer-readable storage medium storing instructions for execution by the one or more processors. The computing device receives first data and classifies the first data using a neural network that includes at least one quantized layer. The classifying includes reading values from the random access memory for a set of weights of the at least one quantized layer of the neural network using first read parameters corresponding to a first error rate.