Stochastic Memory Write for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training and inference processes for artificial neural networks (ANNs) are resource-intensive due to the need for stochastic rounding and error-free memory, which limits their efficiency and accuracy, especially in binary ANNs.
Innovation Solution
Utilizing error-prone memory, such as magnetic random access memory (MRAM), for training and inference by stochastically writing values, eliminating the need for generating stochastic parameter values and reducing the need for error correction, thereby allowing for more efficient storage and classification without significant increases in prediction errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If deterministic rounding is used to store binary ANN parameters in memory, then resource usage (energy and computation) is reduced, but training accuracy deteriorates compared to stochastic rounding
Solution Approach 1:
The patent converts the harmful write errors in error-prone memory into beneficial stochastic rounding effects. By intentionally operating memory at controlled error rates, the previously harmful random write failures become a useful mechanism for generating stochastic rounded values, thereby improving training accuracy while using simpler, more energy-efficient memory hardware.
Solution Approach 2:
The patent changes the operational parameters of error-prone memory by controlling write error rates to specific ranges (e.g., 0.1% to 10%). This parameter adjustment transforms the memory from an unreliable storage device into a controlled stochastic rounding mechanism, enabling accurate binary ANN training with reduced energy consumption.
2Reliability
If error-free memory is used to store ANN parameters, then reliability of parameter storage is improved, but memory density and storage capacity deteriorate due to required redundancy
Solution Approach 1:
The patent inverts the traditional approach by not trying to eliminate errors through redundancy, but rather by controlling and utilizing errors as a functional mechanism. Instead of adding redundancy to achieve reliability, the system accepts controlled error rates and uses them for stochastic rounding, thereby achieving both high density and functional reliability.
Solution Approach 2:
The patent converts the harmful aspect of error-prone memory (write errors) into a beneficial feature for stochastic rounding. This eliminates the need for error correction redundancy while maintaining or improving effective reliability for training purposes, thereby increasing memory density.
3Measurement precision
If stochastic rounding is performed by generating stochastic values before writing to memory, then training accuracy is improved, but resource intensity (computation and energy) increases
Solution Approach 1:
The patent enables the memory system to perform stochastic rounding autonomously during the write operation itself, without requiring external stochastic value generation. The memory's inherent write variability serves the dual purpose of storing values and performing stochastic rounding, eliminating the need for separate random number generation hardware or software.
Solution Approach 2:
The patent extracts the stochastic rounding function from the computational domain and transfers it to the memory domain. By generating stochastic effects directly during the memory write process rather than before it, the system eliminates the computational overhead of stochastic value generation while maintaining training accuracy.
4Measurement precision
If conventional memory with error correction is used for ANN inference, then prediction accuracy is maintained, but inference speed and efficiency deteriorate due to error correction overhead
Solution Approach 1:
The patent converts write errors into beneficial stochastic effects that improve generalization during training. This approach reduces the need for aggressive error correction during inference, as the controlled error model from training makes the network robust to memory errors, thereby improving inference speed without sacrificing accuracy.
5Quantity of substance
If dense memory storage is implemented in error-prone memory, then memory capacity is increased, but the need for redundancy for error correction increases
Solution Approach 1:
The patent converts the harmful write errors into a useful stochastic rounding mechanism. By controlling error rates and using them intentionally, the system eliminates the need for traditional error correction redundancy, thereby achieving high memory capacity without the overhead of redundancy bits or error correction codes.
Data Source
AI summary
A computing device includes one or more processors, random access memory (RAM), and a non-transitory computer-readable storage medium storing instructions for execution by the one or more processors. The computing device receives first data on which to train a neural network comprising at least one quantized layer and performs a set of training iterations to train weights for the neural network. Each training iteration of the set of training iterations includes stochastically writing values to the random access memory for a set of activations of the at least one quantized layer of the neural network using first write parameters corresponding to a first write error rate. The computing device stores trained values for the weights of the neural network. The trained neural network is configured to classify second data based on the stored values.