RPU Array Noise Management via Pre-Post Normalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural network (ANN) training with analog resistive processing unit (RPU) arrays is hindered by analog noise, which buries small output signals and limits accuracy due to noise and signal bound limitations.
Innovation Solution
The implementation of pre- and post-normalization techniques using mean and variance normalization across multiple input and output vectors, updating these values with a discounting factor to reduce noise impact during ANN training, thereby enhancing signal-to-noise ratio (SNR) and improving training accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If analog vector-matrix multiplication operations are performed on RPU array, then computation speed is improved, but noise interference increases and buries small output signals
Solution Approach 1:
The patent applies pre-normalization to input vectors before they enter the RPU array computation. By computing mean and standard deviation from historical data and applying normalization factors in advance, the input signals are scaled to optimal ranges that prevent them from being buried by noise during analog multiplication operations
Solution Approach 2:
The patent implements post-normalization that uses feedback from observed output signals to adjust normalization parameters. By computing mean and standard deviation from actual output vectors and using these to normalize subsequent inputs, the system continuously adapts to noise characteristics and maintains optimal signal-to-noise ratio
2Use of energy by moving object
If small input values are used in backward cycle pass, then energy consumption is reduced, but output signal becomes buried by noise
Solution Approach 1:
The patent dynamically changes the normalization parameters (mean and standard deviation) based on the computational phase. During backward cycle pass, different normalization factors are applied compared to forward pass, allowing small input values to be scaled appropriately so their output signals remain above the noise floor while maintaining low energy consumption
3Measurement precision
If weight values are updated frequently during training, then learning accuracy is improved, but noise impact increases due to bounded weight ranges
Solution Approach 1:
The patent pre-computes and stores normalization factors (mean and standard deviation) before weight updates occur. By having these normalization parameters ready in advance, the system can apply them during weight update operations to scale the updates appropriately, ensuring that frequent weight updates improve learning accuracy without amplifying noise impact from bounded weight ranges
Data Source
AI summary
Advanced noise and signal management techniques for RPU arrays during ANN training are provided. In one aspect of the invention, a method for ANN training includes: providing an array of RPU devices with pre-normalizers and post-normalizers; computing and pre-normalizing a mean and standard deviation of all elements of an input vector x to the array that belong to the set group of each of the pre-normalizers; and computing and post-normalizing the mean μ and the standard deviation σ of all elements of an output vector y that belong to the set group of each of the post-normalizers.