Bayesian Neural Network RRAM Programming via Multi-Cell Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks face overfitting issues due to failure to capture uncertainty in model parameters, especially with scarce or noisy training data, and existing methods for training Bayesian neural networks (BNNs) in resistive Random Access Memories (RRAMs) are limited by constraints such as correlated mean and standard deviation values and dependent synaptic coefficients.
Innovation Solution
The solution involves using a combination of RRAM memristors to expand the hardware compliance domain for programming BNN parameters, allowing independent choice of mean and standard deviation values, and employing multi-dimensional expectation-maximization algorithms to capture the dependence between synaptic coefficients, thereby relaxing the constraints on mean and standard deviation correlation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single RRAM cell is used to implement a synapse in a Bayesian neural network, then the hardware implementation is simple, but the mean and standard deviation values are correlated and cannot be independently chosen
Solution Approach 1:
The patent divides the synapse implementation into multiple RRAM cells, where each cell contributes to representing different aspects of the probability distribution. By segmenting the single-cell implementation into a multi-cell architecture, the system achieves independent control over mean and standard deviation parameters while maintaining the computational functionality of the synapse.
2Reliability
If synaptic coefficients are programmed in a row-wise fashion to handle dependencies, then the independence assumption is relaxed, but the programming process becomes more complex and time-consuming
Solution Approach 1:
The patent performs preliminary computations to calculate the probability distribution parameters before the actual programming process. By pre-computing the necessary statistics and preparing the programming sequences in advance, the system reduces the time required during the actual synaptic coefficient programming phase while maintaining accurate representation of dependencies.
3Use of energy by moving object
If ex situ training is performed in the cloud and parameters are transferred to the inference engine, then the processing resources required are reduced on the implementation platform, but the computation burden shifts to the cloud and communication overhead increases
Solution Approach 1:
The patent performs partial training computations directly on the implementation platform using the multi-cell RRAM architecture, rather than completing all training in the cloud. This partial in-situ training approach reduces communication overhead by transferring only essential parameters while maintaining the ability to leverage cloud resources for computationally intensive tasks, achieving a balanced distribution of computational workload.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enables more effective training and implementation of BNNs in RRAMs by increasing the degrees of freedom for programming, allowing for independent selection of mean and standard deviation values and accounting for coefficient dependencies, thereby improving the robustness and accuracy of BNNs.
Implementation Method 1
RRAMs are nanoscale non-volatile memory devices whose conductive states can be programmed under application of voltage and current pulses
Implementation Method 2
the dot product between an input voltage vector and an array of conductance values stored in the RRAM is obtained as a sum of currents according to Kirchhoff's circuit law
Data Source
AI summary
The present invention concerns a method for programming a Bayesian neural network (BNN) in a RRAM memory. After the BNN has been trained on a dataset D, the joint posterior probability distribution of the synaptic coefficients, p(w|D) is decomposed into a mixture of multivariate mean-field Gaussian components by GMM. The weighting coefficients and the parameters of these multivariate Gaussian components are estimated by MDEM (Multi-Dimensional Expectation Maximization) with two constraints. According to the first constraint, the off-diagonal terms of the covariance matrix of each component are forced to zero. According to the second constraint, the couples of mean values and diagonal terms of the covariance matrix of each component are constrained to belong to a hardware compliance domain determined by a relationship between the conductance mean value and conductance standard deviation of a memristor programmed by a SET or RESET operation. The weighting factors and mean values of these components are then transferred to the chip implementing the BNN, and the memristors of the RRAM are programmed accordingly.