RRAM Bayesian Neural Network Accelerator for Posterior Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural networks face challenges in generalizing well from training data due to uncertainty in synaptic coefficients, particularly with scarce or noisy data, leading to overfitting, and existing hardware accelerators are limited in handling arbitrary posterior probability distributions and require volatile memories with high data transfer rates.
Innovation Solution
A Bayesian neural network (BNN) using resistive memory (RRAM) as a hardware accelerator, where synaptic coefficients are represented by a Gaussian mixture model (GMM) and programmed using RRAM's cycle-to-cycle variability to generate arbitrary posterior probability distributions, reducing the need for volatile memories and high data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional neural networks are used with fixed synaptic coefficients, then the network structure is simple and fast to compute, but the network overfits training data and fails to generalize well, especially with scarce or noisy data
Solution Approach 1:
The patent transforms fixed synaptic coefficients into random variables following posterior probability distributions. Each weight is no longer a single value but a distribution characterized by mean and variance, allowing the network to capture uncertainty and improve generalization while maintaining computational tractability through probabilistic modeling
2Reliability
If Bayesian neural networks with arbitrary posterior probability distributions are implemented, then modeling accuracy and robustness improve, but hardware acceleration becomes limited and requires volatile memories with high data transfer rates
Solution Approach 1:
The patent replaces conventional volatile memory-based hardware accelerators with resistive memory (RRAM) devices. RRAM's inherent analog conductance values and cycle-to-cycle variability are exploited to naturally represent and sample from arbitrary posterior probability distributions, eliminating the need for complex volatile memory systems and high-speed data transfer mechanisms
Solution Approach 2:
The patent leverages the intrinsic cycle-to-cycle variability of RRAM devices as a natural sampling mechanism. Instead of requiring external control systems to generate random samples, the RRAM devices themselves provide stochastic behavior through their physical characteristics, enabling Bayesian inference with minimal additional hardware complexity
3Use of energy by moving object
If RRAM devices are used for dot product computation, then energy consumption and area are reduced, but device-to-device dispersion and cycle-to-cycle conductance dispersion occur
Solution Approach 1:
The patent converts the harmful effect of RRAM variability into a beneficial feature. The device-to-device and cycle-to-cycle conductance dispersion, initially seen as noise and precision errors, are harnessed as natural sampling mechanisms for posterior probability distributions. This variability enables stochastic sampling required for Bayesian inference without additional random number generation hardware
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The solution enables efficient modeling of arbitrary posterior probability distributions, reducing overfitting and resource requirements, allowing BNNs to operate effectively on edge devices with reduced energy and area consumption.
Implementation Method 1
RRAMs generally suffer from device-to-device dispersion and cycle-to-cycle conductance dispersion which are mitigated in order to ensure reliable training and prediction
Implementation Method 2
In such a memory, the dot product between an input voltage vector and an array of conductance values stored in the RRAM is obtained as a sum of currents according to Kirchhoff's circuit law
Data Source
Figure 1~2
Figure 3~4
Figure 5A~5B
AI summary
The present invention concerns a Bayesian neural network (BNN) comprising an input layer (721), and, an output layer (723), and, possibly, one or more hidden layer(s) (722). Each neuron of a layer is connected at its input with a plurality of synapses, the synapses of said plurality being implemented as a RRAM array (711) constituted of cells, each column of the array being associated with a synapse and each row of the array being associated with an instance of the set of synaptic coefficients, the cells of a row of the RRAM being programmed during a SET operation with respective programming current intensities, the programming intensity of a cell being derived from the median value of a Gaussian component obtained by GMM decomposition into K Gaussian components of the marginal posterior probability of the corresponding synaptic coefficient, once the BNN model has been trained on a training dataset. The present invention also concerns a method for programming such a Bayesian neural network after the BNN model has been trained on a training dataset.