Probabilistic Binary Memory for Neural Network Synaptic Weight Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face inefficiencies in storing synaptic weights due to the use of multi-bit memory, which often settles to binary values, leading to waste of storage resources and speed bottlenecks during training and operation.
Innovation Solution
A method and apparatus that compute updates to synaptic weights according to a learning rule and switch binary states in a probabilistic manner within memory locations, reducing the number of bits per synaptic weight and power consumption by representing weights as binary states, utilizing probabilistic nonvolatile binary memory such as spin-torque transfer (STT) devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-bit memory is used to store synaptic weights, then storage precision is improved, but storage resource efficiency deteriorates because weights settle to binary values
Solution Approach 1:
The patent changes the parameter of memory bit depth from multi-bit to 1-bit, and introduces probabilistic states to represent weight values. Instead of using multiple bits to store precise weight values, the system uses probability of switching between 0 and 1 to encode weight magnitude, thereby maintaining precision while reducing storage resource consumption.
Solution Approach 2:
The patent creates a probabilistic copy of weight information where the expected value of binary states represents the original weight. By using probabilistic switching with probability p = weight + 0.5, the system encodes weight information in the statistical behavior of binary states rather than directly storing the weight value, achieving efficient storage with retrieval of precise weight information.
2Measurement precision
If multi-bit synaptic memory is used, then weight precision is improved, but operation speed deteriorates due to addressing bottlenecks
Solution Approach 1:
The patent changes the memory operation from deterministic multi-bit addressing to probabilistic 1-bit switching. This parameter change eliminates complex addressing mechanisms and enables parallel operation, as each synapse can independently switch its binary state based on probabilistic rules, thereby dramatically improving memory operation speed.
Solution Approach 2:
The patent replaces the mechanical/electronic addressing system of multi-bit memory with a probabilistic field-effect system. Instead of sequentially addressing individual memory locations through complex control signals, the system uses probabilistic switching triggered by input signals, allowing simultaneous updates across multiple synapses and eliminating addressing bottlenecks.
3Quantity of substance
If binary states are used to represent weights, then storage efficiency is improved, but representation precision deteriorates
Solution Approach 1:
The patent employs periodic probabilistic switching where the binary state oscillates between 0 and 1 with probability p determined by the weight value. Over time, the frequency of switching encodes the weight information, allowing precise weight representation through temporal averaging. This periodic probabilistic behavior enables continuous weight values to be represented using discrete binary states.
Solution Approach 2:
The patent creates a statistical copy of the weight value through probabilistic switching behavior. The expected value E[state] = p*1 + (1-p)*0 = p = weight + 0.5, which means the binary state's statistical properties encode the original weight information. This copying mechanism preserves precision while using minimal storage resources.
4Use of energy by stationary object
If probabilistic switching is implemented, then power consumption is reduced, but control complexity increases
Solution Approach 1:
The patent implements self-service probabilistic switching where the memory element's inherent physical properties (such as stochastic resonance or thermal noise) provide the randomness needed for probabilistic switching. The system leverages natural physical phenomena rather than requiring external random number generators or complex control circuits, thereby reducing power consumption while maintaining simplicity.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach allows for faster memory operations, reduced power consumption, and efficient storage of synaptic weights, as binary states can be represented with one-bit per synapse, facilitating time averaging of training events and improving memory retention and reliability.
Implementation Method 1
utilizing probabilistic nonvolatile binary memory such as spin-torque transfer (STT) devices
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Certain aspects of the present disclosure support a technique for utilizing a memory in probabilistic manner to store information about weights of synapses of a neural network.