TD-Lambda Synapse Circuit Using Resistive Memory Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a technical difficulty in implementing TD-lambda learning with a neural network approximating the value function in a device in a simple and cost-effective manner.
Innovation Solution
A synapse circuit for a neural network is designed using a first and second resistive memory device, along with a synapse control circuit. The synapse control circuit updates synaptic weights by programming the resistive state of the first memory device based on the conductance of the second memory device, which decays over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If TD-lambda learning is implemented with a neural network in a conventional manner, then learning effectiveness is maintained, but device complexity, surface area, cost, and power consumption increase
Solution Approach 1:
The synapse circuit is segmented into two distinct resistive memory devices: a first resistive memory device dedicated to storing synaptic weights and a second resistive memory device dedicated to storing eligibility traces. This segmentation allows each device to be optimized for its specific function, reducing overall device complexity while maintaining learning effectiveness through specialized storage mechanisms.
Solution Approach 2:
The synapse control circuit serves multiple functions: it updates both the synaptic weights in the first resistive memory device and the eligibility traces in the second resistive memory device, and it performs the temporal difference learning computations. This multi-functionality reduces the need for separate dedicated circuits, thereby reducing device complexity and surface area while maintaining full learning capability.
2Area of stationary object
If TD-lambda learning is implemented with a neural network using conventional structures, then learning capability is preserved, but surface area and cost increase
Solution Approach 1:
The synapse control circuit merges the functionality of weight updates and eligibility trace updates into a single integrated control unit. This merging reduces the overall surface area required by eliminating redundant control circuits while maintaining the full TD-lambda learning capability. The unified control approach also simplifies the manufacturing process by reducing the number of discrete components that need to be assembled.
3Use of energy by moving object
If TD-lambda learning is implemented with a neural network in a detailed manner, then learning precision is maintained, but power consumption increases
Solution Approach 1:
The second resistive memory device is configured to have a conductance that decays over time, dynamically maintaining the eligibility traces without requiring continuous power input for active maintenance. This dynamic approach allows the system to preserve learning precision by maintaining trace information while significantly reducing power consumption compared to active refresh mechanisms.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This implementation allows for efficient and cost-effective TD-lambda temporal difference learning, reducing complexity, surface area, cost, and power consumption while maintaining learning effectiveness.
Implementation Method 1
the second resistive memory device is configured to have a conductance that decays over time
Implementation Method 2
programming a resistive state of the first resistive memory device based on a programmed conductance of the second resistive memory device
Implementation Method 3
the synapse control circuit is configured to update the synaptic weight by applying a voltage or current level generated based on a temporal difference error to an electrode of the second resistive memory device to generate an output current or voltage level
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
The present disclosure relates to a synapse circuit of a neural network for performing TD-lambda temporal difference learning, the neural network approximating a value function, the synapse circuit comprising: a first resistive memory device (506); a second resistive memory device (516); and a synapse control circuit (528) configured to update a synaptic weight (gθ) of the synapse circuit by programming a resistive state of the first resistive memory device (506) based on a programmed conductance of the second resistive memory device (516).