Memristor Crossbar Array for Edge Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning systems face challenges in efficiently processing time-series data due to high memory and calculation requirements, making it difficult to implement on edge devices with limited resources, and existing techniques for neural network learning with memristors are not directly applicable to reinforcement learning.
Innovation Solution
A reinforcement learning system utilizing a crossbar memristor array with voltage application units and storage units to store action decisions and state transitions, allowing for collective manipulation of memristor ohmic values to optimize action selection and learning based on reward history.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning is implemented using traditional computing systems, then learning accuracy and decision-making capability are improved, but memory consumption and calculation requirements increase significantly
Solution Approach 1:
The patent replaces traditional electronic computing systems with a memristor-based neural network system. Memristors inherently store weight values as resistance states, eliminating the need for separate memory and computation units. This substitution reduces memory consumption while maintaining learning accuracy through parallel analog computation across the memristor array.
Solution Approach 2:
The patent merges memory and computation functions into a single integrated system using crossbar memristor arrays. The same hardware structure performs both storage of weight parameters and computation of neural network operations, eliminating the memory bottleneck that limits traditional reinforcement learning systems on edge devices.
2Reliability
If reinforcement learning is implemented using traditional computing systems, then decision-making capability is improved, but calculation amount and processing time increase
Solution Approach 1:
The patent substitutes sequential digital computation with parallel analog computation using memristor crossbar arrays. Multiple neural network operations are performed simultaneously through electrical signal propagation across the array, dramatically improving calculation efficiency while preserving decision-making capability.
Solution Approach 2:
The patent employs periodic voltage signal application to memristor arrays to perform repeated reinforcement learning iterations. By applying voltage sequences in periodic cycles, the system efficiently updates weight values through spike-timing-dependent plasticity (STDP) mechanisms, achieving high-speed learning suitable for real-time edge device deployment.
3Use of energy by moving object
If memristor-based neural network learning is used, then power consumption is reduced, but the technique cannot handle time-series data required for reinforcement learning
Solution Approach 1:
The patent implements preliminary action by storing temporal information about past states and actions in the trace storage unit before performing learning updates. This preliminary preparation of historical data enables the system to process time-series reinforcement learning problems while maintaining the low power consumption benefits of memristor-based computation.
Solution Approach 2:
The patent introduces an intermediary control unit that manages the interaction between the memristor array and the reinforcement learning algorithm. This intermediary coordinates voltage signal application, handles reward signal integration, and manages trace storage operations, enabling the system to process time-series data while preserving energy efficiency.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The system reduces memory and calculation needs, enabling efficient reinforcement learning on edge devices by collectively changing memristor values through voltage signals, improving the probability of reaching desired states and reducing power consumption.
Implementation Method 1
a crossbar memristor array in which a plurality of first direction lines and a plurality of second direction lines are arranged in an intersectional manner, and memristors are provided at respective intersections
Implementation Method 2
The first voltage application unit is configured to individually apply voltage to the plurality of first direction lines. The second voltage application unit is configured to individually apply voltage to the plurality of second direction lines
Data Source
AI summary
According to an embodiment, a reinforcement learning system includes a memristor array in which each of a plurality of first direction lines corresponds to one of a plurality of states, and each of a plurality of second direction lines corresponds to one of a plurality of actions, a first voltage application unit that individually applies voltage to the first direction lines, a second voltage application unit that individually applies voltage to the second direction lines, a action decision circuit that decides action to be selected by an agent in a state corresponding to a first direction line to which a readout voltage is applied, a action storage unit that stores action selected by the agent in each state that can be caused in an environment, and a trace storage unit that stores a time at which the state is caused by action selected by the agent.