Dual-Network Reinforcement Learning for Stable Sensory Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing reinforcement learning techniques face challenges in effectively learning from high-dimensional sensory inputs, particularly with neural networks, due to issues like divergence, high computational cost, and inefficiency in handling large data sets, especially when applied to tasks like visual inputs and sound data.
Innovation Solution
The use of two neural networks, where one generates target action-value parameters and the other is updated based on the target, with periodic regeneration of the first network from the second, allows for efficient training and learning from large data sets, including sensory data like images and sound, enabling online operation and feature discrimination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a single neural network is updated continuously based on its own predictions in Q-learning, then the learning process can adapt quickly to new information, but the system diverges and becomes unstable
Solution Approach 1:
The single neural network is segmented into two separate networks: a primary network for generating predictions and a target network for providing stable target values. This segmentation allows the system to maintain both quick adaptation through continuous primary network updates and stability through the fixed target network, resolving the contradiction between learning speed and stability.
Solution Approach 2:
A target network acts as an intermediary between the primary network and the loss calculation. Instead of the primary network directly comparing its predictions to targets generated by itself (causing divergence), the target network provides stable reference values, mediating the learning process to prevent oscillation while maintaining learning efficiency.
2Adaptability or versatility
If reinforcement learning is applied directly to high-dimensional sensory inputs like images and sound, then the system can learn features directly from raw data, but the computational cost becomes prohibitive and convergence fails
Solution Approach 1:
The neural network architecture is segmented into distinct functional components: convolutional layers for local feature extraction from sensory inputs, pooling layers for dimensionality reduction, and fully connected layers for action-value computation. This segmentation enables the system to handle high-dimensional inputs by processing them in manageable stages, reducing overall computational complexity while maintaining the ability to learn directly from raw sensory data.
Solution Approach 2:
The system transforms high-dimensional sensory inputs (images, sound) into lower-dimensional feature representations through convolutional and pooling operations before processing. This dimensional transformation reduces the computational burden on subsequent layers while preserving the essential features needed for reinforcement learning, enabling direct learning from sensory inputs without prohibitive computational costs.
3Productivity
If experience data is stored and reused for training the neural network, then learning efficiency improves, but the computational cost per update increases proportionally with the data set size
Solution Approach 1:
The system extracts and stores only the essential elements of experience data (state, action, reward, next state) in a compact replay buffer, separating the critical learning information from redundant data. This extraction allows efficient sampling and reuse of experience tuples for training, improving learning efficiency while keeping the storage and processing requirements manageable.
Solution Approach 2:
Instead of processing the entire experience data set at once, the system uses stochastic sampling to select small batches of experience tuples for each training update. This partial processing approach maintains learning efficiency by continuously utilizing stored experience while keeping the computational cost per update low and constant, regardless of the total data set size.
Data Source
Figure 1
Figure 2
Figure 3a~3b
AI summary
We describe a method of reinforcement learning for a subject system having multiple states and actions to move from one state to the next. Training data is generated by operating on the system with a succession of actions and used to train a second neural network. Target values for training the second neural network are derived from a first neural network which is generated by copying weights of the second neural network at intervals.