Parallel RL Control Agent Updates for Existing Machine Controllers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine control devices, such as industrial PCs, lack sufficient computing resources to efficiently perform reinforcement learning, which is necessary for training neural networks, making it difficult to implement deep reinforcement learning for real-world machine control without replacing the existing devices.
Innovation Solution
A machine control system comprising a first data processing device for selecting control actions and a second data processing device with graphics or tensor processing units for updating a control agent through reinforcement learning, allowing the distribution of RL tasks between devices without requiring changes to the existing control device, enabling continuous improvement of control processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning is implemented using existing machine control devices, then the control device can be used without replacement, but the computing resources are insufficient to perform deep reinforcement learning within an acceptable period of time
Solution Approach 1:
The system divides the reinforcement learning task into two separate components: a control agent running on the existing machine control device for real-time control decisions, and a training agent running on external computing resources for model training. This segmentation allows each component to operate on devices optimized for its specific function, resolving the contradiction between using existing devices and achieving sufficient training speed.
Solution Approach 2:
A communication interface acts as an intermediary between the control device and training device, enabling data exchange and coordination. The training device receives state information from the control device, processes training updates, and sends updated control policies back, allowing the system to leverage external computing resources without replacing the existing control device.
2Productivity
If machine control devices are replaced with devices having sufficient computing power, then reinforcement learning can be performed efficiently, but costs increase due to existing device investments
Solution Approach 1:
The system makes external computing resources accessible to multiple machines through a networked architecture. A single powerful training device can serve multiple control devices, amortizing the high cost across several systems and making deep reinforcement learning economically viable without requiring each machine to have expensive dedicated hardware.
3Reliability
If deep reinforcement learning is implemented, then control performance can be continuously improved, but comprehensive computing resources are required that are not provided by typical machine control devices
Solution Approach 1:
The system moves the training computation from the local control device dimension to a networked computing dimension. By leveraging external GPUs, TPUs, or cloud-based computing resources, the system accesses comprehensive computing power without increasing the complexity or resource requirements of the local machine control devices.
Data Source
AI summary
A machine control system including a first data processing device having a first central processing unit, which implements a control agent, the first data processing unit communicating control commands to the machine and receiving process and environmental data from the machine, which contain state information about the state of the machine and/or an environment of the machine. A second data processing device receives the state information from the first data processing device and implements a copy of the control agent. The second data processing device ascertains, with the aid of one or multiple graphics processing units and/or one or multiple tensor processing units, an update of the copy of the control agent by reinforcement learning, using the state information, and communicates the update to the first data processing device.