Predictive Reinforcement Learning Control for Look-Ahead Target Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep reinforcement learning in control systems often results in a 'black box' control rule, making it difficult to determine the correctness of the obtained control rule, especially when applied to industrial processes with aging, non-linearity, and unexpected characteristics, as it requires extensive training and may fail to converge, leading to long simulation times and uncertainty in control rule accuracy.
Innovation Solution
A control device that includes a target value look-ahead unit, a look-ahead target value deviation calculating unit, an adjusted target value deviation calculating unit, and an operation amount calculation unit, which performs reinforcement learning based on adjusted target value deviations to calculate an updated operation amount, ensuring flexible control while accommodating changes in the controlled object's characteristics by using a response model and prior training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If deep reinforcement learning is applied to control of a controlled object, then flexible control can be realized that accommodates aging, non-linearity, and unexpected characteristics, but the control rule becomes a black box making it difficult to judge correctness
Solution Approach 1:
The patent introduces a controlled object model as an intermediary between the deep reinforcement learning control rule and the actual controlled object. The control rule acts on the model, and the model's output is compared with actual measurements to verify correctness. This mediator enables transparency verification without changing the black-box nature of the learned control rule itself.
Solution Approach 2:
The patent creates a copy of the controlled object in the form of a controlled object model that replicates the object's characteristics including aging, non-linearity, and unexpected behaviors. This copy is used for simulation and verification, allowing the control rule to be tested against a realistic representation without risking actual controlled object damage.
2Adaptability or versatility
If deep reinforcement learning is applied to control, then flexible control accommodating aging and non-linearity is achieved, but enormous learning time is required or convergence may fail
Solution Approach 1:
The patent performs preliminary training of the deep reinforcement learning control rule using a controlled object model before applying it to the actual controlled object. This preliminary action includes pre-training with simulated data that reflects aging and non-linear characteristics, so that when the control rule is deployed to the real object, minimal additional learning is needed and convergence is achieved quickly.
Solution Approach 2:
The patent uses simulated data that excessively covers various operating conditions, aging states, and disturbance scenarios in the training phase. By providing more training data than strictly necessary (excessive action) that encompasses all possible situations, the control rule learns robust behaviors that generalize well to real-world conditions, reducing the need for extensive real-world tuning.
3Reliability
If training is repeated until temperature distribution becomes normal in air conditioning simulation, then control convergence is achieved, but the simulation takes an extremely long time
Solution Approach 1:
The patent creates a simplified copy of the air conditioning system's thermal dynamics in the controlled object model. This model replicates the essential thermal diffusion and temperature distribution characteristics but with reduced computational complexity, allowing convergence verification to be performed much faster than full-scale simulations while still ensuring control reliability.
Solution Approach 2:
The patent uses computationally inexpensive simulated training data that can be generated quickly and discarded after use, replacing the need for repeated long-duration physical simulations. The simulated data serves as a disposable, fast alternative for training and verification, enabling rapid iteration without the time cost of physical system simulations.
Data Source
AI summary
There is provision of a control device for outputting an operation amount of a controlled object so as to cause a process value of the controlled object to track a target value. The control device acquires a look-ahead target value within a time series of target values; calculates a look-ahead target value deviation which is a difference between the look-ahead target value and a current process value of the controlled object; calculates an adjusted target value deviation, by calculating a difference between the look-ahead target value and a predicted value of the process value after a look-ahead time length, based on a response model of the controlled object and past change amounts of the operation amount; performs reinforcement learning based on the adjusted target value deviation; and calculates an updated operation amount based on the adjusted target value deviation.