Reinforcement Learning Control with Risk-Based Input Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning methods do not adequately consider constraint conditions during policy learning, leading to an increased probability of state violations in controlled objects, which can adversely affect real-world applications.
Innovation Solution
A reinforcement learning method that calculates a degree of risk for the current state of a controlled object based on predicted future values, adjusting the control input range to prevent constraint violations and maintain learning efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional reinforcement learning methods are used to learn policy without considering constraint conditions, then learning efficiency is improved, but the probability of constraint violations increases
Solution Approach 1:
The patent calculates the degree of risk for constraint violations in advance before executing control actions. By predicting future state values and assessing risk levels beforehand, the system can adjust the control input range proactively to prevent constraint violations while maintaining learning efficiency.
Solution Approach 2:
The patent dynamically adjusts the control input range based on the calculated degree of risk. When risk is high, the range is narrowed to prevent violations; when risk is low, the range can be expanded to improve learning efficiency. This dynamic adjustment resolves the contradiction between safety and efficiency.
2Reliability
If the control input range is narrowed to prevent constraint violations, then reliability is improved, but learning efficiency deteriorates
Solution Approach 1:
The control input range is not fixed but dynamically adjusted based on real-time risk assessment. The system calculates the degree of risk for each state and adjusts the range accordingly, allowing wide ranges when safe and narrow ranges when risk is present, thus maintaining both reliability and learning efficiency.
Solution Approach 2:
The patent changes the parameter of control input range based on the calculated degree of risk. By modifying this parameter dynamically rather than keeping it fixed, the system achieves both constraint satisfaction and maintained learning efficiency through adaptive parameter adjustment.
3Productivity
If the control input range is expanded to improve learning efficiency, then productivity is improved, but the probability of constraint violations increases
Solution Approach 1:
Before expanding the control input range to improve learning, the system performs preliminary risk assessment by calculating future state values. This preliminary action identifies safe regions where expansion is permissible, allowing efficiency improvement without increasing violation probability.
Solution Approach 2:
The system dynamically determines the optimal control input range for each state based on risk calculation. Rather than using a fixed expanded range, the range adapts to the specific state's risk level, enabling efficiency gains in safe states while maintaining safety in risky states.
Data Source
AI summary
A reinforcement learning method executed by a computer includes calculating a degree of risk for a state of a controlled object at a current time point with respect to a constraint condition related to the state of the controlled object, the degree of risk being calculated based on a predicted value of the state of the controlled object at a future time point, the predicted value being obtained from model information defining a relationship between the state of the controlled object and a control input to the controlled object; and determining the control input to the controlled object at the current time point, from a range defined according to the calculated degree of risk so that the range becomes narrower as the calculated degree of risk increases.