Reinforcement Learning Control with Limited Action Ranges
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning methods often result in inappropriate actions that adversely affect control targets due to randomly changed actions, leading to suboptimal performance and potential damage, such as overheating servers or unstable drone flights.
Innovation Solution
A reinforcement learning device that performs reinforcement learning within a limited action range around a basic controller policy, updates to a new policy, and repeats this process to avoid extreme deviations, ensuring more appropriate actions are taken, thereby reducing adverse effects on control targets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning performs random actions on control target, then policy optimization is achieved, but inappropriate actions causing adverse effects occur
Solution Approach 1:
The patent applies preliminary action by establishing a safety constraint mechanism before reinforcement learning actions are executed. The limit action range is predefined based on safety considerations, and the processor checks whether each learned action falls within this range before applying it to the control target. This prevents harmful actions from being executed while still allowing policy optimization within safe boundaries.
Solution Approach 2:
The patent introduces an intermediary mechanism (the limit action range constraint) between the reinforcement learning policy and the control target. This intermediary layer filters out inappropriate actions by comparing learned actions against predefined safety boundaries, allowing only acceptable actions to reach the control target while blocking potentially harmful ones.
2Object-affected harmful factors
If action range is limited for safety, then adverse effects are reduced, but policy optimization effectiveness decreases
Solution Approach 1:
The patent applies dynamics by making the action range adaptive rather than fixed. The limit action range is dynamically adjusted based on the current state of the control target and the progress of policy optimization. As the policy improves and becomes more reliable, the effective action range can be expanded, allowing continued optimization while maintaining safety.
Solution Approach 2:
The patent changes parameters by adjusting the limit action range boundaries based on the control target's state and optimization progress. The processor modifies the acceptable action thresholds dynamically, allowing larger action ranges when the system is stable and the policy is reliable, while tightening constraints when risks are higher or optimization is still in progress.
Data Source
AI summary
A reinforcement learning device includes a processor that determines a first action on a control target by using a basic controller that defines an action on the control target depending on a state of the control target. The processor performs a first reinforcement learning within a first action range around the first action in order to acquire a first policy for determining an action on the control target depending on a state of the control target. The first action range is smaller than a limit action range for the control target. The processor determines a second action on the control target by using the first policy. The processor updates the first policy to a second policy by performing a second reinforcement learning within a second action range around the second action. The second action range is smaller than the limit action range.