Reinforcement Learning Agent Action Selection via State Transition Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In reinforcement learning, agents face difficulties in efficiently learning unknown environments due to the need to perform random actions to reach target states, especially in complex environments with obstacles like narrow passages or environments with gravity, leading to inefficient learning and biased action selection.
Innovation Solution
A data processing device and method that calculates state values and action values based on a state transition model, allowing the agent to select actions that move towards a target state with high unevenness in action values, thereby efficiently exploring and learning the environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the agent performs randomly selected actions to reach the target state, then the agent can explore the environment, but the learning efficiency is low and the agent cannot pass through narrow passages or move against gravity
Solution Approach 1:
The patent transforms the action selection strategy from random selection to value-based selection using calculated action values. By changing the parameter of action selection from randomness to determinism based on calculated values, the agent can efficiently navigate complex environments including narrow passages and gravity-affected areas while maintaining exploration capability through targeted action value calculation for under-explored states
2Reliability
If the agent uses reinforcement learning to calculate action values after reaching the target state, then the agent can control actions based on calculated values, but the agent must perform random actions until the target state is reached
Solution Approach 1:
The patent performs preliminary action value calculation for all possible actions in the current state before executing any action. By calculating action values in advance using the state transition model and state values, the agent determines the optimal action to take, eliminating the need for random trial-and-error actions and directly transitioning to value-based action selection
3Adaptability or versatility
If the agent performs random actions in environments with gravity or narrow passages, then the agent can attempt to explore, but the agent cannot successfully pass through narrow passages or move to upper areas
Solution Approach 1:
The patent changes the action selection parameter from random selection to value-based selection by calculating action values that reflect the probability of reaching high-value states. This parameter change enables the agent to accurately select actions that lead to successful navigation through narrow passages and movement against gravity, while still exploring under-explored regions of the environment
Data Source
AI summary
A data processing device includes a state value calculation unit which calculates a state value of which the value increases as much as a state with a high transition probability for each state of the state transition model, an action value calculation unit which calculates an action value, of which the value increases as a transition probability increases for each state of the state transition model and each action that the agent can perform, a target state setting unit which sets a state with great unevenness in the action value among states of the state transition model to a target state that is the target to reach by action performed by the agent, and an action selection unit which selects an action of the agent so as to move toward the target state.