Reinforcement Learning Policy Function Approximator for Concurrent Actions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning agents face challenges in effectively training to perform multiple concurrent actions in complex virtual environments, such as video games, due to high computational resource requirements and limited ability to mimic human-like behavior in partially observable states.
Innovation Solution
A method and system for training reinforcement learning agents using a policy function approximator that calculates estimated action probabilities, updates these probabilities based on rewards for concurrently performed actions, and incorporates a neural network architecture for improved performance and human-like behavior in environments with many discrete actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning agents train to perform multiple concurrent actions in complex virtual environments, then the agent's ability to perform human-like behavior and handle partially observable states improves, but the computational resource requirements increase significantly
Solution Approach 1:
The patent segments the action selection process by introducing a gating network that divides the control of multiple actions into separate streams. Each action has its own probability estimation pathway, allowing the system to handle multiple concurrent actions independently rather than as a monolithic decision process, thus reducing computational complexity while maintaining the ability to perform human-like multi-action behavior
Solution Approach 2:
The patent adds a temporal dimension to action selection by allowing actions to be selected at different time steps within the same state. The gating network introduces a time-based mechanism where different actions can be activated at different moments, enabling concurrent action performance without requiring the system to evaluate all possible action combinations simultaneously, thereby reducing computational resource requirements
2Adaptability or versatility
If reinforcement learning agents use traditional training methods to learn multiple discrete actions, then the training process is simpler, but the agents fail to effectively mimic human-like behavior in partially observable states
Solution Approach 1:
The patent introduces a gating network as an intermediary component between the state observation and the action selection process. This gating mechanism acts as a mediator that selectively activates different action pathways based on the current state, enabling the agent to handle partially observable states more effectively by filtering and routing information appropriately, thus improving human-like behavior mimicry without excessively increasing overall system complexity
Solution Approach 2:
The patent makes the policy function approximator dynamic by allowing its structure and active components to change based on the current state. The gating network dynamically activates or deactivates specific action pathways depending on the observed state, enabling the agent to adapt its complexity to the situation. This dynamic approach allows the system to maintain simplicity when possible while achieving human-like behavior when needed, balancing complexity and adaptability
Data Source
AI summary
A computer-implemented method comprises instantiating a policy function approximator. The policy function approximator is configured to calculate a plurality of estimated action probabilities in dependence on a given state of the environment. Each of the plurality of estimated action probabilities corresponds to a respective one of a plurality of discrete actions performable by the reinforcement learning agent within the environment. An initial plurality of estimated action probabilities in dependence on a first state of the environment are calculated. Two or more of the plurality of discrete actions are concurrently performed within the environment when the environment is in the first state. In response to the concurrent performance, a reward value is received. In response to the received reward value being greater than a baseline reward value, the policy function approximator is updated, such that it is configured to calculate an updated plurality of estimated action probabilities.