Mode Switching for Classical Planning in Partially Observable RL
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods are inadequate for enabling classical planning in interactive environments with incomplete models, particularly due to partial observability and incorrectly learned transition models in model-based reinforcement learning.
Innovation Solution
A computer-implemented method that switches between probabilistic planning and information gathering modes, using a classical planning algorithm and an information gathering policy to produce paths from an initial state to a goal state, and trains an information gathering policy to increase knowledge about the current state, allowing the agent to switch between using the classical planning algorithm and the information gathering policy based on the current model of the world.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model-based reinforcement learning is used to learn environment dynamics, then data efficiency is improved, but the model becomes opaque and difficult to integrate with external knowledge bases
Solution Approach 1:
The patent segments the planning system into two distinct modes: probabilistic planning mode for actions with resolved plans and information gathering mode for actions that increase knowledge. This segmentation allows the system to handle different types of actions separately, making the overall system more manageable and easier to integrate with external knowledge bases while maintaining the benefits of model-based reinforcement learning.
2Reliability
If classical planning is used in partially observable environments, then planning guarantees are improved, but the agent cannot reach goal states when information is not observable
Solution Approach 1:
The patent implements a dynamic switching mechanism between probabilistic planning mode and information gathering mode based on the current state of knowledge. The system dynamically adapts its behavior by selecting the appropriate mode depending on whether the goal state can be reached with current information or if additional information gathering is needed, thus maintaining planning guarantees while improving adaptability to partial observability.
Solution Approach 2:
The patent introduces an information gathering policy as an intermediary component that bridges the gap between classical planning requirements and partial observability conditions. This intermediary policy enables the agent to gather necessary information about the environment state, allowing classical planning algorithms to function effectively even in partially observable environments where information is not directly available.
3Ease of operation
If the agent always uses probabilistic planning mode, then action execution is simplified, but actions without resolved plans cannot be properly handled
Solution Approach 1:
The patent implements a dynamic switching mechanism between probabilistic planning mode and information gathering mode based on the current state of knowledge. The system dynamically adapts its behavior by selecting the appropriate mode depending on whether the goal state can be reached with current information or if additional information gathering is needed, thus maintaining planning guarantees while improving adaptability to partial observability.
4Loss of information
If the agent always uses information gathering mode, then knowledge about the environment is improved, but goal reaching efficiency is reduced
Solution Approach 1:
The patent implements a dynamic switching mechanism between probabilistic planning mode and information gathering mode based on the current state of knowledge. The system dynamically adapts its behavior by selecting the appropriate mode depending on whether the goal state can be reached with current information or if additional information gathering is needed, thus maintaining planning guarantees while improving adaptability to partial observability.
Solution Approach 2:
The system employs a self-service mechanism where the agent automatically evaluates whether to execute probabilistic planning or information gathering actions based on the current state and available knowledge. This self-service approach eliminates the need for external intervention to determine the appropriate action mode, allowing the agent to autonomously optimize its goal reaching efficiency while maintaining sufficient environmental knowledge.
Data Source
AI summary
A computer-implemented method in a model-based reinforcement learning (RL) system with logic states includes switching an agent between a first mode and a second mode, the first mode being a probabilistic planning mode and the second mode being an information gathering mode. In response to the agent being in the probabilistic planning mode, the agent computes a predictive state representation, given a history of observations and actions taken, and the agent scores action candidates based on planning with the predictive state representation so that actions with resolved plans with confidence to achieve a goal state are scored higher than actions without resolved plans. In response to the agent being in the information gathering mode, the agent scores action candidates based on a Q function of a value of expected information to be gathered from a given pair of state and action.