Dynamics Model for Inverse Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Reinforcement learning is adversely affected in terms of accuracy and computational expense when only state trajectories are available without action information, necessitating a technique for accurate and efficient learning from state trajectories.
Innovation Solution
A computer-implemented method that trains a dynamics model using state-action pairs to predict next state probabilities, enabling the learning of an action policy through supervised learning by back-propagating error gradients, and performs model-free inverse reinforcement learning using estimated rewards to sample environment dynamics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning is performed using only state trajectories without action information, then the system can operate with limited data availability, but the learning accuracy deteriorates and computational expense increases
Solution Approach 1:
The patent introduces an intermediary dynamics model that infers hidden action information from state transitions. This mediator bridges the gap between available state trajectories and the action information needed for accurate reinforcement learning, allowing the system to recover missing action data through learned state-action-next state relationships without requiring direct access to actual action sequences
Solution Approach 2:
The patent creates a copy of the environment dynamics through a trained dynamics model that replicates the state transition process. By copying the underlying dynamics from state trajectories alone, the system can generate synthetic action information and simulate environment interactions without needing original action data, thus maintaining learning accuracy while working with limited data
2Adaptability or versatility
If reinforcement learning is performed using only state trajectories without action information, then the system can operate with limited data availability, but computational expense increases
Solution Approach 1:
The patent performs preliminary action by pre-training a dynamics model on state trajectories before conducting reinforcement learning. This preliminary step creates a reusable computational artifact (the dynamics model) that encodes environmental dynamics, allowing subsequent RL episodes to benefit from this pre-computed knowledge and reducing the computational burden of learning from scratch with limited data
3Reliability
If a dynamics model is trained using state-action-next state triplets, then the system can accurately model environment dynamics, but the requirement for action information increases
Solution Approach 1:
The patent extracts the essential dynamics information from state transitions alone, taking out the dependency on action information. By training the dynamics model to predict next states from current states and inferred actions, the system extracts only the necessary state transition patterns while leaving behind the requirement for complete action data, thus achieving reliable dynamics modeling with reduced information requirements
Data Source
AI summary
A computer-implemented method is provided for learning an action policy. The method includes obtaining, by a processor, environment dynamics including triplets of a state, an action, and a next state. The state in each of the triplets is an expert state. The method further includes training, by the processor using the environment dynamics as training data, a dynamics model which obtains a pair of the state and the action as an input and outputs, for each next state, state-transition probabilities. The method also includes learning, by the processor, the action policy using trajectories of expert states according to a supervised learning technique by back-propagating error gradients through the trained dynamics model.