Robot Policy Learning With On-Policy Dynamics Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In model-based reinforcement learning, the learned model is inevitably only an approximation of the real world, leading to model errors that result in slow convergence and sub-optimal solutions on the real system.
Innovation Solution
The proposed solution involves learning a (potentially time-dependent) model from collected data and optimizing policy parameters based on this learned model. Additionally, a global model and an on-policy correction term are used to improve the accuracy of the transition dynamics model, reducing on-policy model errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a learned model is used for policy optimization, then policy optimization can be performed without continuous interaction with the real system, but the learned model contains errors that lead to slow convergence and sub-optimal solutions
Solution Approach 1:
The learned model is segmented into two distinct components: a global model that captures general dynamics from diverse trajectories and a correction model that specifically addresses on-policy errors. This segmentation allows each component to specialize - the global model provides broad coverage while the correction model focuses on reducing errors in the specific policy region, thereby improving overall model accuracy without sacrificing optimization efficiency
Solution Approach 2:
The correction model is designed to provide localized improvements specifically for on-policy predictions rather than attempting to improve the entire model uniformly. By focusing computational resources on correcting errors where they matter most (in the on-policy region), the system achieves higher accuracy where needed while maintaining the efficiency benefits of using a learned model for policy optimization
2Reliability
If simulation horizon is reduced to mitigate model errors, then error compounding is reduced, but the fundamental problem of model errors from rollouts remains and rollout horizon becomes a challenging hyperparameter
Solution Approach 1:
The correction model is trained using feedback from the differences between global model predictions and actual next states observed in the data. This feedback mechanism allows the correction model to learn and compensate for systematic errors in the global model, providing a self-correcting mechanism that reduces prediction errors without requiring manual adjustment of simulation horizon parameters
Solution Approach 2:
The correction model acts as an intermediary between the global model and the true environment dynamics. Instead of directly using the potentially erroneous global model predictions, the system passes them through the correction model which adjusts them based on learned patterns of error. This intermediary layer effectively mediates the impact of model errors without requiring changes to the simulation horizon
3Measurement precision
If more data is collected from the real system to improve model accuracy, then model precision improves, but data collection time and interaction requirements increase
Solution Approach 1:
The global model is trained preliminarily on a diverse dataset collected from multiple policies before the final policy optimization begins. This preliminary training phase captures a broad range of dynamics and reduces the need for extensive data collection during the actual policy optimization process, saving time while still achieving good model accuracy
Solution Approach 2:
The system changes the parameterization of the model from a single static model to a dynamic combination of a global model and a correction model that adapts based on the policy being evaluated. This parameter change allows the system to maintain high accuracy with less data by effectively reconfiguring how the model uses available information rather than requiring more data
Data Source
AI summary
A computer-implemented method for for learning a policy. The method includes: recording at least an episode of interactions of the agent with its environment following policy and adding the recorded episode to a set of training data; optimizing a transition dynamics model based on the training data such that the transition dynamics model predicts the next states of the environment depending on the states and actions contained in the training data; optimizing policy parameters based on the training data and the transition dynamics model by optimizing a reward. In the method, the transition dynamics model comprises a first model characterizing the global model and a second model characterizing a correction model, which is configured to correct outputs of the first model.