Robot Control Policy Transfer Across Simulation and Real Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep reinforcement learning for robots faces challenges in applying learned movements from a simulation environment to a real environment due to differences such as friction and movement delays, making it difficult to achieve tasks like object grasping and bipedal walking.
Innovation Solution
A learning system that uses curriculum learning with different neural networks for simulation and real environments, where a robot's movement is learned in a simulation environment and the control value for the real environment is determined, allowing for efficient transfer of learned results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deep reinforcement learning is performed in a simulation environment, then learning can be done efficiently, but the learned results cannot be applied to the real environment due to environmental differences
Solution Approach 1:
The patent introduces a domain adaptation layer as an intermediary between the simulation environment policy and the real environment execution. This adaptation layer learns to translate policies trained in simulation to work effectively in the real world, bridging the sim-to-real gap without requiring complete retraining. The adapter network serves as a mediator that adjusts simulation-based policies to account for environmental differences.
Solution Approach 2:
The patent performs preliminary supervised learning using demonstration data from the real environment before performing reinforcement learning in simulation. This preliminary action provides a good initial policy that is already adapted to real environment characteristics, making the subsequent simulation-based reinforcement learning more effective and the transferred policies more reliable.
2Loss of time
If the number of trials for deep reinforcement learning is reduced, then learning time is shortened, but learning quality deteriorates
Solution Approach 1:
The patent performs preliminary supervised learning using real environment demonstration data before reinforcement learning. This preliminary action provides a high-quality initial policy that requires fewer reinforcement learning trials to achieve good performance, thus reducing total learning time while maintaining learning quality.
Solution Approach 2:
The patent uses demonstration data from the real environment to create training samples for supervised learning. By copying expert demonstrations and using them to train an initial policy, the system obtains a high-quality starting point that reduces the number of trials needed for subsequent reinforcement learning.
3Ease of manufacture
If simulation environment is used for learning, then learning can be performed safely and efficiently, but differences in friction and movement delays prevent direct application to real environment
Solution Approach 1:
The domain adaptation layer serves as an intermediary that translates simulation-based policies to real environment commands. It learns the mapping between simulation and real environment dynamics, compensating for differences in friction and movement delays without requiring the robot to be retrained in the real environment.
Solution Approach 2:
The patent introduces learnable parameter transformations in the domain adaptation layer that adjust simulation policy outputs to match real environment characteristics. By learning parameter adjustments rather than complete policy retraining, the system adapts to environmental differences efficiently.
Data Source
AI summary
A robot control device includes at least one memory, and at least one processor, wherein the at least one processor is configured to obtain environmental information in a real environment, obtain information related to an action to be performed by a robot in the real environment based on the environmental information and a first policy, obtain information related to a control value that causes the robot to perform the action based on the information related to the action and a second policy, and control the robot based on the information related to the control value. The first policy is learned by using a virtual robot in a simulation environment.