Decision Model Training With Adaptive Imitation and RL Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training decision models, particularly for complex scenarios like autonomous driving, is challenging due to the need for large amounts of expert data and complex reward functions, leading to low robustness and security risks in supervised learning, and inefficiency in reinforcement learning without human policy utilization.
Innovation Solution
A method combining supervised and reinforcement learning by determining policies using both models, calculating an imitation learning loss based on their difference, and adjusting an adaptive weight to balance imitation and exploration, with data distribution adjustments for improved performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If supervised learning is used to train decision models with expert data, then the model can learn human-like decision policies, but the model lacks robustness and has security risks due to over-reliance on expert data
Solution Approach 1:
The patent combines supervised learning and reinforcement learning into a unified decision model training framework. The supervised learning model learns human-like policies from expert data, while the reinforcement learning model learns from environmental feedback. The two models are merged through a multi-objective loss function that balances imitation learning loss (from policy difference) and reinforcement learning loss, enabling the system to leverage both expert knowledge and environmental interaction for improved robustness and generalization.
2Productivity
If reinforcement learning is used to train decision models, then the model can achieve strong generalization, but the training process is inefficient without utilizing human policy expertise
Solution Approach 1:
The patent applies preliminary action by first training a supervised learning model on expert data to establish a baseline policy before introducing reinforcement learning. The supervised model pre-learning from human expert policies provides a good initial decision-making framework, which then serves as a starting point for the reinforcement learning process. This preliminary supervised training significantly improves training efficiency by avoiding learning from scratch while maintaining the ability to optimize through environmental feedback.
3Adaptability or versatility
If the imitation learning loss weight is fixed, then the training process is simple, but the model cannot adaptively balance imitation and exploration throughout training
Solution Approach 1:
The patent introduces dynamics by making the imitation learning loss weight adaptive rather than fixed. The weight is dynamically adjusted based on training progress and policy differences between the two models. When the policy difference is large, the imitation learning loss weight increases to emphasize copying expert behavior. When the policy difference decreases, the weight reduces to allow more exploration through reinforcement learning. This dynamic adjustment optimizes the balance between imitation and exploration throughout the training process.
Data Source
AI summary
This disclosure provides a method, an apparatus, a device, a medium, and a program product for training a decision model. The method includes: determining a first policy using a supervised learning model and a second policy using a reinforcement learning model within the decision model based on training data; determining an imitation learning loss based on a difference between the first policy and the second policy; and training the decision model based on both the imitation learning loss and a reinforcement learning loss corresponding to the second policy. By combining the imitation learning loss and the reinforcement learning loss, a human-like decision model with excellent performance may be obtained, leveraging the expert data utilization capability of supervised learning and the strong generalization capacity of reinforcement learning. In some embodiments, the trained model is applied to autonomous driving for tasks such as lane-changing.