Closed-Loop Policy Training With Error-Set Upsampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Behavioral cloning policies trained in an open-loop fashion for self-driving vehicles (SDVs) suffer from distributional shift between training and deployment, leading to dangerous or illogical actions due to the model's inability to understand closed-loop effects.
Innovation Solution
A method using closed-loop weighted empirical risk minimization (CW-ERM) that involves training a policy with a closed-loop simulation to collect metrics, constructing an error set, and upsampling this set to generate a final policy, reducing mismatches between training and inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If behavioral cloning policies are trained in an open-loop fashion to predict the next action given an immediate previous action, then the training process is simple and fast, but the model does not understand the closed-loop effects of its actions leading to distributional shift between training and deployment
Solution Approach 1:
The patent applies preliminary action by performing closed-loop simulation and error analysis before final policy deployment. The system first trains an initial policy, then uses closed-loop simulation to identify errors, constructs an error set, and performs upsampling training on problematic cases before the policy is ready for deployment. This preliminary error correction phase ensures the policy understands closed-loop effects before real-world deployment.
Solution Approach 2:
The patent implements feedback by using closed-loop simulation to evaluate policy performance and feed error information back into the training process. The closed-loop metrics from simulation are used to construct an error set that highlights problematic training cases. This feedback loop allows the system to iteratively improve the policy by retraining on weighted error cases, ensuring the policy learns from its mistakes in a controlled environment before deployment.
2Ease of manufacture
If the model is trained on a standard training data set without weighting, then the training data is easy to manage, but the model fails to focus on critical error cases leading to dangerous actions
Solution Approach 1:
The patent applies local quality by differentiating the treatment of different training cases based on their error severity. Instead of uniform training, the system constructs an error set that identifies problematic cases and applies weighted upsampling specifically to these local regions of the training data. This allows the model to focus computational resources on correcting specific harmful behaviors while maintaining efficiency in managing the overall training dataset.
Solution Approach 2:
The patent implements parameter changes by modifying the training data distribution through weighted upsampling. The system changes the sampling probability parameter for error cases in the constructed error set, giving them higher weights during the second training phase. This parameter adjustment ensures critical error cases are overrepresented in the training process, forcing the model to learn from and correct dangerous behaviors without requiring complete remanagement of the training data infrastructure.
Data Source
AI summary
Systems and methods for training a policy are disclosed. In one example, a system includes a processor and a memory with instructions that cause the processor to train the policy using a training data set with training scenes to generate an identification policy and perform a closed-loop simulation on the identification policy to collect closed-loop metrics. Based on the closed-loop metrics, the instructions cause the processor to construct an error set of the training scenes and construct an upsampled training set by upsampling the error set. After that, the policy is trained using the upsampled training set to generate a final policy.