Control Strategy Training with Safe Action Retry in Simulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current reinforcement learning approaches for autonomous driving face challenges in making learning progress due to the predominance of penalties for non-implementable actions, leading to a high number of roll-outs required to learn a high-reward strategy.
Innovation Solution
The method allows multiple attempts to find a safe action during training, continuing simulation runs even after a non-implementable action is encountered, and using a safe action to improve training without requiring a continuous target variable or threshold values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a penalty is added for non-implementable actions and the simulation run is interrupted early, then the control strategy is trained to avoid non-implementable actions, but the learning progress becomes extremely difficult and an excessively high number of roll-outs are needed
Solution Approach 1:
The patent applies partial action by allowing the simulation run to continue partially even after a non-implementable action occurs. Instead of completely interrupting the roll-out, the system continues collecting data and rewards for subsequent states, thereby obtaining partial learning signals that still contribute to training efficiency while maintaining safety constraints.
Solution Approach 2:
The patent implements feedback by providing continuous reward signals throughout the simulation run, including after non-implementable actions. The control strategy receives feedback about the consequences of its actions (positive rewards for safe actions, negative rewards for non-implementable actions) and uses this information to improve its policy, enabling more efficient learning compared to complete interruption approaches.
2Reliability
If the simulation run is interrupted after a single non-implementable action, then non-implementable actions are penalized heavily, but information about the contribution of previous actions to rewards is lost
Solution Approach 1:
The patent maintains continuity of useful action by allowing the simulation run to continue after non-implementable actions. The data collection and reward calculation processes remain continuous, ensuring that information about the contributions of previous actions to final rewards is preserved and utilized for training, rather than being lost due to early interruption.
Solution Approach 2:
The patent uses an intermediary approach by introducing a distinction between the occurrence of non-implementable actions and the termination of simulation runs. The system mediates between the need to penalize non-implementable actions and the need to preserve learning information by allowing runs to continue while still applying penalties, thus acting as an intermediary between these two requirements.
3Reliability
If a model is expanded by a layer to prevent selection of impermissible actions, then actions can be evaluated for safety, but the model requires the target variable to be linear in the action which is often not the case
Solution Approach 1:
The patent extracts the safety evaluation function from the main control policy model. Instead of requiring the control model itself to be linear and include safety constraints, the system separates safety checking into a distinct evaluation step that can use simple linear checks, while the main model remains flexible and nonlinear as needed for complex control tasks.
Solution Approach 2:
The patent segments the control system into separate functional components: the control policy model that selects actions, the safety evaluation layer that checks implementability, and the reward function that provides training signals. This segmentation allows each component to be optimized independently, with the safety layer using simple linear evaluations while the main model handles complex nonlinear control requirements.
Data Source
AI summary
A method of training a control strategy for a control. An exploration strategy for a current version of the control strategy is determined in each of several iterations. Several simulation runs are carried out, in each of which an action is selected in accordance with the exploration strategy, and it being checked if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected. A follow-up state of the state in the sequence of states is ascertained. The sequence of states are collected as data of the simulation run; for the iteration. The value of a loss function is ascertained over the data of the executed simulation runs and the control strategy is adapted so that the value of the loss function is reduced.