Autonomous Vehicle Maneuver Selection With Safe Action Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous vehicle control systems using reinforcement learning (RL) lack safety guarantees, leading to potential unsafe decisions due to the prioritization of efficiency and comfort over safety during training, which complicates training and obscures the understanding of action effects.
Innovation Solution
A safety framework is integrated with a neural network architecture to predefine safe and legal actions, limiting the action space and ensuring only those actions are considered, using mechanisms like dot products and softmax activation to filter out unsafe maneuvers, thereby providing transparent and efficient training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If reinforcement learning prioritizes efficiency and comfort during training, then training convergence speed improves, but safety of vehicle decisions deteriorates
Solution Approach 1:
The patent applies preliminary action by pre-defining the action space to include only safe and legal maneuvers before training begins. The safety framework and neural network architecture are configured upfront to filter out unsafe actions, ensuring that the RL agent only learns from safe experiences. This resolves the contradiction by establishing safety constraints before efficiency optimization occurs during training.
2Adaptability or versatility
If the action space is not limited, then the agent can explore more possibilities, but transparency of action understanding deteriorates
Solution Approach 1:
The patent applies local quality by making the action space heterogeneous with different properties - some actions are marked as safe/legal while others are unsafe/illegal. The neural network outputs probabilities for each action type, and the safety framework selectively enables/disables specific actions based on their safety classification. This allows the agent to explore within safe boundaries while maintaining transparency about which actions are considered safe versus unsafe.
3Reliability
If unsafe actions are filtered out after execution, then safety is maintained, but training efficiency deteriorates
Solution Approach 1:
The patent applies preliminary action by configuring the neural network architecture and safety framework before training to prevent unsafe actions from being selected in the first place. The action space is pre-filtered to only include safe maneuvers, and the RL agent receives feedback only on safe actions. This eliminates the need for post-execution filtering and avoids wasted training iterations on unsafe actions, thereby maintaining safety while improving training efficiency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer implemented method for determining a maneuver to be executed by an autonomous vehicle comprises the following steps carried out by computer hardware components: determining a plurality of potential maneuvers; determining a subset of the potential maneuvers based on a pre-determined condition; determining a mask based on the subset; determining an output of a fully connected layer of a neural network; determining a dot product of the output of the fully connected layer and the mask; and determining the maneuver to be executed based on the dot product.