Maneuver Planning With DP-Initialized Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automation systems for transportation vehicles and robots face challenges in maneuver planning, particularly in complex scenarios like turning with multiple lanes and various road users, where they need to decide which lane to use and which road users to pass for optimal comfort and time efficiency.
Innovation Solution
A method and device that utilize a Markov decision process to discretely describe the environment, determine optimal action values using dynamic programming, and initialize a reinforcement learning agent to learn a mapping for maneuver planning, allowing the agent to build on pre-determined action values rather than starting from scratch, thereby reducing learning time and considering rare states.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If reinforcement learning is used to learn maneuver planning from scratch, then the system can adapt to complex environments, but the learning time becomes excessively long
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing optimal action values for discrete states using dynamic programming before the reinforcement learning process begins. This preliminary computation creates a foundation that accelerates subsequent learning, allowing the system to start with pre-computed knowledge rather than learning everything from scratch.
Solution Approach 2:
The patent segments the learning process into two distinct phases: (1) a preliminary phase using dynamic programming to compute optimal action values for discrete states, and (2) a reinforcement learning phase that builds upon these pre-computed values. This segmentation allows each phase to specialize and contribute differently to the overall learning efficiency.
2Reliability
If the reinforcement learning agent explores all possible states thoroughly, then optimal decisions can be found, but rare and extraordinary states may be overlooked
Solution Approach 1:
The patent uses preliminary action to pre-compute optimal action values for all discrete states including rare and extraordinary ones before reinforcement learning begins. This ensures that even states that may rarely occur in practice have pre-calculated optimal actions, preventing them from being overlooked during the reinforcement learning exploration phase.
Solution Approach 2:
The patent introduces an intermediary structure (the pre-computed optimal action values from dynamic programming) that mediates between the reinforcement learning agent and the environment. This intermediary provides guidance to the agent, ensuring that optimal actions are available even for rare states that the agent might not encounter frequently during exploration.
3Measurement precision
If dynamic programming is used to determine optimal action values, then accurate decisions can be made for discrete states, but the computational complexity increases
Solution Approach 1:
The patent segments the computational task by applying dynamic programming only to discrete states rather than continuous state spaces. This segmentation reduces the computational complexity of the dynamic programming phase while maintaining accuracy for discrete states, and allows reinforcement learning to handle the more complex continuous aspects.
Solution Approach 2:
The patent performs the computationally intensive dynamic programming calculation as a preliminary action before runtime operation. By pre-computing optimal action values offline, the system shifts computational complexity from the operational phase to the setup phase, reducing real-time computational requirements while maintaining accuracy.
Data Source
AI summary
A method for assisting maneuver planning for a transportation vehicle driving by automation or for a robot; wherein a state space of an environment of the transportation vehicle or the robot is discretely described by a Markov decision process; wherein optimal action values for discretized actions are determined by dynamic programming, based on discrete states in the state space; wherein a mapping with states in the state space as input values, and with action values for actions in the state space as output values, is learned by a reinforcement learning method; wherein a reinforcement learning agent is initialized based on the optimal action values determined by the dynamic programming; and wherein the learned mapping is provided for maneuver planning. Also disclosed is a device for assisting maneuver planning for a transportation vehicle driving by automation or for a robot.