Autonomous Vehicle Motion Control with RL Setpoint Hierarchy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing autonomous vehicle control systems, particularly for trucks, struggle with efficient speed control and lane-change maneuvers in multilane environments, often requiring extensive machine learning efforts and lacking effective coordination between high-level decision-making and low-level control actions.
Innovation Solution
A vehicle control method that separates high-level decision-making by a reinforcement-learning (RL) agent and low-level control actions by a feedback controller, allowing the RL agent to focus on strategic decisions while the feedback controller handles actual motion control, using a serial arrangement to enhance efficiency and safety.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single reinforcement learning agent performs both high-level decision-making and low-level motion control, then the system can make autonomous decisions, but the computational load increases and training becomes less efficient
Solution Approach 1:
The control system is divided into two separate RL agents: a high-level agent that performs strategic decision-making (lane changes, speed adjustments) and a low-level agent that handles tactical motion control (steering, acceleration, braking). This segmentation allows each agent to specialize in specific control tasks, reducing the computational burden on individual agents and improving training efficiency while maintaining comprehensive autonomous control capabilities
Solution Approach 2:
The control architecture introduces a hierarchical dimension with multiple control levels. The high-level agent operates at the strategic dimension, selecting desired actions from a reduced action space, while the low-level agent operates at the tactical dimension, executing precise motion commands. This dimensional separation enables more efficient training by allowing each agent to learn from appropriately scaled rewards and state representations
2Extent of automation
If the reinforcement learning agent directly controls motion actuators, then the system can achieve autonomous vehicle control, but the training time and computational resources required increase significantly
Solution Approach 1:
The autonomous control system is segmented into hierarchical layers with distinct responsibilities. The high-level RL agent handles strategic decisions with a simplified action space, while the low-level RL agent manages tactical execution. This segmentation reduces the complexity of the learning problem for each agent, enabling faster convergence and shorter training times while maintaining full autonomous control capability
Solution Approach 2:
The high-level RL agent performs preliminary strategic planning by selecting desired actions (lane changes, speed adjustments) before the low-level agent executes the actual motion commands. This preliminary action approach allows the system to make high-level decisions with reduced computational requirements, then refine the execution through low-level control, thereby reducing overall training time and computational resource requirements
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A computer system for controlling at least one motion actuator in an autonomous vehicle (100), the computer system comprising processing circuitry implementing a feedback controller (120, 122), which is configured to sense an actual motion state (x) of the vehicle and determine a machine-level instruction (u) to the motion actuator for approaching a setpoint motion state (x*), a reinforcement-learning, RL, agent (130), which is trained to perform decision-making regarding the setpoint motion state (x*). The decisions by the RL agent are applied as the setpoint motion state (x*) of the feedback controller, and the machine-level instruction (u) from the feedback controller (120) is applied to the motion actuator.