Vehicle Control Using Reinforcement Learning and Future Velocity Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vehicle testing methods using human drivers lack reproducibility, while robot drivers are costly and require time-consuming tuning of control variables, and existing PID controllers struggle to provide optimal results across different evaluation modes and vehicles.
Innovation Solution
A machine learning-based driver model that automatically learns optimal Accelerator Position Sensor (APS) and Brake Pedal Position Sensor (BPS) values using reinforcement learning, incorporating a vehicle velocity predictor to anticipate future velocity and adjust control commands accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a robot driver is used to improve test reproducibility, then test deviation is reduced, but purchase price and installation difficulty increase
Solution Approach 1:
The patent creates a virtual driver model that copies the control functions of a physical robot driver. This virtual model receives velocity profile inputs and generates APS/BPS command values through machine learning algorithms, eliminating the need for expensive physical robot hardware while maintaining test reproducibility
Solution Approach 2:
The patent replaces the mechanical robot driver system with a software-based virtual driver. The machine learning model (reinforcement learning agent) substitutes the mechanical control mechanisms, processing velocity profiles and generating control commands through computational algorithms rather than physical actuation systems
2Extent of automation
If PID control is used to control vehicle velocity, then test automation is achieved, but control variable tuning becomes time-consuming
Solution Approach 1:
The patent transitions from fixed PID parameters to dynamic, learned parameters. The reinforcement learning agent automatically determines optimal APS/BPS command values based on the current velocity profile and driving conditions, eliminating manual parameter tuning while adapting control behavior to different test scenarios
Solution Approach 2:
The virtual driver model performs self-learning through reinforcement learning algorithms. The system automatically optimizes its control strategy by learning from velocity profile data and reward signals, eliminating the need for external engineers to manually tune control variables for each test case
3Extent of automation
If PI controller is used to find control variable values, then some automation is achieved, but optimal results cannot be provided for all sections and requires retraining for each evaluation mode
Solution Approach 1:
The patent implements dynamic control parameter generation through reinforcement learning. The agent adapts its control strategy in real-time based on the velocity profile and driving conditions, providing optimal APS/BPS commands for different evaluation modes without requiring retraining. The system dynamically adjusts its behavior to match various test requirements
Solution Approach 2:
The reinforcement learning-based virtual driver serves multiple evaluation modes (FTP-75, HWFET, NEDC, etc.) with a single unified model. The system processes different velocity profiles and generates appropriate control commands across all test scenarios, eliminating the need for separate tuned models for each evaluation mode
Data Source
AI summary
A method for controlling a vehicle includes: determining an accelerator position sensor/brake pedal position sensor (APS/BPS) command value based on a state variable and a reward variable including a prediction value for a future velocity of the vehicle predicted based on a past APS/BPS command value of the vehicle; and learning for a reward value according to the reward variable to satisfy a predetermined goal based on a change that the determined APS/BPS command value causes to at least one state variable under given environment information.