Heavy-Haul Train Control Using SAC Reinforcement Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current control methods for heavy-haul trains, such as PID and generalized predictive control, are inadequate for complex environments due to reliance on manual parameter adjustment and complex computing, and data-driven methods like neural networks face challenges with uneven data distribution and overfitting.
Innovation Solution
A reinforcement learning-based method using a soft actor-critic algorithm with an actor network and two critic networks, trained on historical data and expert strategies, to create a heavy-haul train virtual controller that learns to optimize speed and position control in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If PID control algorithm is used for speed tracking control, then automatic driving can be realized, but parameter selection relies on manual adjustment which is not suitable for complex process control
Solution Approach 1:
The patent transforms the manual parameter adjustment process into an automated learning process. The reinforcement learning agent automatically learns and adjusts control parameters through interaction with the train operation environment, eliminating the need for manual parameter selection while maintaining adaptability to complex process conditions.
Solution Approach 2:
The control system performs self-adjustment through the reinforcement learning mechanism. The agent autonomously optimizes control parameters by receiving rewards or penalties based on control performance, enabling the system to self-improve without external intervention and making it suitable for complex dynamic processes.
2Reliability
If generalized predictive control algorithm is used for speed tracking control, then control performance can be improved, but complex computing is involved and does not perform well in real-time response
Solution Approach 1:
The patent replaces complex computational control algorithms with a reinforcement learning-based intelligent control system. Instead of using heavy mathematical computations for prediction and control, the system uses learned policies from reinforcement learning that provide real-time control decisions with reduced computational burden while maintaining or improving control performance.
Solution Approach 2:
The reinforcement learning agent performs preliminary learning offline to develop optimal control policies. During real-time operation, the pre-learned policies are directly applied without requiring complex on-the-fly computations, enabling fast real-time response while maintaining high control performance.
3Adaptability or versatility
If fuzzy control is combined with automatic parking control, then control flexibility can be improved, but fuzzy rules and membership functions are derived from experience and are difficult to control and calculate
Solution Approach 1:
The patent replaces experience-based fuzzy rules and membership functions with parameters learned through reinforcement learning. The learning process automatically determines optimal control strategies and parameter values, eliminating the need for manual derivation of fuzzy rules while maintaining adaptability to different operating conditions.
Solution Approach 2:
The control system automatically learns and adjusts its own parameters through reinforcement learning interaction with the environment. This self-learning mechanism eliminates the need for manual configuration of fuzzy rules and membership functions, reducing computational complexity while maintaining control flexibility through adaptive learning.
4Extent of automation
If ordinary neural network training is used for data-driven control, then control can be achieved, but large amount of actual data is required and uneven distribution of data samples results in overfitting
Solution Approach 1:
The patent introduces a simulation environment as an intermediary between data collection and controller training. The simulation environment generates synthetic training data with balanced distribution across different operating conditions, preventing overfitting to unevenly distributed actual data while enabling effective data-driven control learning.
Solution Approach 2:
The system performs preliminary training in a simulated environment before deployment to actual operations. This preliminary action allows the controller to learn from diverse, evenly distributed synthetic data generated by the simulation, improving generalization capability and preventing overfitting that would occur with limited or unevenly distributed real-world data.
Data Source
AI summary
The present disclosure provides a method and system for controlling a heavy-haul train based on reinforcement learning. The method includes: obtaining operation state information of a heavy-haul train at a current time point; obtaining a heavy-haul train action of a next time point according to the operation state information of the heavy-haul train at the current time point and a heavy-haul train virtual controller, and sending the heavy-haul train action of the next time point to a heavy-haul train control unit to control operation of the heavy-haul train. The heavy-haul train virtual controller is obtained by training a reinforcement learning network according to operation state data of the heavy-haul train and an expert strategy network; the reinforcement learning network includes one actor network and two critic networks; the reinforcement learning network is constructed according to a soft actor-critic (SAC) reinforcement learning algorithm.