Control Agent Training Using Dynamic Models and Action Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data-driven machine learning methods for controlling complex technical systems require large volumes of training data and are susceptible to forecasting errors, particularly in regions with poor data coverage, leading to unfavorable control behavior.
Innovation Solution
A method involving a control agent training process that utilizes a dynamic model and an action evaluation process, such as a variational autoencoder, to optimize control actions by minimizing reproduction errors and requiring fewer training data, using performance-diminishing influencing variables to reduce the negative effects of poor data coverage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large volumes of batch training data are used to cover all operating conditions, then control agent performance improves, but training data requirements and computational resources increase
Solution Approach 1:
The training process is segmented into two distinct phases: (1) training a dynamic model and action evaluation process on available batch training data, and (2) using these trained components to guide reinforcement learning training with significantly fewer additional data points. This segmentation allows efficient utilization of limited data while achieving reliable control performance.
Solution Approach 2:
The dynamic model and action evaluation process serve as intermediary components that bridge the gap between limited batch training data and effective control agent training. These intermediaries evaluate potential actions and guide the reinforcement learning process, enabling reliable control with reduced data requirements.
2Reliability
If control actions are restricted by secondary conditions with trained Q functions, then control behavior in under-covered regions improves, but data intensity and training complexity increase
Solution Approach 1:
The dynamic model and action evaluation process are trained in advance on available batch training data before the reinforcement learning phase. This preliminary action prepares the system to handle under-covered regions effectively during subsequent training with minimal additional data, reducing overall training complexity.
Solution Approach 2:
The action evaluation process provides feedback by evaluating potential control actions based on the dynamic model's predictions. This feedback mechanism guides the reinforcement learning agent to select actions that are likely to perform well, even in under-covered regions, without requiring exhaustive training data.
3Productivity
If reinforcement learning is used to train control agents, then optimized state-specific control actions are generated, but large volumes of training data are required
Solution Approach 1:
The system dynamically adapts the training process by using the trained dynamic model and action evaluation process to guide reinforcement learning. This dynamic approach allows the system to focus learning efforts on the most valuable data points, achieving control optimization with significantly fewer training examples than traditional reinforcement learning.
Data Source
AI summary
To configure a control agent, predefined training data are read in, which specify state datasets, action datasets and resulting performance values of the technical system. Using the training data, a data-based dynamic model is trained to reproduce a resulting performance value using a state dataset and an action dataset. An action evaluation process is also trained to reproduce the action dataset using a state dataset and an action dataset after an information reduction has been carried out, wherein a reproduction error is determined. To train the control agent, training data are supplied, the trained action evaluation process and the control agent. Performance values output by the trained dynamic model are fed into a predefined performance function. Reproduction errors are fed as performance-reducing influencing variables into the performance function. The control agent is trained to output an action dataset optimising the performance function on the basis of a state dataset.