Control Agent Training Using Dynamic Models and Action Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing control agent training methods for technical systems require large amounts of data and are susceptible to poor performance in conditions poorly covered by training data, leading to inefficiencies and prediction errors.
Innovation Solution
A method involving a data-based dynamic model and action evaluation process, such as a variational autoencoder, is used to configure a control agent by minimizing reproduction errors, allowing for efficient training with less data and reducing susceptibility to prediction errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If large amounts of batch training data are used to train a control agent, then the control agent can cover more operating conditions, but the training data requirement increases and training efficiency decreases
Solution Approach 1:
The patent introduces a replay buffer as an intermediary component that stores and manages training data. This replay buffer acts as a mediator between the environment (technical system) and the neural network, enabling efficient reuse of training data through experience replay. The replay buffer stores transitions (state, action, reward, next state) and samples them in batches for training, reducing the need for continuous collection of large volumes of training data while maintaining comprehensive coverage of operating conditions.
2Reliability
If control actions are restricted through constraints to improve control behavior in poorly covered areas, then control performance improves, but training becomes more data-intensive
Solution Approach 1:
The patent implements feedback mechanisms through the replay buffer that continuously stores and revisits training experiences. The system samples from the replay buffer to retrains the neural network on previously encountered states and actions, providing feedback loops that improve control behavior in poorly covered areas. This feedback mechanism allows the system to learn from past experiences without requiring additional data collection, thereby improving reliability in underrepresented operating conditions without increasing data volume.
3Reliability
If Q-function training is used to improve control in underrepresented state space areas, then control behavior improves, but the training process becomes more complex and error-prone
Solution Approach 1:
The patent employs experience replay that copies and reuses training data from the replay buffer multiple times for training the neural network. Instead of requiring complex Q-function training mechanisms, the system copies past experiences and retrains the policy network on these copied data samples. This copying approach simplifies the training process while effectively improving control behavior in underrepresented areas by providing repeated exposure to diverse operating conditions without increasing structural complexity.
Data Source
Figure 1~3
Figure 4
AI summary
To configure a control agent (POL), predefined training data is imported, specifying state data records (S), action data records (A), and resulting performance values (R) of the technical system (TS). Using this training data, a data-driven dynamic model (NN) is trained to reproduce a resulting performance value (R) based on a state data record (S) and an action data record (A). Furthermore, an action evaluation process (VAE) is trained to reproduce the action data record (A) based on a state data record (S) and an action data record (A) after information reduction, thereby determining a reproduction error (DR, D0, D1). To train the control agent (POL), training data is fed to the trained dynamic model (NN), the trained action evaluation process (VAE), and the control agent (POL).Performance values (R1, R2) output by the trained dynamic model (NN) are fed into a predefined performance function (P). Reproduction errors (D0, D1) output by the trained action evaluation process (AEP) are also fed into the performance function (P) as performance-reducing factors. This trains the control agent (POL) to output an action data set (A) that optimizes the performance function (P) based on a state data set (S).