Deterministic Control Agent Selection for Sparse State Spaces
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing control agents for technical systems, trained with batch data, often perform poorly in states with limited training data coverage, and stochastic policies can produce unpredictable control actions, which are not validated reliably.
Innovation Solution
A method using deterministic control agents, where training data is used to select and iteratively adapt multiple control agents, with performance validation through machine learning modules, ensuring reliable and efficient exploration of the state space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If stochastic policies are used to explore the state space, then exploration efficiency is improved, but reliability and predictability of control actions deteriorate
Solution Approach 1:
The patent segments the control system into multiple deterministic control agents instead of using a single stochastic policy. Each agent processes state inputs independently and produces deterministic outputs, eliminating the unpredictability of stochastic policies while maintaining exploration capability through the diversity of multiple agents.
Solution Approach 2:
The patent combines multiple deterministic control agents to achieve the exploration functionality previously provided by stochastic policies. By merging the outputs of multiple deterministic agents through the machine learning module, the system achieves both reliability (deterministic behavior) and exploration efficiency (diverse action selection).
2Loss of time
If batch training data is used to train control agents, then training speed is improved, but performance in poorly covered state spaces deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the machine learning module evaluates the performance of control agents and uses this information to iteratively improve their behavior. The system learns from the outcomes of actions taken in poorly covered states, enabling continuous improvement of performance in previously unexplored or sparsely covered regions of the state space.
Solution Approach 2:
The patent employs multiple control agents that are pre-trained on batch data to provide preliminary control actions. These agents explore the state space and generate experiences in poorly covered regions, which are then used to refine the control policy iteratively, combining the speed of batch training with improved performance in uncovered states.
3Adaptability or versatility
If multiple control agents are used to explore the state space, then exploration coverage is improved, but system complexity increases
Solution Approach 1:
The patent makes the machine learning module serve multiple functions: it evaluates control actions, selects among multiple agents, and learns from experiences. This multi-functional component reduces the need for separate complex subsystems, managing the complexity introduced by having multiple control agents while maintaining comprehensive exploration coverage.
Data Source
Figure 1~3
Figure 4
Figure 5
AI summary
To control a technical system (TS), training data is read in, with each training data set (TD) comprising a state data set (S), an action data set (A), and a resulting performance value (R) of the technical system. Using the training data, a first machine learning module (NN1) is trained to reproduce a resulting performance value (R) based on a state data set (S) and an action data set (A). Furthermore, a multitude of different deterministic control agents (P1, P2,...) are each supplied with state data sets (S), and the resulting output data is fed into the trained first machine learning module (NN1) as action data sets. Depending on the performance values output by the trained first machine learning module (NN1), several of the control agents are then selected. According to the invention, the technical system is controlled by the selected control agents (SP1,...Each of the following processes is controlled: ,SPK), whereby further state data sets (ES), action data sets (EA), and performance values (ER) are acquired and added to the training data. The above process steps are then repeated with this supplemented training data, starting from the training of the first machine learning module (NN1).