Gas Turbine RL Control With Randomized Objective Deviation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Complex systems like gas turbines face challenges in determining optimal control policies due to large or partially known state spaces, leading to inefficient control and difficulty in distinguishing between system state changes and control actions, especially when generic control objectives change.

Innovation Solution

A method using Reinforcement Learning that provides training data to explore the state space for a control policy maximizing an expected total reward, with randomized deviations from set points, allowing for efficient and stable training and adaptation to changing objectives, enabling better differentiation between control actions and system state effects.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If deterministic control policies are used for complex systems, then control precision is improved, but adaptability to changing objectives deteriorates

Engineering Contradiction:

Improvecontrol precisionVSAvoidadaptability to changing objectives

Core Design Contradiction:

Measurement precisionVSAdaptability or versatility

Solution Approach 1:

The control policy transitions from a static deterministic mapping to a dynamic probabilistic policy that adapts to changing objectives. The neural network outputs action probabilities that can be adjusted in real-time based on new control objectives, allowing the system to dynamically switch between different control goals without retraining the entire model.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The invention changes the parameter representation from deterministic control values to probabilistic action distributions. By modeling the policy as a probability distribution over actions rather than fixed action values, the system can naturally handle objective changes by modifying the distribution parameters without losing the precision learned during training.

Inventive Principle:

Principle #35Parameter changes

2Adaptability or versatility

If multiple finite control policies are maintained for different objectives, then adaptability is improved, but device complexity increases

Engineering Contradiction:

Improvehandling multiple control objectivesVSAvoidcontrol policy complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

A single neural network policy is designed to handle multiple control objectives simultaneously. The network takes as input both the system state and the current control objective, producing a unified probabilistic policy that adapts its action distribution based on the active objective, eliminating the need for separate policies for each objective.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The probabilistic action distribution serves as an intermediary between the neural network and the actual control actions. This intermediate representation allows the system to smoothly transition between different objectives by adjusting the distribution parameters, avoiding the need for hard switching between discrete policies.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If control policies are switched between different objectives, then adaptability is improved, but operational stability deteriorates

Engineering Contradiction:

Improveobjective switching capabilityVSAvoidoperational stability

Core Design Contradiction:

Adaptability or versatilityVSStability of the object's composition

Solution Approach 1:

The control policy uses dynamic probabilistic sampling to transition between objectives. Instead of hard switching, the system dynamically adjusts the action probability distribution based on the current objective, allowing for smooth transitions that maintain operational stability while adapting to new goals.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The probabilistic nature of the policy provides a cushioning effect during objective transitions. By maintaining a distribution over actions rather than committing to a single deterministic action, the system can gradually shift its behavior as objectives change, preventing abrupt transitions that would disrupt operational stability.

Inventive Principle:

Principle #11Beforehand cushioning (Prior cushioning)

4Measurement precision

If training data is collected during manual sweeps, then data quality is improved, but productivity decreases

Engineering Contradiction:

Improvetraining data qualityVSAvoiddata collection efficiency

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The system uses its own operational data during normal plant operation to train the control policy. Instead of requiring separate manual data collection campaigns, the system continuously learns from the data generated during regular turbine operation, improving both data quantity and training efficiency without sacrificing quality through selective data filtering and augmentation techniques.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11164077B2Randomized reinforcement learning for control of complex systems

Publication Date: 2021.11.02 SIEMENS AG

AI summary

A method of controlling a complex system and a gas turbine being controlled by the method are provided. The method comprises providing training data, which training data represents at least a portion of a state space of the system; setting a generic control objective for the system and a corresponding set point; and exploring the state space, using Reinforcement Learning, for a control policy for the system which maximizes an expected total reward. The expected total reward depends on a randomized deviation of the generic control objective from the corresponding set point.