Reinforcement Learning Initialization for Faster Process Control Convergence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current reinforcement learning methods for machine learning models in process control, such as temperature adjustment and liquid level control, face challenges like lengthy learning times and poor convergence due to random action choices and inappropriate action widths, especially in N-order delay systems like temperature control, leading to suboptimal control performance.
Innovation Solution
A learning device that initializes a machine learning model through preliminary learning using initialization data, including state and action data, to define options for action choices based on action data distributions, allowing the model to start reinforcement learning with prior knowledge, thereby reducing randomness and improving control performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning is performed without preliminary learning initialization, then the machine learning model can learn autonomously through trial-and-error, but the learning time becomes excessively long and convergence is poor
Solution Approach 1:
The patent applies preliminary action by performing preliminary learning before reinforcement learning to initialize the machine learning model with basic control capabilities. This preliminary phase teaches the model fundamental control behaviors and appropriate action ranges, so that when reinforcement learning begins, the model starts from a knowledgeable baseline rather than complete randomness, significantly reducing the time needed to achieve good control performance
Solution Approach 2:
The extraction unit extracts sample data from initialization data obtained through preliminary learning, and the definition unit defines options for actions based on the distribution of action data. This creates a structured framework that guides the reinforcement learning process, allowing the model to efficiently explore within meaningful boundaries rather than searching the entire action space randomly
2Adaptability or versatility
If random action choices are used in reinforcement learning, then the model can explore the action space, but the action width is inappropriate leading to poor convergence
Solution Approach 1:
The patent changes the parameters of action selection by transitioning from purely random action choices to actions selected from predefined options based on distribution characteristics. The definition unit defines discrete action options according to the statistical distribution of action data obtained during preliminary learning, which constrains the action space to meaningful ranges and improves convergence while maintaining exploration capability
Solution Approach 2:
The extraction unit and definition unit act as intermediaries between the preliminary learning phase and reinforcement learning. They process the continuous action data from preliminary learning into discrete, structured options that serve as a bridge, allowing the model to maintain exploration capability while operating within appropriate action boundaries that improve convergence
3Productivity
If the machine learning model starts reinforcement learning from scratch, then it can learn optimal policies through trial-and-error, but it lacks prior knowledge leading to suboptimal control performance
Solution Approach 1:
The patent recovers and utilizes prior knowledge by performing preliminary learning before reinforcement learning. The initialization data obtained through preliminary learning contains valuable information about appropriate control actions and their distributions, which would otherwise be lost if reinforcement learning started from scratch. This preliminary phase preserves and structures prior knowledge for effective reuse
Solution Approach 2:
The patent merges two learning phases: preliminary learning that captures prior knowledge about control behaviors, and reinforcement learning that optimizes policies through trial-and-error. By combining these phases and using the output of preliminary learning to initialize and guide reinforcement learning, the system retains valuable prior knowledge while still allowing for optimal policy learning
Data Source
AI summary
Provided is a learning device including: a data acquisition unit configured to acquire, before control of a control target provided in equipment by a machine learning model that outputs an action corresponding to a state of the equipment, initialization data including state data indicating the state of the equipment and action data indicating an action on the control target; and a preliminary learning unit configured to initialize the machine learning model by performing preliminary learning on the basis of the initialization data before start of reinforcement learning corresponding to the control of the control target by the machine learning model.