Hierarchical RL Control for Chiller Plant Time-Scale Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing reinforcement learning (RL) techniques struggle to learn realistic operation of industrial machinery due to machinery constraints, such as actions being executed at different time scales, leading to inefficiencies and wear and tear, particularly in HVAC systems like chiller plants.
Innovation Solution
A hierarchical reinforcement learning scheme is employed, using a high-level controller neural network for long-term decisions and a low-level controller neural network for short-term decisions, optimizing energy savings and temperature control while adhering to operational constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If traditional reinforcement learning techniques are used to control HVAC equipment, then energy optimization can be achieved, but the equipment may be turned on and off frequently or left on for extended periods, causing wear and tear and unrealistic operation
Solution Approach 1:
The control system is segmented into two hierarchical levels: a high-level controller that makes long-term decisions about equipment operational states (on/off decisions every few hours), and a low-level controller that makes short-term adjustments to operating parameters (temperature setpoints between high-level decisions). This segmentation allows each controller to operate at its appropriate timescale, preventing unrealistic frequent switching while maintaining energy optimization.
2Device complexity
If a single controller is used for both long-term and short-term decisions, then system complexity is reduced, but the controller cannot optimally reason across different time scales required for real-world facility control
Solution Approach 1:
The control system adds a temporal dimension by introducing hierarchical time scales. The high-level controller operates on a long timescale (making decisions every few hours about equipment states), while the low-level controller operates on a short timescale (adjusting parameters between high-level decisions). This dimensional separation in time allows the system to optimally reason across different time scales, improving control efficiency despite increased structural complexity.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling a facility through hierarchical reinforcement learning. In particular, the facility is controlled using a high-level controller neural network that makes high-level decisions and a low-level controller neural network that makes low-level controller decisions.