Methods and systems for training HVAC control using surrogate model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
HVAC systems face challenges in accurately training reinforcement learning models for efficient control due to the need for substantial data, where simulated experience data may not capture actual operations, and real experience data collection is slow, requiring a method to improve both short-term and long-term control.
Innovation Solution
A method that combines simulated and real experience data to train reinforcement learning models, using a dynamic model to generate initial simulated data and subsequently retraining with real data, with the option to retrain the dynamic model to generate additional simulated data for continuous improvement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If simulated experience data is used to train the reinforcement learning model, then training speed is improved, but data accuracy deteriorates
Solution Approach 1:
The patent combines simulated experience data and real experience data into a unified training dataset. The reinforcement learning model is trained initially on simulated data for rapid learning, then fine-tuned on real data collected from actual HVAC system operation, merging the advantages of both data sources to achieve both fast training and high accuracy.
Solution Approach 2:
The system performs preliminary training using simulated experience data before deploying the model to real HVAC systems. This preliminary action allows the model to learn basic control policies quickly from simulation, reducing the time needed for real-world training and minimizing the burden on actual HVAC systems during data collection.
2Measurement precision
If real experience data is collected from HVAC system operation, then data accuracy is improved, but training time increases
Solution Approach 1:
Instead of relying solely on extensive real data collection, the system uses a partial approach by combining a smaller amount of real experience data with simulated data. The real data is used strategically to correct and refine the policies learned from simulation, achieving accurate training without the time cost of collecting large volumes of real operational data.
3Measurement precision
If the reinforcement learning model is retrained frequently, then control accuracy is improved, but computational cost increases
Solution Approach 1:
The system implements periodic retraining of the reinforcement learning model using newly collected real experience data. Instead of continuous retraining, the model is retrained at scheduled intervals or when sufficient new data has accumulated, maintaining control accuracy while reducing unnecessary computational overhead and energy consumption.
Data Source
AI summary
Systems and methods for training a reinforcement learning (RL) model for HVAC control are disclosed herein. A calibrated simulation model is used to train a surrogate model of the HVAC system operating within a building. The surrogate model is used to generate simulated experience data for the HVAC system. The simulated experience data can be used to train a reinforcement learning (RL) model of the HVAC system. The RL model is used to control the HVAC system based on the current state of the system and the best predicted action to perform in the current state. The HVAC system generates real experience data based on the actual operation of the HVAC system within the building. The real experience data is used to retrain the surrogate model, and additional simulated experience data is generated using the surrogate model. The RL model can be retrained using the additional simulated experience data.