HVAC Reinforcement Learning Training Using Simulated and Real Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
HVAC systems face challenges in accurately training reinforcement learning models for optimal control due to the need for substantial data, where simulated experience data may not accurately capture actual system dynamics, and real experience data collection is slow, requiring a method to improve both short-term and long-term control efficiency.
Innovation Solution
A method that combines simulated and real experience data to train reinforcement learning models, using a dynamic model to generate initial simulated data and subsequently retrain the model with real data, potentially retraining the dynamic model to generate additional simulated data for continuous improvement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If simulated experience data is used to train the reinforcement learning model, then the training speed is improved and data collection time is reduced, but the accuracy of system dynamics representation deteriorates
Solution Approach 1:
The patent combines simulated experience data and real experience data into a unified training dataset. The reinforcement learning model is trained initially on simulated data for rapid iteration, then retrained on real data from the actual HVAC system to improve accuracy. This merging approach allows the system to leverage both the speed of simulation and the fidelity of real-world data.
Solution Approach 2:
The patent performs preliminary training on simulated experience data before deploying the model to the real HVAC system. This preliminary action allows the reinforcement learning model to learn basic control policies quickly without waiting for extensive real data collection, and then refine its performance through subsequent retraining on actual system data.
2Measurement precision
If real experience data is collected from the HVAC system, then the training accuracy is improved, but the data collection time increases significantly
Solution Approach 1:
The patent uses a partial approach by collecting real experience data for retraining rather than relying exclusively on real data from the beginning. The system collects real data progressively and uses it to retrain the model periodically, rather than requiring complete real data datasets before training can begin. This allows training to proceed with improved accuracy as real data becomes available.
Solution Approach 2:
The patent introduces simulated experience data as an intermediary training source that bridges the gap between theoretical models and real-world data. The simulated data serves as a mediator that allows the reinforcement learning model to develop initial competencies before being exposed to the complexities of real HVAC system data, thereby reducing the time needed for real data collection.
3Measurement precision
If the reinforcement learning model is retrained frequently with real data, then the control policy accuracy is improved, but the computational resources and training time are consumed
Solution Approach 1:
The patent implements periodic retraining of the reinforcement learning model using real experience data. Instead of continuous retraining, the system collects real data over time and retrains the model at intervals, allowing the model to operate with its current policy between retraining cycles. This periodic approach balances accuracy improvement with computational resource management.
Solution Approach 2:
The patent changes the training parameters by switching between simulated and real data sources at different stages. The model is trained on simulated data initially, then retrained on real data when sufficient data is collected. This parameter change in data source allows the system to optimize between training speed and accuracy at different phases of deployment.
Data Source
AI summary
Systems and methods for training a reinforcement learning (RL) model for HVAC control are disclosed herein. Simulated experience data for the HVAC system is generated or received. The simulated experience data is used to initially train the RL model for HVAC control. The HVAC system operates within a building using the RL model and generates real experience data. A determination may be made to retrain the RL model. The real experience data is used to retrain the RL model. In some embodiments, both the simulated and real experience data are used to retrain the RL model. Experience data may be sampled according to various sampling functions. The RL model may be retrained multiple times over time. The RL model may be retrained less frequently over time as more real experience data is used to train the RL model.