Water Distribution Pump Control Using Hybrid RL Setpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing water distribution network optimization systems are expensive, require extensive computing power, and fail to guarantee convergence to a feasible solution in real-time, leading to hesitation in user adoption due to significant differences from existing controls.
Innovation Solution
An AI-powered controller using reinforcement learning (RL) with a warm-start strategy, formulating the optimization problem as a Markov Decision Process (MDP) and leveraging the GYM framework to provide real-time optimal control setpoints that are robust and incrementally introduced, ensuring energy savings and constraint compliance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If clustering and genetic algorithms are used to optimize water distribution networks, then the number of iterations to converge is reduced, but convergence is still not guaranteed and computing power requirements remain high
Solution Approach 1:
The patent uses simulation models to create virtual copies of the water distribution network, allowing the RL agent to learn optimal control policies in a simulated environment before deploying to the real system. This copying approach enables extensive training without risking real system performance while ensuring convergence through the mathematical guarantees of RL theory.
Solution Approach 2:
The patent replaces traditional mechanical optimization algorithms (clustering and genetic algorithms) with a reinforcement learning agent that uses trial-and-error learning combined with mathematical convergence guarantees. The RL agent substitutes the swarm-based mechanical optimization with an intelligent agent that learns optimal policies through interaction with the environment, providing both speed and convergence guarantees.
2Productivity
If swarm-based optimization algorithms are used, then optimization solutions can be generated, but extensive computing power and significant iterations are required (over 10 hours)
Solution Approach 1:
The patent performs preliminary training of the RL agent in a simulated environment before deployment to the real water distribution network. The agent learns optimal control policies in advance through extensive simulation episodes, so that when deployed to the real system, it can provide real-time optimization recommendations without requiring extensive processing time during actual operation.
Solution Approach 2:
The patent creates simulation copies of the water distribution network to pre-train the RL agent. By training in the simulated environment multiple times before deployment, the system transfers the learned knowledge to the real system, enabling real-time optimization without the computational burden of running swarm algorithms during actual operation.
3Productivity
If optimization algorithms generate significantly different control recommendations from existing practices, then improved optimization is achieved, but user adoption is delayed due to hesitancy
Solution Approach 1:
The patent implements a gradual deployment strategy where the RL agent's recommendations are initially mixed with existing control practices (e.g., 80% existing + 20% RL recommendations), then progressively shifts toward more RL-generated recommendations as operators become comfortable. This partial action approach allows operators to benefit from optimization improvements while maintaining familiarity with existing practices, reducing resistance to adoption.
Solution Approach 2:
The patent incorporates feedback mechanisms where the system monitors operator responses and system performance, adjusting the degree of RL recommendation implementation based on user acceptance and actual outcomes. This feedback loop builds operator confidence by demonstrating tangible benefits while allowing them to control the pace of change, thereby improving adoption rates.
Data Source
AI summary
A method and system control a water distribution network. A database is maintained of prior states based on a residential water demand, a tank level, and an energy tariff. A current state of the water distribution network is determined. Rewards are determined and include a tank level constraint, an energy cost, and a toggle count. A query based model is used to determine a set of control points used to control a first prior state. An RL agent is trained based on the prior states and rewards. The RL agent determines a control setpoint (that changes the pump speed) that maintains the tank level, minimizes the energy cost, and complies with the toggle count. The RL agent determines time slots and selects one of the time slots. Hybrid setpoints are generated to control the water distribution network within the selected time slot.