Reinforcement Learning Volt-Var Control for Power Distribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale power distribution systems face challenges in voltage regulation and reactive power flow due to non-convex optimization problems and integer decision variables, making traditional methods inefficient for systems with thousands of buses.
Innovation Solution
The use of reinforcement learning with deep learning models and integer reparameterization techniques, such as Gumbel-Softmax estimators and straight-through estimators, to train control policies that can handle discrete and integer actions in controllable devices, enabling scalable voltage and reactive power control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional optimization algorithms are used for volt-var control, then the control problem can be solved with exact methods, but the computational complexity becomes intractable for large scale systems with thousands of buses due to nonconvexity and discrete variables
Solution Approach 1:
The patent replaces traditional mechanical optimization algorithms with a reinforcement learning-based neural network system. The neural network learns optimal control policies through simulation training, substituting the need for complex real-time optimization computations with pre-trained intelligent agents that can directly output control actions for discrete devices.
Solution Approach 2:
The control policies are trained offline in simulation environments before deployment to the actual power distribution system. This preliminary training phase allows the system to learn optimal strategies in advance, so that during real-time operation, only inference is needed rather than solving complex optimization problems from scratch.
2Reliability
If controllable devices are discretely controlled to maintain voltage profile, then voltage regulation can be achieved, but the decision variables become discrete making the optimization problem highly nontrivial and difficult to scale
Solution Approach 1:
The patent substitutes discrete optimization algorithms with continuous neural network-based reinforcement learning policies. The neural network outputs continuous control signals that are then applied to discrete devices, avoiding the computational intractability of directly optimizing discrete variables while still achieving the desired discrete control outcomes.
Solution Approach 2:
The patent changes the parameter space from discrete device states to continuous neural network outputs. By training the reinforcement learning policy in continuous action space and then mapping to discrete device states, the system transforms the difficult discrete optimization problem into a more tractable continuous learning problem.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for controlling a power distribution system having a number of discretely controllable devices includes processing a system state, defined by observations acquired via measurement signals from a number of meters, using a reinforcement learned control policy including a deep learning model, to output a control action including integer actions for the controllable devices. The integer actions are determined by using learned parameters of the deep learning model to compute logits for a categorical distribution of predicted actions from the system state, that define switchable states of the controllable devices. The logits are processed to reduce the categorical distribution of predicted actions for each controllable device to an integer action for that controllable device. The control action is communicated to the controllable devices for effecting a change of state of one or more of the controllable devices, to regulate voltage and reactive power flow in the power distribution system.