Artificial intelligence for grid planning

Abstract

Methods, systems, and apparatus, including medium-encoded computer program products, for planning of electrical power grids using artificial intelligence, and, in particular, using reinforcement learning (RL) for grid planning.

Description

Attorney Docket No. 43374-0871 WO1ARTIFICIAL INTELLIGENCE FOR GRID PLANNINGCROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Application No. 63 / 777,518, filed on March 25, 2025, the contents of which are hereby incorporated by reference.TECHNICAL FIELD

[0001] This specification relates to electrical power grids, and specifically the planning of electrical power grids using artificial intelligence.BACKGROUND

[0002] Electrical power grids transmit electrical power to loads such as residential and commercial buildings. The electrical power grids comprise a variety of electrical components, including power sources, loads, transmission lines, distribution stations, and the like.

[0003] In the face of increasing demand, e.g., due to electric vehicles, residential electrification, etc., and new generation e.g., new solar, battery, or other power sources, transmission and distribution grid planners face the question of what to build and when to build to ensure reliable operation of the grid. Given the multivariate nature of a power grid, the decision space regarding grid planning is immense, and evaluating a decision, e.g., anew line to relieve congestion, is a costly and manual process. Typical analysis, for example, may take an independent sendee operator (ISO) or regional transmission organization (RTO) many months. Additionally, grid planners’ actions today affect future load / generation growth, so planners must account for future uncertainty. Planning has thus been heavily reliant on human expertise to propose and evaluate small numbers of potential decisions using available grid simulators.

[0004] Moreover, finding optimal solutions is very' difficult, if not impossible, when manual planning is involved. This is because the immense decision space simply cannot be evaluated by a human to the degree necessary to find an optimized solution.Attorney Docket No. 43374-0871 WO1SUMMARY

[0005] This specification describes technologies that relate to planning of electrical power grids using artificial intelligence, and, in particular, using reinforcement learning (RL) for grid planning.

[0006] In general, innovative aspects of the subject matter described in this specification can be embodied in computer-implemented methods that include the actions of accessing a grid model representing a current state of an electrical grid, wherein the grid model comprises a graph data structure having nodes and edges corresponding to physical grid components, and wherein each node and edge is associated with a set of physical state variables indicative of electrical or thermal conditions of the physical grid components; processing the grid model using a trained reinforcement learning (RL) agent to determine a plurality of potential grid modification actions for the electrical grid; generating a plurality of updated grid models by simulating an application of each of the plurality of potential grid modification actions to the grid model; evaluating the plurality’ of updated grid models using a value function to compute a numeric value for each updated grid model, wherein the numeric value quantifies physical reliability’ metrics and economic costs associated with the respective updated grid model; ranking the plurality’ of potential grid modification actions based on the numeric values computed for their corresponding updated grid models; and selecting one of the plurality of potential grid medication actions as a grid modification. Other aspects include systems and software operable to perform such operations.

[0007] In an aspect, selecting one of the plurality of potential grid medication actions as a grid modification comprises: outputting the ranked plurality of potential grid modification actions via a user interface to facilitate physical modification of the electrical grid; and receiving a selection of one of the plurality of potential grid medication actions as a grid modification.

[0008] In an aspect, evaluating the plurality of updated grid models using a value function to compute a numeric value for each updated grid model comprises calculating the numeric value by performing a power flow simulation on the grid model.

[0009] In an aspect, the power flow simulation identifies at least one of voltage violations, thermal violations, or line congestion.

[0010] In an aspect, incorporating a causal graph during training to generate a causal reinforcement learning agent configured to provide causality information identifying relationships between grid variables and suggested actions.Attorney Docket No. 43374-0871 WO1

[0011] In another aspect, a computer-implemented method for training a reinforcement learning agent for grid planning includes the actions of: accessing a grid model representing a state of an electrical grid; defining a value function that generates a numeric value based on at least one of reliability metrics or economic costs associated with the grid model; providing the grid model to a reinforcement learning agent; receiving, from the reinforcement learning agent, a selected action from a plurality of potential actions to modify the grid model; updating the grid model based on the selected action; calculating a reward for the reinforcement learning agent using the value function based on the updated grid model; and adjusting a policy of the reinforcement learning agent based on the reward to maximize the value function over a sequence of actions. Other aspects include systems and software operable to perform such operations.

[0012] In an aspect, the value function calculates the numeric value by performing a power flow simulation on the grid model.

[0013] In an aspect, the power flow simulation identifies at least one of voltage violations, thermal violations, or line congestion.

[0014] In an aspect, the actions include training a neural network to approximate the value function, wherein the neural network predicts a grid state outcome to substitute for a power flow simulation.

[0015] In an aspect, selecting one of the plurality of potential grid medication actions as a grid modification comprises: receiving a selection of a human-selected action that differs from a top-ranked action produced by the reinforcement learning agent; and updating the value function based on the human-selected action using reinforcement learning from human feedback.

[0016] In an aspect, the actions include incorporating a causal graph during training to generate a causal reinforcement learning agent configured to provide causality information identifying relationships between grid variables and suggested actions.

[0017] In an aspect, a computer-method for power grid planning includes accessing a grid model representing a current state of a power grid; identifying a violation within the grid model; applying a reinforcement learning agent to the grid model to determine a plurality of potential mitigation actions to address the violation; calculating, by a value function, a score for each of the plurality of potential mitigation actions based on at least one of a reliability metric or a cost metric; and outputting a ranked list of the potential mitigation actions based on the score. Other aspects include systems and software operable to perform such operations.Attorney Docket No. 43374-0871 WO1

[0018] In an aspect, the method includes selecting a mitigation action from the ranked list; updating the grid model based on the selected mitigation action; and iteratively applying the reinforcement learning agent to the updated grid model to determine a sequence of infrastructure expansions.

[0019] In an aspect, the value function comprises a neural network trained to approximate power flow simulation results.

[0020] In an aspect, the method includes receiving a selection of a mitigation action from a human operator; and updating the value function based on the selection via reinforcement learning from human feedback.

[0021] In an aspect, the reinforcement learning agent utilizes a causal graph to provide an explanation of the potential mitigation actions.

[0022] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Unlike existing technology for grid planning, which largely consists of grid modeling and simulation software, the systems and methods that utilize an agent as disclosed herein increase the speed of grid planning and also unlock lower-cost or cleaner trajectories for grid build out, i.e., more optimal solutions than available solely through manual planning. Additionally, while existing methods of automating studies, such as by scripting what studies to run and how to change the grid model, still rely on planner judgement to determine what studies and models to study, the systems and methods described herein teach a machine-learned agent to make these determinations, which results in a more optimal solution than those generated by existing systems and methods. These may augment, or in some cases replace, the planner judgement. The use of a graph-based model with asset features of nodes and edges readily facilities the use of a reinforcement learning agent and training of the agent, as the asset features label for each node and edge limits the corresponding search space of decisions.

[0023] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the invention will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Fig. 1 is a diagram of an example system for electrical power grid modeling.Attorney Docket No. 43374-0871 WO1

[0025] Fig. 2 is a diagram an environment for simulating electrical grid transmission and distribution.

[0026] Fig. 3 is a block diagram of the grid planning subsystem.

[0027] Figs. 4A - 4C are depictions of a graph of a grid model undergoing potential grid changes.

[0028] Fig. 5 is a flow diagram of an example process of evaluating grid model changes using a grid model planning agent.

[0029] Fig. 6 is a flow diagram of an example process of training a grid model planning agent.

[0030] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0031] This specification describes technologies that relate to planning of electrical power grids using artificial intelligence, and, in particular, using reinforcement learning (RL) for grid planning. In particular, this written description is directed to an automated system for electrical grid planning and optimization using RL. By utilizing an RL agent that interacts with the grid model, such as a graph-based grid model, and a value function, the system rapidly evaluates an immense decision space to propose optimal grid expansion and violation-mitigation actions.

[0032] These features and additional features are described in more detail below.

[0033] Fig. 1 is a diagram of an example system 100 for electrical power grid modeling. The system 100 includes a grid model server system 102. The server system 102 may be hosted within a data center 104. which can be a distributed computing system having many (e.g., tens, hundreds, or thousands) of computers in one or more locations.

[0034] The server system 102 includes a modelling system 150. The modelling system 150 may implement a number of modelling functions as subsystems. In this example implementation, the modelling system 150 includes a model converter subsystem 152, a unified modelling subsystem 154, and a grid planning subsystem 156.

[0035] The system 150, and each subsystem 152, 154 and 156, can be provided as one or more computer executable software modules or hardware modules. That is, some or all of the functions of system and subsystems can be provided as a block of computer code, which upon execution by a processor, causes the processor to perform functionsAttorney Docket No. 43374-0871 WO1described below. Some or all of the functions can be implemented in electronic circuitry, e.g., by individual computer systems (e.g.. servers), processors, microcontrollers, afield programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0036] The sen- er system 102 also includes electric grid models 190. The electric grid models 190 can include virtual representations of components of an electric grid located within a geographic region. The geographic region can include, for example, an area of hundreds of square meters, several square kilometers, hundreds of square kilometers, or thousands of square kilometers. The geographic region can correspond to a location of an electrical distribution feeder or multiple feeders. In some cases, the geographic region can correspond to a location of a bulk power system within and throughout, e.g., a state, county, province, or country.

[0037] The grid models 190 include elements that represent the components of an electrical grid and interconnections among the elements. The components can include inverters, relays, PPCs, Energy' Management Systems, RASs, Automatic Generator Controls, alarm systems and so on. In addition, components can include other elements relevant to the transmission and distribution of power, such as transmission towers and utility' poles. Elements in the grid model can include references to descriptive information about the components that can include various metadata, such as a unique element identifier for an element, information about the component represented by the element such as make, model, deployment date, damage reports, photographs, service history, role of the element and so on. A role can include whether the element is used for transmission or distribution, or both. The descriptive information can further include information about the environment at or around components, such as temperature and humidity measured at various times. The grid models 190 can further include descriptions of components that connect components, such as power lines. For such connection components, the grid model can include a description of the components connected by the connection component, and a description of the connection component that connects.

[0038] Elements of a grid model 190 can have associated operating conditions that specify constraints on operation of the component. For example, an operating condition can indicate that the temperature at the component cannot exceed a maximum value or that the voltage at a component must remain within a given range. Operating conditions can be expressed as Boolean expressions and can be associated with an element representing a component.Attorney Docket No. 43374-0871 WO1

[0039] The individual models within the electric grid models 190 may be of different formats, and may be used to model different aspects of the electric grid. Each model is a particular virtual representation of the physical grid, based on the electrical components present, how they are connected in a topology, their configurations, parameters and characteristics. The models may be of different types, and may be proprietary or based on open standards. Example proprietary models include PSLF (Positive Sequence Load Flow) and PSS / E (Power System Simulator for Engineering) models. Example open standard models include CIM (Common Interchange Format), IEEE CDF (Common Data Format).

[0040] Electric grid models are also specific to applications, e.g., PSCAD / EMTDC (Power System Computer Aided Design / ElectroMagnetic Transients including DC) formats are used for Electromagnetic Transients (EMT) studies. At other times, the details of the electric grid are represented using textual and graphical representations, e.g., Single Line Diagrams. All of these formats represent the same physical grid in different ways, and in varying degrees of temporal and spatial resolution.

[0041] In some implementations, the electric grid models 190 may be received from third parties, represented by the grid models 112 provided to the modelling system 150. In other implementations, the electric grid model 190 includes models derived from the grid models 112 and / or from other data, as will be described below.

[0042] Grid w ire paths 114 may be provided as data representing of paths of electric grid wires over a geographic region. The paths of the electric grid wires can be, for example, grid w ire paths that are visible in overhead images of the geographic region. In some examples, the wire paths 114 can be provided by vector data. The vector data can be generated through image processing techniques including segmentation processes that are used to identify locations and paths of grid wires, or provided by third parties. The vector data can define characteristics of the grid wire paths (e.g., position, length, direction) by a list or set of vectors. The vector data can include, for example, coordinate positions corresponding to endpoints of vectors. In some examples, the coordinate positions of vector endpoints can each be defined by a geographic latitude and longitude.

[0043] The data provided to the modelling system 150 can also include planning data 116. Planning data 116 may be provided by third parties, e.g., utilities or powder producers, or may be provided by the administrators of the modelling system 150. The planning data 116 may specify build outs, forecast future utility demand, and otherwiseAttorney Docket No. 43374-0871 WO1define demands and changes to the underlying grid that requires modelling and simulation.

[0044] The modeling system 150 can also receive auxiliary data 170 that can be used in modelling the grid, but which is not itself actual electrical grid data. This can include aerial imagery7172, property boundaries 174, and transportation routes 176 and topological features 178. The auxiliary data 170 can be used to determine modelling and planning constraints for the electrical grid that may not be specified in existing grid models 112 or other grid-specific data.

[0045] The aerial imagery' 172 can include imagery' collected from overhead sensors. Overhead sensors can include, for example, aerial and satellite sensors. Overhead sensors can include visible light cameras, infrared sensors, RADAR sensors, and LIDAR sensors. The aerial imagery 172 can include visible light data, e.g., red-green-blue (RGB) data, collected by the overhead sensors. The aerial imagery 172 can also include hyperspectral data, multispectral data, infrared data, RADAR data, and LIDAR data collected by the overhead sensors. The aerial imagery 172 can include two-dimensional (2D) data, 2.5D data, or 3D data. The aerial imagery7172 can include multiple channels or layers of imagery data. For example, the aerial imagery7172 can include an RGB layer, a height model layer, a digital surface model layer, and a vegetation index layer. The data 172 can also include geolocation data specifying locations of features depicted in the images. Property boundaries 174 can include image data indicating demarcations between properties, communities, municipalities, towns, counties, etc., within the geographic region. Transportation routes 176 can include image data indicating paths of roads, railroads, sidewalks, waterways, etc. Topological features 178 can include image data indicating elevations, land forms, etc.

[0046] In some examples, auxiliary data 170 can include non-image features. Nonimage features can include an identification of the geographic region. The identification of the geographic region can include, for example an identification of a state, province, county, or city. In some examples, the geographic region can include an identification of geographic boundaries of the geographic region, e.g., longitudinal and latitudinal boundaries. In some examples, the auxiliary data 170 can include property boundaries and transportation routes in vector format. In some examples, the auxiliary training data 170 can be represented as continuous valued features, embedded features, or categorical features.Attorney Docket No. 43374-0871 WO1

[0047] The model converter subsystem 152 can be used to convert from a first model type to a second model type. The unified modelling subsystem 154 can be used to generate and maintain a unified grid model from the grid models 112 and other data. The grid planning subsystem 156 can be used for planning grid expansions and evaluating impacts of changes to the grid and demand changes. Example operations of the grid planning subsystem 156 are described with reference to Figs. 3 - 6 below.

[0048] Interdependencies among components can be included in a unified grid model (or '‘grid model,” for brevity), which is a model that spans the total i ty of components from generators to end loads (e.g., households). A unified model can be a software representation of power system components and electrical networks that can include mathematical representations of the components used for simulation and analysis.Physical components of the electrical grid can be represented by elements of the grid model.

[0049] One of the models 190 can be a unified grid model. A unified grid model can be built using various data sources including topological data, geographical data, and characteristics of individual grid assets. Such data can be obtained from various data sources such as imagery and LIDAR measurements of actual grid components, sensor data (e.g., measurements obtained from actual grid operations), and utility7data. Utility¬ data can include information relating to various aspects of the electrical grid, including conductor types, poles and attachments, phase connections, among many other examples.

[0050] Electrical power grids include a broad range of interconnected components that can be organized into two broad categories: transmission components that deliver power from power generation along high voltage wires across long distances to substations, and distribution components that distribute power from substations to endpoints such as homes and businesses. Some elements, such as substations, participate in both transmission and distribution. The components can be of various types such as inverters (Solar, Wind, HVDC, etc.), relays, Power Plant Controllers (PPCs), Energy Management Systems, Remedial Action Systems (RAS), Automatic Generator Controls, alarm systems and so on.

[0051] The operation of one component often influences the operation of other components. For example, a PPC regulates and controls networked inverters within a power plant. In addition, various components can operate differently under different load conditions. Further, the output of one component can influence the load of otherAttorney Docket No. 43374-0871 WO1components. Understanding how the totality' of components in the grid operate can aid in proper grid operation.

[0052] Fig. 2 is a diagram an environment 200 for simulating electrical grid transmission and distribution. Simulations can be used to determine how various components will operate under such vary ing load conditions. The model used for simulation can be called an electrical grid simulation model (or "simulation model." for brevity), which can operate on a unified grid model or on a subset of a unified grid model.

[0053] The environment can include a simulation system 201, one or more electrical grid simulation models 257 based on the electric grid models 190. The grid simulation models 257 are stored in a simulation repository 255.

[0054] The grid models 190 can include references to one or more simulation models 257 that apply to the grid models 190. In some implementations, each element of a simulation model 257 includes a reference to a simulation model 257 for that element. In some implementations, a simulation model 257 can apply to a subset, or ’ region." of the grid model 190.

[0055] The simulation models 257 can include a description of how elements in a grid region (yvhich can be an entire grid or a subset of a grid) are predicted to behave under various electrical conditions, yvhere an electrical condition can include various loads and other conditions (e.g., weather conditions). In some implementations, simulation models 257 can include one or more functions that can accept as input loads and conditions and can produce predicted loads at the elements within and at the boundaries of the portion of the grid being simulated. In some implementations, simulation models 257 can be machine learning models, such as neural networks, configured to accept as input loads and conditions and to produce as output predicted load within and at the boundaries of the portion of the grid being simulated. Other forms of grid models, including deterministic models, can be used, and various forms of computer simulations (functions, neural netyvorks. computer code, etc.) can be used in combination.

[0056] Such simulation models 257 can accept as input simulated loads at the boundary' of the grid region, and can produce predictions that can include (i) predicted loads at one or more of the components within the grid region, (ii) predicted loads at the boundary of the region, or (iii) both predicted loads at the components within the grid region and predicted loads at the boundary of the region.Attorney Docket No. 43374-0871 WO1

[0057] The simulation models 257 can apply to an entire grid region, or to a portion of a grid region. In implementations in which a simulation model 257 applies to an entire grid region, the simulation model 257 can accept as inputs and produce outputs for the entire grid region. In some implementations, multiple simulation models 257 apply to the elements in a grid region. For example, each element within a grid region can have an associated simulation model 257, and the simulation can be performed by simulating each element with the grid region. In another example, multiple sub-region within a grid region can have associated simulation models 257, and the simulation can be performed by simulating each sub-region with the grid region.

[0058] In this example implementation, the simulation system 201 includes a simulation model obtaining engine 210, a user interaction engine 217, a boundary condition determination engine 220 and a grid simulation engine 225. The user interaction engine 217 that provides user interface presentation data to computing devices 205 such as personal computer, laptops, smart phones and tablet computer.When rendered by the computing device 205, the user interface presentation data can enable a user to provide information to the user interaction engine 217 that can be used by the simulation system 201. For example, the user interaction engine 217 can provide descriptions of grid model subsets 253 to the grid model engine 215.

[0059] In some implementations, the grid model engine 215 can obtain a grid model 190 and provide grid model subsets 253 to the boundary condition determination engine 220 and to the grid simulation engine 225. A grid model subset 253 can be a proper subset of a grid model 190, and can include elements and connections among the elements. A grid model subset 253 can represent a functional subset of a grid model. For example, one grid model subset 253 can include transmission elements and a second grid model subset 253 can include distribution elements. In another example, one grid model subset 253 can include elements operated by one entity (e.g., a power company), and a second grid model subset 253 can include elements operated by a different entity. The grid model subsets 253 are derived from the extant unified grid model 190.

[0060] The grid model engine 215 can obtain a grid model 190 or grid model subsets 253 using techniques suitable for the data repository, such as structured query language (SQL) operations to retrieve data from a relational database or file system operations provided by an operating system to retrieve models from a file system.

[0061] The boundary condition determination engine 220 can accept grid model subsets 253 and determine boundary conditions 222 between the grid model subsets 253.Attorney Docket No. 43374-0871 WO1Boundary conditions 222 can represent intersections between elements of one grid model subset 253 and a second grid model subset 253. Boundary conditions can include both overlapping elements (e.g., the same elements that are in each grid model, or elements that are directly coupled to each other in the grid models, such as conductors on either side of a transformer) and conditions that must exist at the elements (e.g., same voltage, same current, or same power).

[0062] A condition can be specified as a Boolean expression that must evaluate to TRUE. For example, a boundary condition can specify that for an element common to two grid model subsets, both grid model subsets the voltage must be the same. In another example, a boundary condition can specify that a property' (e.g., a voltage) must be within a specified range for each element subject to the boundary condition. A boundary condition can be an operating condition, as described above.

[0063] The simulation model obtaining engine 210 can obtain simulation models 257 from a simulation model repository' 245. The simulation model obtaining engine 210 can obtain simulation models 257 using techniques suitable for the data repository, such as structured query language (SQL) operations to retrieve data from a relational database or file system operations provided by an operating system to retrieve models from a file system. The simulation model obtaining engine 210 can provide simulation models 257 to the grid simulation engine 225.

[0064] The grid simulation engine 225 can accept simulation models 257. grid model subsets 253 and boundary conditions 222 and provide predicted operational values such as voltage and current. The grid simulation engine 225 can execute simulation models 257 on the grid model subsets 253 and using the boundary' conditions 222 as constraints.

[0065] Adding electrical power grid interconnections can improve the operation of an electrical grid, for example, by adding capacity, including clean and renewable energy sources, such as solar power systems. In addition, since electrical demands evolve, an electrical grid can undergo additions and changes on a continual basis. New buildings, renewable power plants, stationary storage, mobile storage, and expansions to existing buildings, facilities, and loads are some examples of potential changes that can be proposed and made to existing electrical distribution feeders.

[0066] Fig. 3 is a block diagram of the grid planning subsystem 156. The subsystem 156 includes a grid model 190, a grid model planning agent 300, and a reinforcement learning system 302. which implements a value function 304 during training and that isAttorney Docket No. 43374-0871 WO1used during inference. In some implementations, the grid model planning agent 300 is an RL agent.

[0067] The grid model 190, grid model planning agent 300, and value function 304 are used in a reinforcement loop. The grid model planning agent 300 makes a grid model decision (“Action”), the grid model 190 is updated according to the decision, and the value function 304 is applied to the updated grid model 190 and fed back to the grid model planning agent 300. In the case of reinforcement learning, for example, the value function provides a reward and state representation to the model for further adjustment. In this way, the grid model planning agent 300is trained to perform grid model 190 updates.

[0068] The value function 304 processes grid model 190 and quantifies how successful the grid has been planned, i.e., how successful a potential grid modification is. In some implementations, the value function 304 receives data describing the state of the grid model 190 and produces one or more numeric values encapsulating both the reliability and economic cost of the system. In some implementations, the value function can output a single numeric value that reflects an overall quality of the grid model 190 given a current state or when considering a potential grid modification. In other implementations, the value function can output a vector of numeric values, where each element of the vector reflects a particular quality of a particular aspect of the grid model 190.

[0069] In some implementations, power flow simulations are used by the system to determine voltage or thermal violations, line congestion, unserved load, etc. The value function 304 can also take into account operating and investment costs required with new construction. Influence graphs, neural networks, steady state simulations, dynamic simulations, transient simulations or other appropriate simulation techniques can be used as value function 304 implementations. More generally, the system can use any appropriate mechanism that translates, e.g., by ML, or by a deterministic process, the representation of the physical grid to a vector space where distant elements in the original space map to nearby points in the vector space if a change in one element will have an effect in the other.

[0070] In summary’, the value function 304 captures whether a particular snapshot of the grid model 190 has been planned well or not. The value function 304 feeds its response to the grid model planning agent 304, which is then trained to learn an optimal policy.Attorney Docket No. 43374-0871 WO1

[0071] In some implementations grid model planning agent 300 is trained to be able to make decisions that it predicts will maximize the value function 304 over multiple decisions. The grid model planning agent 300 leams a policy that it uses to make a decision on which action to take on the grid model 190. In some implementations, the grid model planning agent 300 (sometimes, not necessarily always) surfaces the changes to the grid model to a planning engineer that allows the engineer to identify promising candidates for potential grid buildout.

[0072] By way of example, a single loop of the subsystem 156 may be as follows. First, the subsystem 156 is provided with a grid model 190 that includes data describing a grid model problem requiring remediation. One example is a thermal violation of a line. The value function 304 determines the cost of the model 190 as a whole, accounting for the high cost of a thermally overloaded line. The grid model planning agent 300 evaluates the potential decision space, evaluating potential options for both immediate but also future value, and produces a set of actions (grid model 190 changes), and each modification or series of modifications are ranked by a value calculated by the value function 304.

[0073] In some implementations, the agent 300 selects the highest rated potential grid modification as the grid modification to implement. In other implementations, the potential modifications can be presented to a user as a ranked set of actions, e.g., “Here our options for building a single asset, ranked 1-10,’" and / or a ranked set of sequences of actions, e.g., “here are 1-10 best ways to address this problem, but each one may include several sequential steps.”

[0074] In some implementations, the best (highest ranked) option / action is selected and the agent 300 and the grid model is updated in accordance with the best option. The value function 304 is then reevaluated and another iteration may be performed.

[0075] The agent 300 can thus be used for grid planning to determine optimal mitigations for various kinds of grid violations. The agent 300 can be fed a sequence of violations, and for each violation produce an optimal mitigation action.

[0076] As there is often not a single answer to a violation, the agent 300 can also produce a ranked list of mitigations. In some implementations, a person can select a mitigation that need not be top ranked, based on exogenous factors not captured in the agent 300, including human experience, emergent limitations, etc., for which the agent 300 has not been trained, e.g., a sudden unavailability of a particular transformer, for example, etc.Attorney Docket No. 43374-0871 WO1

[0077] In some implementations, when a human selects a mitigation, that information is fed back to improve the value function 304. This is especially useful when the selection solution is not a top ranked solution. This technique, reinforcement learning from human feedback, can be done by a separate offline process.

[0078] As described above, grid model 190 maintains data describing the state of the grid, its topology, demand and generation, etc. In some implementations, the grid model 190 can reflect a static point in time with a specific defined problem to address, e.g.. an interconnection study where a new generator addition has the potential to overload a piece of transmission equipment, requiring a mitigation strategy, or it can reflect an evolving grid that has prescheduled interconnection or grid element commissioning / decommissioning.

[0079] In some implementations, the grid model 190 may be represented as a graphbased state space representation with physical variables. An example of such a representation is shown in Fig. 4A. Here, the grid model 190 is represented in a graph structure 420, with nodes 430, 432, 434, 436, 438 and 440 connected to edges 450. 452, 454, 456, 458, 460, 462, 464 and 466. The nodes can represent certain elements of the grid, such as transformers, power sources, substations, batteries, loads, etc. The edges can represent buses, transmission lines, or other physical structures that connect nodes. Each node and edge as a set of asset features {A} that are specific to that node or edge. For example, the set of asset features {A} for a particular edge representing a transmission line may be the rated voltage of the line, the thermal capacity, the ampacity, etc., and the set of asset features {A} for a node representing a transformer may be the primary and secondary voltages, the transformer rating, the location, etc. Asset features may also include other features not related to performance specifications, such as age of the asset, replacement cost, maintenance costs, etc.

[0080] The graph structure 420 of Fig. 4 A thus explicitly defines the RL state space using these specific localized asset features. These asset features can be provided to the value function 304 to evaluate both immediate and future economic / reliability costs for proposed grid modifications. For example, assume that node 438 represents a substation, and the region served by the substation will increase demand 10% over the next five years. Thus, in the current state, the grid 420 has a numeric value score of R = R1. The value of R1 may be, for example, negative, indicating a remediation is required to address the increased demand.Attorney Docket No. 43374-0871 WO1

[0081] The search space for potential grid modifications will include increasing substation capacity, or. alternatively moving some of the substation load to another substation. This is based on the asset features {A} and nodes and edges that are deemed to be relevant to the violation. Potential grid modifications to realize the increase in substation capacity are illustrated in Fig. 4B, which shows a grid graph 422 with potential modifications to transmission lines 458 and 464. Line 464 provides the main power feed to the substation 464, and line 458 is a distribution line. Both lines are increased in capacity, and a resulting value of R = R2 is generated by the value function.

[0082] Potential grid modifications to realize the moving of a portion of the substation load to another substation are illustrated in Fig. 4C, which shows a grid graph 424 with a new distribution line 468. The new line 468 takes over a portion of the load previously served by the line 458 of the substation 438. This decision can be made, for example, based on a search space indicating the substation 436 has excess capacity that can serve the demand required by the substation 438. A resulting value of R = R3 is generated by the value function.

[0083] When calculating R2 and R3, the value function 304 can take into account a variety of asset features. For example, in Fig. 4B, the main cost may be the upgrading of the transmission lines to higher capacity. In Fig. 4C, however, the line costs may not be as expensive, as the capacity of the lines in Fig. 4C need not be the capacity of lines in Fig. 4B, but other costs, such as. for example, securing an easement, could be factored by the value function 304.

[0084] Provided both values R2 and R3 result in a higher score than Rl, both can be presented as potential grid modifications. Conversely, if a score turns out to be less than Rl. the potential grid modification can be discarded. For example, if the value of R3 is less than RL then the potential grid modification may be ignored.

[0085] Fig. 5 is a flow diagram of an example process 500 of evaluating grid model changes using a grid model planning agent. The process 500 can be performed by one or more computers programmed to perform the operations described below.

[0086] The process 500 accesses a grid model representing a current state of an electrical grid (502). For example, the grid planning agent 300 can access the grid model 190 of Fig. 4 A.

[0087] The process 500 evaluates the grid model using a trained reinforcement learning (RL) agent to determine a plurality of potential grid modification actions for the electrical grid (504). For example, the grid planning agent 300 accesses the assetAttorney Docket No. 43374-0871 WO1features of the grid model 190 to determine potential modifications to the electrical grid, and generates the potential modifications of Figs. 4B and 4C.

[0088] The process 500 generates a lurality of updated grid models by simulating an application of each of the plurality of potential grid modification actions to the grid model (506). For example, the grid planning agent constructs grid models 422 and 424 of Figs. 4B and 4C.

[0089] The process 500 evaluates the plurality of updated grid models using a value function to compute a numeric value for each updated grid model 500 (508). For example, the value function 304 is applied to the grid models 422 and 422 to generate the scores R2 and R3.

[0090] The process 500 ranks the plurality of potential grid modification actions based on the numeric values computed for their corresponding updated grid models (510). For example, the potential grid modifications of Figs. 4B and 4C are ranked according to the scores R2 and R3.

[0091] The process 500 selects one of the plurality of potential grid modifications actions as a grid modification (512). For example, the potential grid modification with the highest score is selected automatically, or, alternatively, a human operation may select the potential grid modification as the grid modification.

[0092] Fig. 6 is a flow diagram of an example process of training a grid model planning agent. The process 600 can be performed by one or more computers programmed to perform the operations described below.

[0093] The process 600 defines a value function that generates a numeric value based on at least one of reliability7metrics or economic costs associated with a grid model (602). For example, the value function can be based on the asset features {A} of the grid model 420. The features can be weighted by the value function, for a set of features {Al ... An} a corresponding set of weights {W1... Wn} can be applied.

[0094] The process 600 provides the grid model to a reinforcement learning agent (604). The agent 300 accesses the grid model 420, for example, and begins to evaluate potential grid modifications. The grid modifications can be in response to an identified violation, for example.

[0095] The process 600 receives, from the reinforcement learning agent, a selected action from a plurality of potential actions to modify the grid model (606). For example, the agent 300 may select one of the potential modifications as the action, based on the value function. The process 600 updates the grid model based on the selected actionAttorney Docket No. 43374-0871 WO1(608). For example, if the change to increase power line capacity is selected, then the grid model 420 of Fig. 4A is updated to the model 422 of Fig. 4B.

[0096] The process 600 calculates a reward for the reinforcement learning agent using the value function based on the updated grid model (610). For example, the reinforcement learning system 302 calculates the reward using a reward function. The reward function can be any appropriate reinforcement learning reward function, and can take into account (a) physical reliability metrics derived from power flow simulations (e.g., thermal violations, voltage limits, unserved load), (b) economic costs (operating and capital investment for new construction), and (c) environmental factors, for example.

[0097] The process 600 adjusts a policy of the reinforcement learning agent based on the reward to maximize the value function over a sequence of actions (612). For example, the agent 304 may learn that the attendant cost of increasing transmission line capacity may lessen the likelihood of maximizing the value function, and thus may¬ adjust its policy accordingly, e.g., slightly de-emphasize transmission line changes, for example.

[0098] Optionally, during training a causal graph can be used to obtain a causal agent 104 that can provide the human with causality information at inference time to allow explanation of the mitigations suggested. The cause graphs illustrates the causal relationship between events or variables. This involves determining which factors cause which outcomes, and how interventions might affect those outcomes. This enables the incorporation of knowledge or assumptions about the underlying causal relationships in the environment to inform decision-making.

[0099] Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-implemented computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.Attorney Docket No. 43374-0871 WO1

[0100] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus can also be or further include special purpose logic circuitry', e.g., a central processing unit (CPU), a FPGA (field programmable gate array), or an ASIC (application-specific integrated circuit). In some implementations, the data processing apparatus and / or special purpose logic circuitry may be hardware-based and / or software-based. The apparatus can optionally include code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. The present disclosure contemplates the use of data processing apparatuses with or without conventional operating systems, for example Linux, UNIX, Windows, Mac OS, Android, iOS or any other suitable conventional operating system.

[0101] A computer program, which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. While portions of the programs illustrated in the various figures are shown as individual modules that implement the various features and functionality through various objects, methods, or other processes, the programs may instead include a number of sub-modules, third party services, components, libraries, and such, as appropriate. Conversely, the features and functionality of various components can be combined into single components as appropriate.Attorney Docket No. 43374-0871 WO1

[0102] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., a central processing unit (CPU), a FPGA (field programmable gate array), or an ASIC (application-specific integrated circuit).

[0103] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0104] Computer-readable media (transitory or non-transitory. as appropriate) suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memoi ' devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The memory may store various objects or data, including caches, classes, frameworks, applications, backup data, jobs, web pages, web page templates, database tables, repositories storing business and / or dynamic information, and any other appropriate information including any parameters, variables, algorithms, instructions, rules, constraints, or references thereto. Additionally, the memory may include any other appropriate data, such as logs, policies, security' or access data, reporting files, as well as others. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.Attorney Docket No. 43374-0871 WO1

[0105] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display), or plasma monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0106] The term “graphical user interface,” or GUI, may be used in the singular or the plural to describe one or more graphical user interfaces and each of the displays of a particular graphical user interface. Therefore, a GUI may represent any graphical user interface, including but not limited to, a web browser, a touch screen, or a command line interface (CLI) that processes information and efficiently presents the information results to the user. In general, a GUI may include a plurality of user interface (UI) elements, some or all associated with a web browser, such as interactive fields, pulldown lists, and buttons operable by the business suite user. These and other UI elements may be related to or represent the functions of the web browser.

[0107] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subj ect matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet, and a wireless local area network (WLAN).

[0108] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communicationAttorney Docket No. 43374-0871 WO1network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0109] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular implementations of particular inventions. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of sub-combinations.

[0110] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be helpful. Moreover, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0111] Particular implementations of the subject matter have been described. Other implementations, alterations, and permutations of the described implementations are within the scope of the following claims as will be apparent to those skilled in the art. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results.

[0112] Accordingly, the above description of example implementations does not define or constrain this disclosure. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this disclosure.

[0113] What is claimed is:

Claims

Attorney Docket No. 43374-0871 WO1CLAIMS1. A computer-implemented method, comprising:accessing a grid model representing a current state of an electrical grid, wherein the grid model comprises a graph data structure having nodes and edges corresponding to physical grid components, and wherein each node and edge is associated with a set of physical state variables indicative of electrical or thermal conditions of the physical grid components;processing the grid model using a trained reinforcement learning (RL) agent to determine a plurality of potential grid modification actions for the electrical grid;generating a plurality of updated grid models by simulating an application of each of the plurality of potential grid modification actions to the grid model;evaluating the plurality of updated grid models using a value function to compute a numeric value for each updated grid model, wherein the numeric value quantifies physical reliability metrics and economic costs associated with the respective updated grid model;ranking the plurality of potential grid modification actions based on the numeric values computed for their corresponding updated grid models; andselecting one of the plurality7of potential grid medication actions as a grid modification.

2. The computer-implemented method of claim 1, wherein selecting one of the plurality of potential grid medication actions as a grid modification comprises:outputting the ranked plurality of potential grid modification actions via a user interface to facilitate physical modification of the electrical grid; andreceiving a selection of one of the plurality7of potential grid medication actions as a grid modification.

3. The computer-implemented method of claim 1, wherein evaluating the plurality7of updated grid models using a value function to compute a numeric value for each updated grid model comprises calculating the numeric value by performing a power flow simulation on the grid model.

4. The computer-implemented method of claim 3, wherein the power flow simulation identifies at least one of voltage violations, thermal violations, or line congestion.Attorney Docket No. 43374-0871 WO15. The computer-implemented method of claim 1, further comprising incorporating a causal graph during training to generate a causal reinforcement learning agent configured to provide causality information identifying relationships between grid variables and suggested actions.

6. A computer-implemented method for training a reinforcement learning agent for grid planning, the method comprising:defining a value function that generates a numeric value based on at least one of reliability metrics or economic costs associated with a grid model;access, by a reinforcement learning agent, the grid model;receiving, from the reinforcement learning agent, a selected action from a plurality of potential actions to modify the grid model;updating the grid model based on the selected action;calculating a reward for the reinforcement learning agent using the value function based on the updated grid model; andadjusting a policy of the reinforcement learning agent based on the reward to maximize the value function over a sequence of actions.

7. The computer-implemented method of claim 1, wherein the value function calculates the numeric value by performing a power flow simulation on the grid model.

8. The computer-implemented method of claim 3, wherein the pow er flow simulation identifies at least one of voltage violations, thermal violations, or line congestion.

9. The computer-implemented method of claim 1, further comprising training a neural network to approximate the value function, wherein the neural network predicts a grid state outcome to substitute for a power flow simulation.

10. The computer-implemented method of claim 1, wherein selecting one of the plurality of potential grid medication actions as a grid modification comprises:receiving a selection of a human-selected action that differs from a top-ranked action produced by the reinforcement learning agent; andupdating the value function based on the human-selected action using reinforcement learning from human feedback.Attorney Docket No. 43374-0871 WO111. The computer-implemented method of claim 1, further comprising incorporating a causal graph during training to generate a causal reinforcement learning agent configured to provide causality information identifying relationships between grid variables and suggested actions.

12. A computer-method for power grid planning comprising:accessing a grid model representing a current state of a power gnd; identifying a violation within the grid model;applying a reinforcement learning agent to the grid model to determine a plurality of potential mitigation actions to address the violation;calculating, by a value function, a score for each of the plurality of potential mitigation actions based on at least one of a reliability7metric or a cost metric; and outputting a ranked list of the potential mitigation actions based on the score.

13. The computer-implemented method of claim 1, further comprising:selecting a mitigation action from the ranked list;updating the grid model based on the selected mitigation action; and iteratively applying the reinforcement learning agent to the updated grid model to determine a sequence of infrastructure expansions.

14. The computer-implemented method of claim 1, wherein the value function comprises a neural network trained to approximate power flow simulation results.

15. The computer-implemented method of claim 1, further comprising:receiving a selection of a mitigation action from a human operator; and updating the value function based on the selection via reinforcement learning from human feedback.

16. The computer-implemented method of claim 1 , wherein the reinforcement learning agent utilizes a causal graph to provide an explanation of the potential mitigation actions.

17. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations of any one of claims 1 - 16.Attorney Docket No. 43374-0871 WO118. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of any one of claims 1 - 16.

Images