Reinforcement learning-based inertia flywheel control method and system

Abstract

Disclosed in the present application are a reinforcement learning-based inertia flywheel control method and system. The method comprises: storing into an experience replay buffer transition experience tuples obtained by interaction with a simulated environment and a neural network, and randomly sampling a set number of experience tuples from the experience replay buffer at the current moment, so as to update the neural network; on the basis of rewards in the current experience replay buffer, determining whether a neural network update ending condition is satisfied, and if yes, using the updated neural network as a trained neural network, and otherwise continuing to update the neural network until the ending condition is satisfied; and inputting to the trained neural network a state variable of an output end of an inertia flywheel requiring control quantity calculation, so as to obtain an action, and on the basis of the action and a torque set value, acquiring a control quantity to be input to the inertia flywheel. By means of the solution of the present application, the neural network can adjust the policy thereof on the basis of actual performance, so as to more accurately predict and calculate the action corresponding to the state variable of the inertia flywheel output end.

Description

A reinforcement learning-based inertial flywheel control method and system Technical Field

[0001] This application generally relates to the field of energy storage technology. More specifically, this application relates to a reinforcement learning-based inertial flywheel control method and system. Background Technology

[0002] In existing technologies, inertia flywheel technology, with its large inertia, strong reactive power support, and high overload capacity, has become one of the effective ways to solve the frequency and voltage regulation problems of new energy power systems. Inertia flywheels can provide efficient inertia and voltage support, effectively mitigating system frequency and voltage fluctuations. Their high overload capacity enables them to respond quickly to system frequency and voltage fluctuations, providing a large amount of mechanical inertia and reactive power in a short time, which helps maintain the stability of system frequency and voltage.

[0003] However, when used as frequency and voltage regulation equipment in power systems, inertial flywheels face the problem of their own low-frequency oscillations and the impact of low-frequency oscillations in the power grid. This can cause serious damage to the power system, such as reducing system stability and shortening equipment lifespan.

[0004] In view of this, there is an urgent need to provide a reinforcement learning-based inertial flywheel control scheme to solve the problem of low-frequency oscillation of the inertial flywheel itself and to address the impact of low-frequency oscillation of the power grid. Summary of the Invention

[0005] In order to at least solve one or more of the technical problems mentioned above, this application proposes a reinforcement learning-based inertial flywheel control scheme in several aspects.

[0006] In a first aspect, this application provides an inertial flywheel control method based on reinforcement learning, comprising: processing a transformation experience tuple (s) obtained through an interactive simulation environment and a neural network. t ,a t ,r t ,s t+1 ) Stored in the experience pool, and a set number of experience tuples are randomly sampled from the experience pool at the current time to update the neural network, where s t Let a be the state quantity at the output of the inertial flywheel at the current time t. t For the action corresponding to the state variable at the current time t, r t s is the reward at the current time t. t+1The next action is determined based on the rewards in all experience tuples in the current experience pool to determine whether the neural network update termination condition is met. If the neural network update termination condition is met, the updated neural network is used as the trained neural network. If the neural network update termination condition is not met, the process returns to the step of updating the neural network until the neural network update termination condition is met, and the neural network updated by the last step of updating the neural network is used as the trained neural network. The state quantity of the inertia flywheel output terminal that needs to be calculated for control is input to the trained neural network to obtain the action, and the control quantity to be input to the inertia flywheel is obtained based on the action and torque setpoint.

[0007] In some embodiments, the state quantities include active power, the derivative of active power, reactive power, the derivative of reactive power, terminal voltage, the derivative of terminal voltage, rotational speed, and the derivative of rotational speed.

[0008] In some embodiments, the action includes a damping voltage component and a damping torque component.

[0009] In some embodiments, the termination condition for the neural network update includes: the sum of rewards in all experience tuples in the current experience pool is greater than a preset value for the sum of rewards.

[0010] In some embodiments, r t It is obtained through a reward function, wherein the reward function is: t step The set time step is t, where t is the current time, and ω is the current time step. e ω represents the actual rotational speed of the rotor at the output end of the inertia flywheel. ref It is the rated speed of the rotor at the output end of the inertia flywheel.

[0011] In some embodiments, the torque setpoint is the torque setpoint input to the output terminal of the inertia flywheel, and the torque setpoint is obtained by a torque setpoint formula, wherein the torque setpoint formula is: T is the torque setpoint, P ref ω is the active power output of the inertial flywheel. m K is the angular velocity of the rotor at the output end of the inertia flywheel. flag This is the charge / discharge flag for the inertia flywheel. K is used when the inertia flywheel is charging. flag The value of K is -1, and the inertia flywheel discharges. flag The value is +1, and ΔP is the power compensation term; R a This is the phase resistance at the output terminal of the inertia flywheel.

[0012] In some embodiments, the control quantity includes a drive signal input to the inertia flywheel and an excitation voltage input to the inertia flywheel.

[0013] In some embodiments, obtaining the control quantity to be input to the inertial flywheel based on the action and torque setpoint includes: adding the damping torque component and the torque setpoint to obtain a torque base value; obtaining the measured torque value corresponding to the inertial flywheel for which control quantity calculation is required, and subtracting the measured torque value from the torque base value to obtain an error value; inputting the obtained error value into a PID controller, performing proportional, integral, and derivative operations, and then outputting it to a gain amplifier for processing to obtain a first signal; subtracting the first signal from the q-axis current at the output terminal of the inertial flywheel for which control quantity calculation is required, and then inputting the first signal into a PID controller, performing proportional, integral, and derivative operations to obtain a second signal; and then subtracting the second signal from the q-axis current at the output terminal of the inertial flywheel for which control quantity calculation is required to obtain a second signal; and finally, subtracting the second signal from the q-axis current at the output terminal of the inertial flywheel for processing to obtain a second signal. e i d L d Signal and ω e The Flux signals are summed to obtain a third signal. This third signal is then input to an integrator for integration. The integrated signal is then input to a pulse width modulator to obtain the β-axis component of the drive signal input to the inertial flywheel, where ω e i represents the actual rotational speed of the rotor at the output end of the inertia flywheel. d For the d-axis current at the output of the inertia flywheel that requires control calculation, L d d-axis inductance of the inertia flywheel output terminal that requires control quantity calculation, and Flux is the magnetic flux of the inertia flywheel output terminal that requires control quantity calculation.

[0014] In some embodiments, obtaining the control quantity to be input to the inertia flywheel based on the action and torque setpoint further includes: inputting the damping voltage component, the measured voltage value of the inertia flywheel output terminal to be calculated, and the voltage setpoint value of the inertia flywheel output terminal to be calculated to the excitation controller to obtain the excitation voltage.

[0015] In a second aspect, this application provides a reinforcement learning-based inertial flywheel control system, which uses the reinforcement learning-based inertial flywheel control method as described in any embodiment of the first aspect to control the inertial flywheel. The system includes: a neural network update module, which stores the transformation experience tuples obtained through an interactive simulation environment and a neural network into an experience pool, and randomly samples a set number of experience tuples from the experience pool at the current moment to update the neural network; a data processing module, which determines whether the neural network update termination condition is met based on the rewards in all experience tuples in the experience pool at the current moment; in response to meeting the neural network update termination condition, the updated neural network is used as the trained neural network; and in response to not meeting the neural network update termination condition, the neural network update module updates the neural network again until the neural network update termination condition is met, and the neural network updated during the last execution of the neural network update is used as the trained neural network; and a control quantity acquisition module, which inputs the state quantity of the inertial flywheel output terminal that needs to be calculated into the trained neural network acquisition action, and acquires the control quantity to be input to the inertial flywheel based on the action and torque setpoint.

[0016] Using the reinforcement learning-based inertial flywheel control method provided above, this embodiment determines whether the neural network needs to be updated again based on the rewards in all experience tuples in the current experience pool. The finally updated neural network is used as the trained neural network to obtain the action corresponding to the state quantity of the inertial flywheel output terminal that needs to be calculated for control. Based on the action and torque setpoint, the control quantity to be input to the inertial flywheel is obtained. This enables the neural network to adjust its strategy according to the actual performance, more accurately predict and calculate the action corresponding to the state quantity of the inertial flywheel output terminal, and more accurately control the state of the inertial flywheel, thereby improving the control accuracy of the inertial flywheel.

[0017] Furthermore, in some embodiments, by setting the action to include a damped voltage component and a damped torque component, the problem of large rotor oscillation at the output end of the inertial flywheel can be solved when obtaining the control quantity to be input to the inertial flywheel based on the action and torque setpoint, thereby improving its resistance to power grid impact and self-stability. Even further, in some embodiments, by using a sum of rewards greater than a preset value in all experience tuples in the current experience pool as the termination condition for neural network updates, it can be ensured that the neural network reaches a certain performance level after training, while avoiding overfitting of the neural network to the training data and ineffective training of the neural network. Attached Figure Description

[0018] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0019] Figure 1 shows an exemplary flowchart of the reinforcement learning-based inertial flywheel control method according to an embodiment of this application;

[0020] Figure 2 shows a schematic diagram of the composition of the inertia flywheel according to an embodiment of this application;

[0021] Figure 3 shows a schematic diagram of the interaction between the intelligent agent model and the simulation environment in an embodiment of this application;

[0022] Figure 4 shows a schematic diagram of obtaining the inertia flywheel control quantity based on the motion and torque setpoints according to an embodiment of this application;

[0023] Figure 5 shows a schematic diagram of obtaining the inertia flywheel control quantity based on motion and torque setpoints in some embodiments of this application;

[0024] Figure 6 shows a schematic diagram of acquiring the drive signal according to an embodiment of this application;

[0025] Figure 7 shows an exemplary structural block diagram of the inertia flywheel control system according to an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It should be understood that the terms "comprising" and "including" used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0029] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] In existing technologies, the traditional solution to the low-frequency oscillation problem of inertial flywheels is to add a power system stabilizer (PSS). This mainly involves the PSS adjusting the excitation system of the generator within the flywheel to improve the excitation system and thus suppress low-frequency oscillations. However, the PSS modifies the generator's reactive power output by adjusting the excitation current. Since changes in the excitation current require time to affect the generator's electromagnetic characteristics, this adjustment method is relatively slow. Furthermore, due to the dynamic characteristics of the excitation system, the control accuracy of the PSS may be limited, especially under rapidly changing system conditions, where the excitation system's response may be imprecise, resulting in poor control performance. In addition, the PSS needs to be integrated with the existing excitation system, increasing system complexity.

[0032] In view of this, embodiments of this application provide an inertial flywheel control scheme based on reinforcement learning, which can solve the problem of low-frequency oscillation of the inertial flywheel itself and its response to the impact of low-frequency oscillation of the power grid.

[0033] Figure 1 shows an exemplary flowchart of an inertial flywheel control method 100 based on reinforcement learning according to an embodiment of this application.

[0034] As shown in Figure 1, in step S110, the transformation experience multivariate (s) obtained through the interactive simulation environment and neural network is... t ,a t ,r t ,s t+1) Stored in the experience pool, and a set number of experience tuples are randomly sampled from the experience pool at the current time to update the neural network, where s t Let a be the state quantity at the output of the inertial flywheel at the current time t. t For the action corresponding to the state variable at the current time t, r t s is the reward at the current time t. t+1 For the action in the next moment.

[0035] In the embodiments of this application, the inertia flywheel includes: a flywheel, an electromagnetic coupler, a synchronous condenser, and a converter controller.

[0036] Figure 2 shows a schematic diagram of the composition of the inertia flywheel 200 according to an embodiment of this application.

[0037] As shown in Figure 2, the inertia flywheel 200 includes a flywheel 210, an electromagnetic coupler 220, a synchronous condenser 230, and a converter controller 240. Specifically, the electromagnetic coupler 220 includes an inner rotor and an outer rotor. The rotor of the flywheel 210 is connected to the inner rotor via a first drive shaft 221, and the rotor of the synchronous condenser 230 is connected to the outer rotor via a first drive shaft 222. The inner rotor uses permanent magnets, and the outer rotor has an armature winding. The electromagnetic coupler 220 is equivalent to a permanent magnet synchronous motor in which the stator and rotor rotate simultaneously, enabling the electromagnetic coupler 220 to transmit torque and inertia between the flywheel 210 and the synchronous condenser 230. The stator of the synchronous condenser 230 is directly connected to the power grid 250. The converter controller 240 can control the electromagnetic coupler 220, and the power grid 250 can supply power to the electromagnetic coupler 220 through the converter controller 240.

[0038] In the embodiments of this application, the synchronous condenser 230 is used as the output terminal of the inertia flywheel 200.

[0039] Specifically, the converter controller 240 employs a back-to-back converter. When the external frequency rises or falls, the converter controller 240 controls the output torque of the electromagnetic coupler 220. Since the electromagnetic coupler 220 is directly connected to the rotor of the synchronous condenser 230 via a mechanical shaft, the output torque of the electromagnetic coupler 220 is the input torque of the synchronous condenser 230. Because the electromagnetic coupler 220 is equivalent to a permanent magnet synchronous motor with both stator and rotor rotating simultaneously, its output torque can be controlled by a vector controller.

[0040] In the embodiments of this application, the aforementioned simulation environment is constructed based on the parameters of the real-time running data of the inertial flywheel. It is mainly constructed by giving the system state space (i.e., output quantity) and action space (i.e., input quantity) during the real-time running process of the inertial flywheel in the deep reinforcement learning training process.

[0041] In the embodiments of this application, a neural network including an Actor network and a Critic network is built based on graph convolutional neural networks and deep reinforcement learning algorithms, and an intelligent agent model is constructed based on the neural network and the experience pool.

[0042] In the embodiments of this application, after the simulation environment and the intelligent agent model are constructed, the constructed simulation environment and the intelligent agent model are initialized. Then, the initialized intelligent agent model interacts with the initialized simulation environment to obtain the initial state s0 of the inertia flywheel output. Then, the interaction between the intelligent agent model and the simulation environment is repeated to obtain multiple conversion experience tuples.

[0043] Figure 3 shows a schematic diagram of the interaction between the intelligent agent model and the simulation environment in an embodiment of this application.

[0044] As shown in Figure 3, the Actor network includes a first optimizer, an online policy network, and a target policy network, while the Critic network includes a second optimizer, an online Q network, and a target Q network. The first optimizer and the second optimizer may be the same or different.

[0045] During the interaction between the intelligent agent model and the simulation environment, the Actor network collects the state quantity s at the current time t of the inertial flywheel output terminal from the simulation environment. t Based on the state quantity s at the current time t of the inertia flywheel output terminal t Random noise is introduced to obtain the output state variable s. t The corresponding action a t And apply it to the simulation environment to obtain the action a at the current time t. t The corresponding reward r t and the state quantity s at the output of the inertial flywheel in the simulation environment at the next moment. t+1 And the state quantity s at the current time t will be generated by the output terminal of the inertial flywheel. t Output state variable s t The corresponding action a t At the current time t, action a t The corresponding reward r t and the state quantity s at the output of the inertial flywheel in the simulation environment at the next moment. t+1 The transformed experience plural (s) constitute t ,a t ,r t ,s t+1 The data is stored in the experience pool. Random noise is introduced to obtain the output state variable s. t The corresponding action a t It can enhance the ability of intelligent agent models to explore the environment.

[0046] Next, the Critic network randomly samples a predetermined number of empirical plurals (s) from the current time-instance empirical pool.i ,a i ,r i ,s i+1 ), where s i Let a be the state quantity at time i of the output terminal of the inertial flywheel. i Let r be the action corresponding to the state variable at time i. i For the reward at time i, s i+1 This refers to the action at time i+1. Specifically, during the random sampling process, the specific value of the sampled empirical tuple can be set according to the total number of empirical tuples in the current empirical pool, which is not limited in this application.

[0047] Then, by sampling a predetermined number of empirical plural groups (s) i ,a i ,r i ,s i+1 Calculate the objective function y corresponding to the target Q-network in the Critic network. i The target Q-network will target the function y i Send to an online Q-network, which is based on the objective function y. i The loss function L of the online Q-network is obtained. The Q-gradient is obtained by minimizing the loss function L and sent to the second optimizer. The second optimizer updates the parameters of the online Q-network based on the received Q-gradient. The online policy network in the Actor network obtains the policy gradient based on the gradients of the online Q-network with respect to the online policy network and the gradients of the online Q-network with respect to the parameters of the online Q-network. This policy gradient is sent to the first optimizer, which updates the parameters of the online policy network based on the received policy gradient. Furthermore, the online policy network performs soft updates to the target policy network using a smoothing factor, and the online Q-network also performs soft updates to the target Q-network using a smoothing factor.

[0048] In the embodiments of this application, the state quantities at the output terminal of the inertia flywheel include active power, the derivative of active power, reactive power, the derivative of reactive power, terminal voltage, the derivative of terminal voltage, rotational speed, and the derivative of rotational speed. Since the output terminal of the inertia flywheel uses a synchronous condenser, the state quantities collected in the simulation environment are the active power, the derivative of active power, reactive power, the derivative of reactive power, terminal voltage, the derivative of terminal voltage, rotational speed, and the derivative of rotational speed at the stator terminal of the synchronous condenser.

[0049] In the embodiments of this application, the action includes a damping voltage component and a damping torque component. Specifically, the damping voltage component acts on the excitation controller that controls the excitation current of the synchronous condenser in the inertia flywheel, and the damping torque component serves as the input signal to the vector controller of the electromagnetic coupler in the inertia flywheel. The excitation controller changes the magnetic field strength generated by the synchronous condenser by adjusting the DC current supplied to the rotor of the synchronous condenser, thereby affecting the active and reactive power output of the synchronous condenser.

[0050] Specifically, the damping voltage component, by adjusting the excitation current, changes the magnetic field strength of the synchronous condenser, thereby suppressing its rotor oscillation. When the power grid is subjected to impacts or disturbances, the damping voltage component can quickly adjust the excitation current, helping the synchronous condenser to rapidly recover to a stable state and reducing the impact of the impact on the system. The damping torque component, by adjusting the output torque of the electromagnetic coupler, acts directly on the rotor of the synchronous condenser, reducing the rotor oscillation. When the power grid is subjected to impacts or disturbances, the damping torque component can quickly adjust the output of the electromagnetic coupler, helping the inertial flywheel to rapidly recover to a stable state and reducing the impact of the impact on the system.

[0051] Therefore, by including damped voltage and damped torque components in the action, the problem of large rotor oscillation at the output end of the inertia flywheel can be solved when the control quantity to be input to the inertia flywheel is obtained based on the action and torque setpoint, thereby improving its resistance to power grid impact and self-stability.

[0052] In the embodiments of this application, the reward r at the current time t t It is obtained through a reward function, wherein the reward function is:

[0053] t step The set time step is t, where t is the current time, and ω is the current time step. e ω represents the actual rotational speed of the rotor at the output end of the inertia flywheel. ref It is the rated speed of the rotor at the output end of the inertia flywheel.

[0054] The reward function shows that it uses speed deviation as the evaluation criterion for action. Since speed deviation directly reflects the oscillation of the synchronous condenser rotor, a small speed deviation indicates that the rotor oscillation is effectively suppressed, and the system stability is good. The goal of this application is to minimize the oscillation of the synchronous condenser when dealing with sudden changes in output power and grid impacts, and speed deviation directly reflects the degree to which this goal is achieved. Therefore, by setting a reward function that uses speed deviation as the evaluation criterion for action, the effectiveness of oscillation suppression can be directly reflected.

[0055] After completing step S110, in step S120, it is determined whether the neural network update termination condition is met based on the rewards of all experience tuples in the experience pool at the current moment.

[0056] In some embodiments of this application, the termination condition for neural network updates may be that the sum of rewards in all experience tuples in the current experience pool is greater than a preset value for the sum of rewards. In other embodiments of this application, the termination condition for neural network updates may also be that the mean of rewards in all experience tuples in the current experience pool is greater than a preset value for the mean of rewards; this application does not impose any restrictions on this.

[0057] By using a condition that the sum of rewards from all experience plurals in the current experience pool exceeds a preset value, training will only stop when the accumulated reward sum surpasses this preset value. This means the neural network's performance on the training data has reached the expected standard, and training can be stopped once the network has reached this level, preventing overfitting of the training data and avoiding the network from excessively learning specific noise and details. Furthermore, using this condition also avoids wasting computational resources and time when the performance improvement is very limited or has stalled.

[0058] In response to the fulfillment of the neural network update termination condition, in step S130, the currently updated neural network is regarded as the neural network that has completed training.

[0059] If the neural network update termination condition is not met, return to step S110 and re-execute the neural network update step until the neural network update termination condition is met. When step S110 is executed multiple times, the neural network updated in the last execution of step S110 is considered the completed neural network.

[0060] After obtaining the trained neural network, in step S140, the state quantity of the inertia flywheel output terminal that needs to be calculated for control quantity is input to the trained neural network to obtain the action, and the control quantity to be input to the inertia flywheel is obtained based on the action and torque set value.

[0061] Specifically, because neural networks can process input state variables and output actions in a very short time, efficient control of the inertial flywheel can be achieved by inputting the state variables of the inertial flywheel output terminal, which require control variable calculations, into the trained neural network to acquire actions. Simultaneously, since the trained neural network can find the optimal control strategy, inputting the state variables of the inertial flywheel output terminal, which require control variable calculations, into the trained neural network to acquire actions allows for more accurate prediction and calculation of the actions corresponding to the state variables of the inertial flywheel output terminal. This leads to more accurate control of the inertial flywheel's state, thereby improving the control precision of the inertial flywheel.

[0062] In the embodiments of this application, the torque setpoint is the torque setpoint input to the output terminal of the inertia flywheel. The torque setpoint is obtained through a torque setpoint formula, wherein the torque setpoint formula is:

[0063] T is the torque setpoint, P ref ω is the active power output of the inertial flywheel. m K is the angular velocity of the rotor at the output end of the inertia flywheel. flag This is the charge / discharge flag for the inertia flywheel. K is used when the inertia flywheel is charging. flag The value of K is -1, and the inertia flywheel discharges. flag The value is +1, and ΔP is the power compensation term; R a This is the phase resistance at the output terminal of the inertia flywheel.

[0064] The power compensation term is used to compensate for the difference between the output power and the input mechanical power of the synchronous condenser. By introducing the power compensation term into the torque setting value, the actual situation where the output power of the synchronous condenser is lower than the input mechanical power due to the loss of the synchronous condenser winding is taken into account. As a result, the obtained torque setting value is more in line with actual needs. This makes the obtained control quantity more accurate when obtaining the control quantity to be input to the inertia flywheel based on the action and torque setting value.

[0065] In embodiments of this application, the control quantity includes a drive signal input to the inertia flywheel and an excitation voltage input to the inertia flywheel. After acquiring the control quantity, it is input to the inertia flywheel to control it.

[0066] Figure 4 shows a schematic diagram of obtaining the inertia flywheel control quantity based on the motion and torque setpoints according to an embodiment of this application.

[0067] As shown in Figure 4, the intelligent agent model collects the state variables at the output of the inertial flywheel from the simulation environment. Based on the collected state variables and the reward function, a reinforcement learning algorithm is used to update the adopted strategy. After multiple updates, when the optimal strategy is obtained, the output includes a damping torque component and a damping voltage component. Next, the damping torque component is added to the aforementioned torque setpoint to obtain the torque base value, which is used as the input to the vector controller to obtain the drive signal. Simultaneously, the damping voltage component is input to the excitation controller for processing to obtain the excitation voltage. Then, the control quantity, including the drive signal and the excitation voltage, is input to the inertial flywheel to control it.

[0068] In some embodiments of this application, the process of obtaining the inertia flywheel control quantity based on the motion and torque setpoint can be referred to Figure 5.

[0069] As shown in Figure 5, the intelligent agent model collects the state variables of the stator end of the synchronous condenser from the simulation environment. The trained intelligent agent model outputs actions including damped torque and damped voltage components. Next, the damped torque component is added to the aforementioned torque setpoint to obtain the torque base value. This torque base value is used as the input to the vector controller to obtain the drive signal. The vector controller controls the output torque of the electromagnetic coupler based on the drive signal, so that the output torque directly acts on the rotor of the synchronous condenser, reducing the rotor oscillation. Simultaneously, the damped voltage component, the measured voltage value of the stator end of the synchronous condenser (which requires control quantity calculation), and the voltage setpoint value of the inertia flywheel output end (which also requires control quantity calculation) are input to the excitation controller to obtain the excitation voltage. The excitation voltage controls the excitation current flowing into the rotor winding of the synchronous condenser. Then, by adjusting the excitation current, the magnetic field strength of the synchronous condenser is changed to suppress the rotor oscillation.

[0070] Specifically, the excitation controller can calculate the excitation voltage using its internally configured control algorithm. The control algorithm used can be an adaptive control algorithm, a model predictive control algorithm, etc., and this application does not impose any restrictions on this.

[0071] Specifically, the torque base value is used as the input of the vector controller to obtain the drive signal, as shown in Figure 6.

[0072] As shown in Figure 6, the measured torque T corresponding to the inertial flywheel for which control quantity calculations are required is obtained. e The torque base value T eref Compared with the measured torque value T e The difference is used to obtain the error value; this error value is input into the PID controller, where it undergoes proportional, integral, and derivative operations before being output to the gain amplifier for processing to obtain the first signal; this first signal is then compared with the q-axis current i at the output of the inertia flywheel, which is the control quantity to be calculated. qThe subtraction is input to the PID controller, where it undergoes proportional, integral, and derivative operations to obtain the second signal; the second signal is then compared with ω. e i d L d Signal and ω e The Flux signals are summed to obtain a third signal, which is then input to an integrator for integration. The integrated signal is then input to a pulse width modulator to obtain the β-axis component u of the drive signal input to the inertial flywheel. β , where ω e i represents the actual rotational speed of the rotor at the output end of the inertia flywheel. d For the d-axis current at the output of the inertia flywheel that requires control calculation, L d d-axis inductance of the inertia flywheel output terminal that requires control quantity calculation, and Flux is the magnetic flux of the inertia flywheel output terminal that requires control quantity calculation.

[0073] While acquiring the β-axis component of the drive signal, the d-axis current at the output of the inertia flywheel is given by current i. dref The d-axis current i at the output terminal of the inertia flywheel, which requires control calculation. d The subtraction yields the error value, which is then input into the PID controller. After proportional, integral, and derivative operations, the controller outputs a fourth signal. This fourth signal is then compared with ω. e i q L q The signals are subtracted and then fed into an integrator for integration. The integrated signal is then fed into a pulse width modulator to obtain the α-axis component u of the drive signal input to the inertial flywheel. α , where ω e i represents the actual rotational speed of the rotor at the output end of the inertia flywheel. q L is the q-axis current at the output of the inertia flywheel, which requires control calculation. q The q-axis inductance is the output terminal of the inertia flywheel for which control calculations are required.

[0074] In the embodiments of this application, the aforementioned pulse width modulator may be an SVPWM pulse width modulator.

[0075] In summary, through the reinforcement learning-based inertial flywheel control scheme provided above, this embodiment determines whether the neural network needs to be updated again based on the rewards in all experience tuples in the current experience pool. The finally updated neural network is used as the trained neural network to obtain the action corresponding to the state quantity of the inertial flywheel output terminal that needs to be calculated for control. Based on the action and torque setpoint, the control quantity to be input to the inertial flywheel is obtained. This enables the neural network to adjust its strategy according to actual performance, more accurately predict and calculate the action corresponding to the state quantity of the inertial flywheel output terminal, and more accurately control the state of the inertial flywheel, thereby improving the control accuracy of the inertial flywheel.

[0076] Furthermore, in some embodiments, by setting the action to include a damped voltage component and a damped torque component, the problem of large rotor oscillation at the output end of the inertial flywheel can be solved when obtaining the control quantity to be input to the inertial flywheel based on the action and torque setpoint, thereby improving its resistance to power grid impact and self-stability. Even further, in some embodiments, by using a sum of rewards greater than a preset value in all experience tuples in the current experience pool as the termination condition for neural network updates, it can be ensured that the neural network reaches a certain performance level after training, while avoiding overfitting of the neural network to the training data and ineffective training of the neural network.

[0077] This application also provides a reinforcement learning-based inertia flywheel control system, which can control the inertia flywheel using the aforementioned reinforcement learning-based inertia flywheel control method 100.

[0078] Figure 7 shows an exemplary structural block diagram of an inertia flywheel control system 700 according to an embodiment of this application.

[0079] As shown in Figure 7, the system 700 includes a neural network update module 710, a data processing module 720, and a control quantity acquisition module 730. The data processing module 720 is interconnected with the neural network update module 710, and the control quantity acquisition module 730 is connected to the data processing module 720. In the embodiments of this application, the neural network update module 710, the data processing module 720, and the control quantity acquisition module 730 can be separate units or integrated into the same controller.

[0080] Specifically, the neural network update module 710 is used to store the transformation experience tuples obtained through the interactive simulation environment and the neural network into the experience pool, and randomly sample a set number of experience tuples from the experience pool at the current moment to update the neural network.

[0081] Specifically, the data processing module 720 is used to determine whether the neural network update termination condition is met based on the rewards in all experience tuples in the experience pool at the current moment. In response to meeting the neural network update termination condition, the updated neural network is used as the trained neural network. In response to not meeting the neural network update termination condition, the neural network update module is used to update the neural network again until the neural network update termination condition is met. The neural network updated during the last execution of the neural network update is used as the trained neural network.

[0082] Specifically, the control quantity acquisition module 730 is used to input the state quantity of the inertia flywheel output terminal that needs to be calculated into the trained neural network to acquire the action, and acquire the control quantity to be input to the inertia flywheel based on the action and torque set value.

[0083] When the system 700 uses the aforementioned reinforcement learning-based inertial flywheel control method 100 to control the inertial flywheel, the neural network update module 710 executes the aforementioned step S110, the data processing module 720 executes the aforementioned steps S110 and S120, and the control quantity acquisition module 730 executes the aforementioned step S140. The specific execution process can be found above and will not be repeated here.

[0084] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for inertia flywheel control based on reinforcement learning, characterized in that, include: The transformation experience multivariate (s) obtained through interactive simulation environment and neural network t ,a t ,r t ,s t+1 ) Stored in the experience pool, and a set number of experience tuples are randomly sampled from the experience pool at the current time to update the neural network, where s t Let a be the state quantity at the output of the inertial flywheel at the current time t. t For the action corresponding to the state variable at the current time t, r t s is the reward at the current time t. t+1 For the action in the next moment; Determine whether the neural network update termination condition is met based on the rewards in all experience tuples in the experience pool at the current moment. In response to the fulfillment of the neural network update termination condition, the updated neural network is designated as the trained neural network; and In response to the failure to meet the neural network update termination condition, return to the step of updating the neural network until the neural network update termination condition is met, and take the neural network updated by the last step of updating the neural network as the neural network that has been trained. The state quantity of the inertial flywheel output terminal that needs to be calculated for control is input to the trained neural network to obtain the action, and the control quantity to be input to the inertial flywheel is obtained based on the action and torque setpoint.

2. The reinforcement learning-based inertia flywheel control method according to claim 1, characterized by, The state quantities include active power, the derivative of active power, reactive power, the derivative of reactive power, terminal voltage, the derivative of terminal voltage, rotational speed, and the derivative of rotational speed.

3. The reinforcement learning-based inertia flywheel control method according to claim 1, characterized by, The action includes a damping voltage component and a damping torque component.

4. The reinforcement learning-based inertia flywheel control method according to claim 1, characterized by, The conditions for ending the neural network update include: the sum of rewards in all experience tuples in the current experience pool is greater than a preset value for the sum of rewards.

5. The reinforcement learning-based inertia flywheel control method according to claim 1 or 4, characterized by, r t obtained by a reward function, wherein the reward function is: t step is the set time step, t is the current time, ω e is the actual rotational speed of the output rotor of the inertia flywheel, ω ref is the rated rotational speed of the output rotor of the inertia flywheel.

6. The reinforcement learning-based inertia flywheel control method according to claim 1, wherein, The torque setpoint is the torque setpoint input to the output terminal of the inertia flywheel. The torque setpoint is obtained using a torque setpoint formula, which is: T is the torque setpoint, P ref is the active power output of the flywheel, ω m is the angular velocity of the flywheel output rotor, K flag is the flywheel charge-discharge flag, K flag is -1 when the flywheel is charging, K flag is +1 when the flywheel is discharging, ΔP is the power compensation term; R a R is the resistance of the phase output of the inertia flywheel.

7. The reinforcement learning-based inertia flywheel control method according to claim 3, characterized by, The control quantities include the drive signal input to the inertia flywheel and the excitation voltage input to the inertia flywheel.

8. The reinforcement learning-based inertia flywheel control method according to claim 7, characterized by, The control quantities to be input to the inertial flywheel are obtained based on the motion and torque setpoints, including: The damping torque component and the torque set value are added together to obtain the torque base value; Obtain the measured torque value corresponding to the inertia flywheel for which control quantity calculation is required, and subtract the measured torque value from the torque base value to obtain the error value; The obtained error value is input into the PID controller, and after proportional, integral and derivative operations, it is output to the gain amplifier for processing to obtain the first signal; The first signal is subtracted from the q-axis current at the output of the inertia flywheel, which is to be calculated as the control quantity, and then input into the PID controller to obtain the second signal after proportional, integral and derivative operations. The second signal and ω e i d L d Signal and ω e The Flux signals are summed to obtain a third signal. This third signal is then input to an integrator for integration. The integrated signal is then input to a pulse width modulator to obtain the β-axis component of the drive signal input to the inertial flywheel, where ω e i represents the actual rotational speed of the rotor at the output end of the inertia flywheel. d For the d-axis current at the output of the inertia flywheel that requires control calculation, L d d-axis inductance of the inertia flywheel output terminal that requires control quantity calculation, and Flux is the magnetic flux of the inertia flywheel output terminal that requires control quantity calculation.

9. The reinforcement learning-based inertia flywheel control method according to claim 5, wherein, Obtaining the control quantity to be input to the inertia flywheel based on the action and torque setpoints also includes: inputting the damping voltage component, the measured voltage value of the inertia flywheel output terminal that needs to be calculated for control quantity calculation, and the voltage setpoint value of the inertia flywheel output terminal that needs to be calculated for control quantity calculation to the excitation controller to obtain the excitation voltage.

10. A reinforcement learning based inertia flywheel control system, characterized by, The system controls an inertial flywheel using the reinforcement learning-based inertial flywheel control method as described in any one of claims 1-9, wherein the system comprises: The neural network update module is used to store the transformation experience tuples obtained through the interactive simulation environment and the neural network into the experience pool, and to randomly sample a set number of experience tuples from the experience pool at the current moment to update the neural network. The data processing module is used to determine whether the neural network update termination condition is met based on the rewards in all experience tuples in the experience pool at the current moment. In response to meeting the neural network update termination condition, the updated neural network is used as the trained neural network. In response to not meeting the neural network update termination condition, the neural network update module is used to update the neural network again until the neural network update termination condition is met. The neural network updated during the last execution of the neural network update is used as the trained neural network. The control quantity acquisition module inputs the state quantity of the inertial flywheel output terminal that needs to be calculated into the trained neural network to acquire the action, and obtains the control quantity to be input to the inertial flywheel based on the action and torque setpoint.

Images