Training device and control device

Abstract

This training device comprises: an input unit that inputs training data, which is experimental data or a numerical calculation result regarding control of nuclear fusion plasma; and a training unit that trains, by means of offline reinforcement learning using the training data, a control model which outputs a control instruction.

Description

Learning device and control device

[0001] This invention relates to a technology for controlling nuclear fusion plasma.

[0002] Controlling a fusion plasma device requires predicting its behavior and adjusting parameters such as density and temperature. The operation phases of a fusion plasma device include a phase to start plasma operation, a phase to heat the plasma to a state where a fusion reaction occurs, a phase to maintain a high-temperature, high-density plasma, and a phase to lower the plasma and end operation. Control is required throughout all of these phases.

[0003] While control would be possible if all possible physical phenomena during such operating phases could be accurately calculated, understanding the interactions of multiple phenomena would require an enormous amount of computation, making it impractical. In response to this, a machine learning approach has been reported that uses methods to learn from experimental data and numerical calculation results to estimate the temporal changes in plasma behavior.

[0004] Predicting disruptive instabilities in controlled fusion plasmas through deep learning, NATURE | VOL568 | 25APRIL 2019 https: / / doi.org / 10.1038 / s41586-019-1116-4Decision Transformer: Reinforcement Learning via Sequence Modeling Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch https: / / doi.org / 10.48550 / arXiv.2106.01345

[0005] Regarding the behavior of a fusion plasma, Non-Patent Document 1 discloses a method of learning experimental data and estimating it in a time series. However, in a large experimental device (such as a tokamak type device, a helical type device, a linear type device, etc.), using online reinforcement learning for learning has a risk of damaging the device in case of experimental failure. It has been difficult to construct an estimation system that estimates the behavior of such a plasma and further operates including control instructions.

[0006] The present invention has been made in view of the above points, and an object thereof is to provide a technology that enables learning the behavior of a fusion plasma without affecting a fusion plasma device and performing control of the fusion plasma based on the learned results.

[0007] According to the disclosed technology, there is provided a learning device including an input unit that inputs learning data which is experimental data or numerical calculation results regarding control of a fusion plasma, and a learning unit that performs learning of a control model that outputs a control instruction by offline reinforcement learning using the learning data.

[0008] According to the disclosed technology, there is provided a technology that enables learning the behavior of a fusion plasma without affecting a fusion plasma device and performing control of the fusion plasma based on the learned results.

[0009] It is a diagram showing a configuration example of a learning device 100. It is a flowchart for explaining the operation of the learning device 100. It is a diagram showing a configuration example of a control device 200. It is a flowchart for explaining the operation of the control device 200. It is a diagram for explaining Example 3. It is a diagram showing a hardware configuration example of the device.

[0010] Hereinafter, embodiments (these embodiments) of the present invention will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments.

[0011] (Overview of Embodiment) In this embodiment, in order to solve the above-described problems, a learning device 100 described below uses an offline reinforcement learning method to learn a control model that estimates the behavior of plasma and outputs a control instruction using already acquired numerical calculation results or experimental data. A control device 200 estimates the behavior of plasma using the learned control model and outputs a control instruction for the plasma. By this technique, it is possible to establish a plasma control method through appropriate reward design.

[0012] Note that the offline reinforcement learning algorithm used in the learning device 100 is not limited to a specific algorithm, but in this embodiment, it is assumed that the algorithm of Decision Transformer disclosed in Non-Patent Document 2 is used.

[0013] Hereinafter, the configurations and operations of a learning device 100 that learns a control model using offline reinforcement learning and a control device 200 that controls plasma using the control model will be described using Examples 1 to 6. Example 1 is the most basic example, and Examples 2 and later explain the differences from Example 1 (or the points added to Example 1).

[0014] In this example, separate devices are used in the learning phase and the inference phase, but the same device may be used in the learning phase and the inference phase. For example, the learning device 100 may include a function of performing inference and control output using the learned control model. Also, both the learning device 100 and the control device 200 may be called information processing devices.

[0015] [Example 1] (Device Configuration and Operation Flow in Learning Phase) FIG. 1 shows a configuration example of the learning device 100. As shown in FIG. 1, the learning device 100 includes an input unit 110, a learning unit 120, and an output unit 130. The learning unit 120 holds a control model 300 to be learned.

[0016] In Example 1, it is assumed that Decision Transformer (Non-Patent Document 2) is used. Therefore, the control model 300 is a neural network model including an embedding layer, a causal transformer layer, a linear decoder layer, etc. disclosed in Non-Patent Document 2.

[0017] The operation of the learning device 100 will be described according to the procedure of the flowchart in FIG. 2. In S101, learning data is input from the input unit 110. The learning data is time-series data of a fusion plasma, which is experimental data or numerical calculation results. The learning data is input to the learning unit 120.

[0018] More specifically, the learning data set used for offline reinforcement learning is time-series data of performance parameters such as the temperature, density, or shape of the fusion plasma during device operation. The learning data set also includes parameters related to control, such as heating power, heating position, fuel injection amount, or externally applied magnetic field strength.

[0019] In S102, the learning unit 120 learns (optimizes) the parameters of the control model 300 by offline reinforcement learning using the input learning data. In S103, the output unit 130 outputs the learned control model 300 (specifically, model parameters, hyperparameters, etc.). The learned control model 300 is used in the inference phase.

[0020] <Detailed Example of Learning Process> The learning process executed by the learning unit 120 will be described in more detail. First, as learning data (time-series data), the trajectory τ = {R １ , s １ , a １ ,...,, R ｔ－１ , s ｔ－１ , a ｔ－１ , R ｔ , s ｔ , a ｔ ,...,, R Ｔ , s Ｔ , a Ｔ} is input to the learning unit 120.

[0021] t (1 to T) is the time. sｔ This represents the state of the plasma at time t. The state of the plasma refers to, for example, either or both of the plasma's temperature distribution and / or plasma's density distribution.

[0022] a ｔ is the control variable (action in reinforcement learning) for the plasma at time t. Examples of control variables for the plasma include fuel input, heating power, heating location, and externally applied magnetic field strength. Note that the control variables may be only a part of these, or may include other variables as well.

[0023] R ｔ is an immediate reward r ｔ Rather, it is the sum of future rewards (sum of immediate rewards) to be obtained from t+1 onward. The learning unit 120 (specifically the Decision Transformer) learns how this sum of future rewards will be determined by the action selection. From the experimental data obtained offline (or the results of numerical calculations), the reward (immediate reward) at each time point in the operation of the device can be calculated, and its sum can also be calculated, so the R of each t ｔ This can be set in advance. The action selection during inference can be changed in conjunction with whether this reward sum setting is high or low.

[0024] Immediate reward r at time t ｔ For example, a smaller difference between the state at time t and the target plasma performance parameter results in a better value (e.g., a larger value). Also, as will be explained in detail in Example 2 later, r ｔ In some cases, this is set based on control variables such as heating power.

[0025] The learning unit 120 provides {R １ ,s １ , a １ ,.... , R ｔ－１ ,s ｔ－１ , a ｔ－１ , R ｔ ,s ｔ When} is entered, the control model 300 will a ｔThe parameters of the control model 300 are trained to output (from t=1 to T). In other words, the output from the control model 300 and a ｔ The learning process is designed to minimize the error between the two values. For example, backpropagation can be used for learning.

[0026] (Device configuration and operation flow of the inference phase) Figure 3 shows an example of the configuration of a control device 200 that performs inference (and control). As shown in Figure 3, the control device 200 includes an input unit 210, an inference unit 220, and an output unit 230.

[0027] The inference unit 220 holds a trained control model 300. Alternatively, the inference unit 220 can be considered as the control model 300 itself. The trained control model 300 is the control model 300 trained by the aforementioned learning device 100.

[0028] The operation of the control device 200 will be explained according to the steps in the flowchart in Figure 4. In S201, input data is received from the input unit 210. The input data is the state (observation result) for the control.

[0029] In S202, the inference unit 220 inputs input data to the control model 300 and obtains output data from the control model 300. The output data is the control variable (action). In S203, the output unit 230 outputs the control variable (control instruction) to the fusion device. This executes the control of the plasma.

[0030] <Detailed Example of Inference Processing> The inference processing will be explained in more detail. In the inference unit 220, the control model 300 is s １ The input is a, and the output from the control model 300 is a １ Then, the plasma control is performed, s ２ We observe this. Next, in the control model 300, s ２ The input is a, and the output from the control model 300 is a ２ Then, the plasma control is performed, s ３ Observe the following. Repeat this process. In other words, the control model 300 is s ｔ The input is a, and the output from the control model 300 is a ｔThen, the plasma control is performed, s ｔ＋１ The process of observing is repeated.

[0031] In other words, the control instruction output from the inference unit 220 is a control value to be input to the device after the time of the input data, with the time step width of the time series data as the minimum unit (a above). ｔ )

[0032] In the inference phase, s ｔ This refers to the state (observation result) obtained by observing the temperature distribution and density distribution of the plasma at time t. In this case, the observation result may be the result obtained when actual control is performed, or it may be the result obtained by numerical calculation.

[0033] a ｔ These are control variables (actions) output from the control model 300, such as the amount of fuel input, heating power, heating position, and externally applied magnetic field strength, at time t.

[0034] [Example 2] Next, Example 2 will be described. In Example 2, the reward in offline reinforcement learning during the learning phase is set so that the control variables (e.g., fuel input amount, heating power, heating position, externally applied magnetic field strength, etc.) are used efficiently.

[0035] For example, given experimental data where the required plasma temperature is achieved by using the heating power to the limits of the device, we will explore whether it is possible to realize a control method that can achieve equivalent performance with less heating power using offline reinforcement learning.

[0036] In other words, since using the heating power to its maximum is something to be avoided when considering the efficiency of the device, the reward design is such that the reward decreases when a lot of heating power is used. In this case, the reward design is based on the r of each time step that forms the sum of the rewards. ｔ The key is to design it properly. For example, experimental data a that maximizes the use of heating power. ｔ r for ｔ A penalty is imposed (for example, r obtained based on the state). ｔ (Subtract a certain value from it).

[0037] [Example 3] Next, Example 3 will be described. In Example 3, multiple control models learned by offline reinforcement learning are prepared, and the operation is optimized by selecting the weighting (contribution rate) between the control models according to the plasma operation stage.

[0038] In other words, a control device 200 is constructed that allows the selection of a control model based on the operating stage of the fusion plasma, using two or more control models obtained through offline reinforcement learning. In this case, a single control model may be selected, or the system may be configured to distribute the contribution rates of multiple models.

[0039] In the learning of each of the multiple control models by the learning device 100, a different value (e.g., plasma performance value) is set as the reward for each model. Alternatively, the control models may be learned for each plasma operation stage. For example, control model 1 may be learned using experimental data from operation stage 1, and control model 2 may be learned using experimental data from operation stage 2.

[0040] Figure 5 shows an example of using multiple control models with weights. In the example in Figure 5, the inference unit 220 of the control device 200 includes control model A, control model B, ..., control model X.

[0041] As for how to use multiple control models, one control model may be selected (switched) from among the multiple control models based on an external instruction, or the weighted sum of the outputs from the multiple control models may be used as the output from the inference unit 220.

[0042] When selecting one control model from multiple control models, for example, the inference unit 220 receives information indicating the operating stage of the fusion plasma from an external source via the input unit 210, and selects and uses a control model suitable for that operating stage.

[0043] When using weighted sums, for example, W ａ +W ｂ +.... +W ｘAssuming that = 1, and the outputs from control model A, control model B, ..., control model X are output A, output B, ..., output X, then "W ａ *Output A+W ｂ *Output B+.... +W ｘ * Output X is the output (control instruction) from the inference unit 220.

[0044] Furthermore, the inference unit 220 receives information indicating the operating stage of the fusion plasma from an external source via the input unit 210, and, according to that operating stage, assigns a weight (W ａ , W ｂ .... W ｘ You may adjust the following.

[0045] [Example 4] Next, Example 4 will be described. Example 4 is an example of reward design in the learning phase. As mentioned above, the reward R in offline reinforcement learning ｔ is an immediate reward r ｔ It is the sum over time. The reward in the reward design described below is an immediate reward r ｔ That is the case.

[0046] In reward design, rewards may be set based on the "target value, range of variation, or distribution" of a single parameter such as the "temperature or density" of the plasma, or they may be set based on a combination of multiple parameters.

[0047] As a more specific example, for the single parameter mentioned above, the reward could be an indicator showing whether the value of the plasma's "temperature or density," or the range of variation in the plasma's "temperature or density," falls within a certain range. For the distribution, a Gaussian distribution or a shape obtained from certain experimental data could be used, and the difference from that shape could be set as the reward.

[0048] Regarding combinations of multiple parameters, for example, by adding the above indicators (let's call them A and B), r ｔ =w １ *A+w ２ *By adjusting the weight w as shown in B, it can be expressed as a single indicator.

[0049] Furthermore, the reward design in offline reinforcement learning during the learning phase may be divided into parameters for physical phenomena that change over short periods of time and parameters for the long duration of overall plasma operation. This allows for the suppression of sudden phenomena and the optimization of long-term plasma maintenance.

[0050] For example, in experimental data, the effect of heating power on temperature rise can cause changes in measured values ​​over short periods of time, so that temperature (plasma temperature) is set as one parameter. In addition, an index proportional to the length of time the plasma temperature can be maintained is set as an additional parameter, and the results are evaluated as a sum.

[0051] For example, the reward r for plasma temperature ｔ r1 ｔ The reward r is an index proportional to the length of time the plasma temperature can be maintained. ｔ r2 ｔ And the weight lol １ , lol ２ Then, the reward r ｔ =w １ *r1 ｔ +w ２ *r2 ｔ This is the result.

[0052] [Example 5] Next, Example 5 will be described. In the explanation so far, the reward R in the learning phase has been described. ｔ While sequential values ​​are set, the reward is not limited to sequential values. In Example 5, the reward can also be set to reflect the overall success or failure of the operation, such as whether the operation was completed successfully.

[0053] For example, since the learning unit 120 learns behavior based on the sum of rewards, for experimental data where the operation was completed normally, the reward R at the last time point is used. ｔ By setting a large value, it becomes possible to learn from the differences between this data and other failed or interrupted data.

[0054] [Example 6] Next, Example 6 will be described. Example 6 is an example of the data used for learning in the learning phase.

[0055] The data used for training can be one-dimensional time-series data, or two-dimensional time-series data of physical quantities such as the cross-sectional shape of a plasma or the distribution of density, temperature, and magnetic field (e.g., intensity maps). Furthermore, data representing three-dimensional or more three-dimensional objects may also be used for training.

[0056] Also, the reward value (r ｔ In the calculation of the reward, one-dimensional information, two-dimensional information, or data representing three-dimensional or more objects may be used. However, since a scalar value must be used as the reward value, for example, features can be extracted from two-dimensional information using a method such as CNN, and the reward value can be set using those features.

[0057] (Example Hardware Configuration) Any of the devices described in this embodiment (learning device, inference device, information processing device, etc.) can be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0058] In other words, the device can be realized by using hardware resources such as the CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as portable memory), saved, and distributed. It can also be provided via a network, such as the Internet or email.

[0059] Figure 6 shows an example of the hardware configuration of the computer described above. The computer in Figure 6 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by bus B. The computer may also be equipped with a GPU.

[0060] The program that enables processing on the computer is provided on a recording medium 1001, such as a CD-ROM or memory card. When the recording medium 1001 containing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it may also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files and data.

[0061] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when a program startup command is received. The CPU 1004 implements the functions related to the memory device 1003 according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc., based on a program. The input device 1007 consists of a keyboard and mouse, buttons, or a touch panel, etc., and is used to input various operation commands. The output device 1008 outputs the calculation results.

[0062] (Effects of the Embodiment) As described above, the technology according to this embodiment makes it possible to control large fusion devices that are unsuitable for online reinforcement learning, as well as novel devices with limited experimental data, using offline reinforcement learning by utilizing existing numerical calculation results and experimental data.

[0063] Furthermore, since offline reinforcement learning can be performed using existing numerical calculations and experimental data even with conventional equipment, it becomes possible to efficiently consider experimental scenarios without having to conduct additional experiments.

[0064] The following additional information is disclosed regarding the embodiments described above.

[0065] <Notes> (Note 1) A learning device comprising: an input unit for inputting learning data which is experimental data or numerical calculation results regarding the control of a fusion plasma; and a learning unit for learning a control model that outputs control instructions by offline reinforcement learning using the learning data. (Note 2) The learning device according to Note 1, wherein the learning data is time-series data including performance parameters indicating the state of the fusion plasma and parameters related to the control of the fusion plasma. (Note 3) The learning device according to Note 2, wherein the learning data further includes a reward, and the reward is a value set based on the performance parameters and a target value. (Note 4) A control device comprising an inference unit that outputs control instructions using the control model learned by the learning device according to any one of Notes 1 to 3.

[0066] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims.

[0067] 100 Learning device 200 Control device 110, 210 Input unit 120 Learning unit 130, 230 Output unit 220 Inference unit 300 Control model 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A learning device comprising: an input unit that inputs learning data which is experimental data or numerical calculation results regarding the control of a nuclear fusion plasma; and a learning unit that learns a control model that outputs control instructions by offline reinforcement learning using the learning data.

2. The learning device according to claim 1, wherein the learning data is time-series data including performance parameters indicating the state of the fusion plasma and parameters related to the control of the fusion plasma.

3. The learning device according to claim 2, wherein the learning data further includes a reward, the reward being a value set based on the performance parameter and the target value.

4. A control device comprising an inference unit that outputs a control instruction using the control model learned by the learning device described in any one of claims 1 to 3.

Images