Evaluation method

Abstract

This evaluation method evaluates the validity of an inference result (R1) output by a machine learning model (M). In this evaluation method, an output variable of the machine learning model (M) includes the inference result (R1) of a target measurement value of a virtual sensor and a validation inference result (R2). The verification inference result (R2) is an inference result of a control amount of the device or an inference result of a measurement value of an actual sensor, which can be actually measured by the device. Furthermore, this evaluation method includes: a step for acquiring output results (R1, R2) of the machine learning model (M); and a step for comparing the verification inference result (R2) with a true value and determining the validity of the inference result (R1). This makes it possible to evaluate the validity of the inference result (R1) even for operating conditions in a range not included in the training data. That is, the validity of an inference result of a machine learning model for an unmeasurable value can be verified by using a known true value or a measurable value.

Description

Evaluation method

[0001] The present invention relates to a technique for evaluating the validity of an inference result output by a machine learning model.

[0002] Conventionally, for various industrial devices, by predicting and estimating the state of the device using various sensor data and machine learning, predictive maintenance, anomaly detection, root cause analysis of malfunctions, etc. have been carried out. In recent years, the utilization of various sensor data acquired from various industrial devices has been promoted.

[0003] For example, Patent Document 1 describes a virtual inspection service system that creates a virtual inspection model considering machine differences for each device type by generating various virtual inspection models and using the virtual inspection model evaluated to have the highest accuracy among them to perform accurate virtual inspections.

[0004] Japanese Patent Application Laid-Open No. 2023-135761

[0005] However, when evaluating the validity of a machine learning model based on the estimation accuracy, as in the virtual inspection service system of Patent Document 1, an appropriate evaluation cannot always be performed. For example, when performing a virtual inspection on the operating conditions of a new device that has not been evaluated in advance, the correct data does not exist in the collected learning data. In such a case, although it is necessary to obtain new correct data in order to perform an appropriate evaluation, it is difficult to obtain the correct data. In particular, when the value desired as an output is not measurable, the correct data cannot be obtained.

[0006] The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for confirming the validity of an inference result of a value that cannot be measured by a machine learning model using a value whose true value is known or measurable.

[0007] To solve the above problems, the first invention of the present application is an evaluation method for evaluating the validity of the inference result of a machine learning model that outputs an inference result of a measurement value of a virtual sensor in a device, wherein the output variables of the machine learning model include an inference result of the measurement value of the virtual sensor and a verification inference result, wherein the verification inference result is an inference result of a control variable of the device or an inference result of a measurement value of a real sensor that can actually be measured in the device, and the method includes a) a step of acquiring the output result of the machine learning model, and b) a step of comparing the verification inference result with a true value to determine the validity of the inference result of the measurement value of the virtual sensor included in the output result.

[0008] The second invention of this application is an evaluation method of the first invention, wherein the machine learning model includes LSTM (Long Short-Term Memory).

[0009] The third invention of this application is an evaluation method for the first or second invention, wherein the machine learning model includes two output layers.

[0010] The fourth invention of this application is an evaluation method for any one of the first to third inventions, wherein the input variables of the machine learning model include the control quantity of the device.

[0011] The fifth invention of this application is an evaluation method of the fourth invention, wherein in step a), the control quantity within the range of the unlearned range is input to the machine learning model.

[0012] The sixth invention of this application is an evaluation method for any one of the first to fifth inventions, wherein in step b), the validity is determined based on the root mean square error between the confirmation reasoning result and the true value.

[0013] According to the first to sixth inventions of this application, the validity of the inference results of a machine learning model for values ​​that cannot be measured in reality can be confirmed using values ​​whose true values ​​are known or measurable.

[0014] In particular, according to the fifth invention of this application, the validity of the inference results of the measured values ​​of the virtual sensor can be evaluated when the device is operating with control quantities in a range where the machine learning model has not been trained.

[0015] This is a diagram showing an example of an applicable device. This is a diagram showing an example of sensor position during the training process of a machine learning model. This is a block diagram showing the functions of a computer. This is a diagram showing the input and output of a machine learning model. This is a flowchart showing the flow of the evaluation method for a machine learning model.

[0016] Embodiments of the present invention will be described below with reference to the drawings.

[0017] <1. Example of a device for applying a machine learning model> First, as an example of a device to which the machine learning model M to be evaluated is applied in the estimation method of the present invention, a substrate processing apparatus 1 will be described. Figure 1 is a diagram showing an overview of the substrate processing apparatus 1, which is an example of a device for applying a machine learning model.

[0018] The substrate processing apparatus 1 is a device used in the semiconductor wafer manufacturing process to process the surface of a disc-shaped substrate W (silicon wafer) by supplying a processing liquid to the surface of the substrate W. As shown in Figure 1, the substrate processing apparatus 1 comprises a chamber 10, a substrate holding unit 20 housed within the chamber 10, a rotating mechanism 30, a processing liquid supply unit 40, a processing liquid collection unit 50, and a shielding plate 60.

[0019] Chamber 10 forms a processing space for processing the substrate W. An airflow supply unit 11 is provided at the top of Chamber 10 to supply a downflow airflow.

[0020] The substrate holding section 20 is a mechanism for holding the substrate W horizontally inside the chamber 10. The substrate holding section 20 has a disc-shaped spin base 21 and a plurality of chuck pins 22. The plurality of chuck pins 22 hold the peripheral edge of the substrate W, and the substrate W is positioned with a small gap between it and the upper surface of the spin base 21. The rotation mechanism 30 is a mechanism for rotating the substrate holding section 20.

[0021] The processing liquid supply unit 40 is a mechanism that supplies processing liquid to the upper surface of the substrate W held by the substrate holding unit 20. The processing liquid supply unit 40 has an upper nozzle 41 that supplies processing liquid to the upper surface of the substrate W and a lower nozzle 42 that supplies processing liquid to the lower surface of the substrate W.

[0022] The processing liquid collection unit 50 is a mechanism for collecting the processing liquid after use. The processing liquid collection unit 50 has an inner cup 51, a middle cup 52, and an outer cup 53, each connected to a different liquid discharge passage. The inner cup 51, middle cup 52, and outer cup 53 can move independently up and down between a lower position and an upper position by a lifting mechanism (not shown in the figure). The lower position of each cup 51, 52, and 53 is the position shown in Figure 1, where the upper end is positioned below the substrate W. The upper position of each cup 51, 52, and 53 is where the upper end is positioned above the upper surface of the substrate W.

[0023] During processing of the substrate W, with one of the three cups 51, 52, or 53 of the processing liquid collection unit 50 positioned in the upper position, the substrate holding unit 20 and the substrate W are rotated by the rotating mechanism 30, while the processing liquid is supplied to the surface of the substrate W by the processing liquid supply unit 40. The processing liquid supplied to the surface of the substrate W and used to treat the substrate W is scattered outward by the centrifugal force caused by the rotation of the substrate W, collected by one of the cups 51, 52, or 53 of the processing liquid collection unit 50, and recovered through the liquid discharge channel.

[0024] The barrier plate 60 is a component used to suppress the diffusion of gas near the surface of the substrate W when performing some processing, such as drying the substrate W after supplying the processing liquid. The barrier plate 60 has a disc-shaped outer shape and is positioned horizontally above the substrate holding portion 20. The barrier plate 60 is connected to the lifting mechanism 61. When the lifting mechanism 61 is operated, the barrier plate 60 moves up and down between an upper position, which is above the upper surface of the substrate W held by the substrate holding portion 20, and a lower position, which is closer to the upper surface of the substrate W than the upper position.

[0025] When the processing liquid is supplied to the substrate W from the processing liquid supply unit 40, the shut-off plate 60 is retracted to the upper position. After the processing liquid is supplied, when the substrate W is dried, the shut-off plate 60 is lowered to the lower position by the lifting mechanism 61. Then, drying gas is blown from the outlet 62 toward the upper surface of the substrate W. At this time, the shut-off plate 60 prevents the diffusion of the gas. As a result, drying gas is efficiently supplied to the upper surface of the substrate W.

[0026] <2. About the Machine Learning Model> Next, we will describe the machine learning model M for estimating the airflow near the substrate W in the substrate processing apparatus 1 described above. The estimation method of the present invention can be used, for example, to evaluate the validity of the inference results of this machine learning model M.

[0027] This machine learning model M estimates the airflow near the substrate W from various control values ​​that are the driving conditions of the substrate processing apparatus 1 and time-series data detected by multiple airflow sensors inside the substrate processing apparatus 1. Figure 2 shows an example of the sensor positions during the learning process of the machine learning model M. Figure 3 is a block diagram showing the functions of the computer 90 that performs the learning process of the machine learning model M, the airflow prediction process around the substrate W by the machine learning model M during substrate processing in the substrate processing apparatus 1, and the evaluation process for the machine learning model M. Figure 4 shows the input and output of the machine learning model M.

[0028] As shown in Figure 2, the substrate processing apparatus 1 has 10 permanent sensors Sp0, Sp1, Sp2, Sp3, Sp4, Sp5, Sp6, Sp7, Sp8, and Sp9 installed in the chamber 10. In addition, during the training process of the machine learning model M, four temporary sensors St1, St2, St3, and St4 are installed at four points P1, P2, P3, and P4 around the substrate W for which airflow is to be estimated. The permanent sensors Sp0 to Sp9 and the temporary sensors St1 to St4 each measure three values: wind speed, azimuth angle, and depression angle at each point.

[0029] Here, the time-series data of wind speed detected by the permanent sensors Sp0 to Sp9 are referred to as wind speed data D10 to D19, the time-series data of azimuth angles detected by the permanent sensors Sp0 to Sp9 are referred to as azimuth angle data D20 to D29, and the time-series data of depression angles detected by the permanent sensors Sp0 to Sp9 are referred to as depression angle data D30 to D39.

[0030] Similarly, the time-series data of wind speed detected by temporary sensors St1 to St4 are referred to as wind speed data D41 to D44, the time-series data of azimuth angles detected by temporary sensors St1 to St4 are referred to as azimuth angle data D51 to D54, and the time-series data of depression angles detected by temporary sensors St1 to St4 are referred to as depression angle data D61 to D64.

[0031] The detection values ​​mentioned above, detected by the permanent sensors Sp0 to Sp9 and the temporary sensors St1 to St4, are input to the computer 90.

[0032] The computer 90 is an information processing device for performing the learning process of the machine learning model M, the airflow prediction process around the substrate W by the machine learning model M during substrate processing in the substrate processing device 1, and the evaluation process for the machine learning model M.

[0033] The computer 90 is electrically connected to the substrate processing apparatus 1. The computer 90 receives detection values ​​from permanent sensors Sp0 to Sp9 and temporary sensors St1 to St4, as well as control values ​​Cv1 to Cv4 when these detection values ​​are detected. The computer 90 also outputs the airflow prediction results around the substrate W output by the machine learning model M, or control values ​​based on said airflow prediction results, to the substrate processing apparatus 1. The computer 90 may also serve as the control unit for controlling the substrate processing apparatus 1.

[0034] As conceptually shown in Figure 2, the computer 90 includes a processor 901 such as a CPU, memory 902 such as RAM, and a storage unit 903 such as a hard disk drive. The storage unit 903 stores a computer program Pg for performing learning processing of the machine learning model M, airflow prediction processing around the substrate W by the machine learning model M during substrate processing in the substrate processing apparatus 1, and evaluation processing of the machine learning model M.

[0035] As shown in Figure 3, the computer 90 includes a data acquisition unit 91, a learning unit 92, an airflow prediction unit 93, and an evaluation unit 94. The functions of the data acquisition unit 91, the learning unit 92, the airflow prediction unit 93, and the evaluation unit 94 are realized by the computer 90's processor 901 operating according to a computer program Pg.

[0036] During the training process of the machine learning model M, temporary sensors St1 to St4 are installed in the chamber 10, and the airflow inside the chamber 10 is measured by the permanent sensors Sp0 to Sp9 and the temporary sensors St1 to St4 without supplying processing liquid in the substrate processing apparatus 1.

[0037] The substrate processing device 1 then inputs control values ​​Cv1 to Cv4, which are part of the operating conditions during measurement, the detection results of the permanent sensors Sp0 to Sp9, and the detection results of the temporary sensors St1 to St4 to the data acquisition unit 91 of the computer 90. Here, the control values ​​Cv1 to Cv4 may include, for example, the rotation speed of the substrate W, the flow rate of the airflow supplied by the airflow supply unit 11, the positions of each cup 51, 52, 53, and the position of the shut-off plate 60. The rotation speed of the substrate W is the rotation speed of the substrate holding unit 20 by the rotation mechanism 30. The data acquisition unit 91 then passes this input data to the learning unit 92.

[0038] The learning unit 92 performs machine learning on the machine learning model M using supervised learning. This machine learning model M is a model that takes the control values ​​of the substrate processing device 1 and / or the measured values ​​of actual sensors that can be measured as input variables, and outputs the first inference result R1 and the second inference result R2 as output variables.

[0039] As shown in Figure 4, the machine learning model M has an input layer M1 into which input variables are input, an LSTM layer M2, and two fully connected layers M3 and M4 as output layers. LSTM stands for Long Short-Term Memory. In other words, this machine learning model M is a shared multitask model with two output layers for each hidden layer.

[0040] By including the LSTM, the machine learning model M is suitable for prediction based on time-series data. Note that instead of the LSTM, many other time-series estimation models such as RNN may be used for the machine learning model M. Note that RNN means Recurrent neural network (recursive neural network).

[0041] Also, the machine learning model M is a multi-task model having a fully-connected layer M3 that outputs the first inference result R1 and a fully-connected layer M4 that outputs the second inference result R2 as output layers, but the present invention is not limited thereto. The machine learning model M may be a model that outputs both the first inference result R1 and the second inference result R2 from one output layer.

[0042] In the present embodiment, the input variables of the machine learning model M are control values Cv1 to Cv4, wind speed data D10 to D19 detected by the permanent sensors Sp0 to Sp9, azimuth data D20 to D29, and depression angle data D30 to D39. And the teacher data of the output variable in the learning process of the machine learning model M is the control value Cv1 which is one of the control values Cv1 to Cv4, and the wind speed data D41 to D44, azimuth data D51 to D54, and depression angle data D61 to D64 detected by the hypothetical sensors St1 to St4.

[0043] As a result, as shown in FIG. 4, the machine learning model M becomes an estimation model that inputs the control values Cv1 to Cv4, the wind speed data D10 to D19, the azimuth data D20 to D29, and the depression angle data D30 to D39 respectively detected by the permanent sensors Sp0 to Sp9, and outputs the first inference result R1 and the second inference result R2.

[0044] The first inference result R1 of the present embodiment is the estimated wind speeds E11 to E14, the estimated azimuths E21 to E24, and the estimated depression angles E31 to E34 at the four points P1 to P4. That is, the first inference result R1 is the inference result of the measured values of the virtual sensors (virtual airflow sensors) provided at the four points P1 to P4. Note that the estimated wind speeds E11 to E14, the estimated azimuths E21 to E24, and the estimated depression angles E31 to E34 are time-series data respectively.

[0045] Furthermore, the second inference result R2 in this embodiment is the estimated control value EC1, which is the estimated control value Cv1, and the estimated wind speed E15, which is the estimated wind speed data D19 from the permanent sensor Sp9. The second inference result R2 is a verification inference result. That is, the second inference result R2 is an output result for evaluating the validity of the inference result output by the machine learning model M. The second inference result R2 includes at least one of the inference results of the control quantity of the substrate processing apparatus 1 and the inference results of the measured values ​​of the actual sensors of the substrate processing apparatus 1 that can be measured. It is preferable to select values ​​for the verification inference result that have known true values ​​and are related to the target inference result. In the above embodiment, the control value Cv1 and wind speed data D19, which are true values, are known as verification inference results, and the estimated control value EC1 and estimated wind speed E15 are selected as the second inference result R2, which is the verification inference result, as values ​​related to each value of the first inference result R1.

[0046] In this embodiment, the control value Cv1 and wind speed data D19 included in the second inference result R2 are both included in the input variables, but the present invention is not limited thereto. The inference result included in the second inference result R2 may be an inference result of a control quantity of a device not included in the input variables, or an inference result of a measurable measurement value not included in the input variables.

[0047] Once the training process of the machine learning model M is complete, the trained machine learning model M is handed over to the airflow prediction unit 93. Also, before the substrate processing starts in the substrate processing apparatus 1, the temporary sensors St1 to St4 are removed. During substrate processing, it is not possible to detect the airflow at the four points P1 to P4 using the temporary sensors St1 to St4. In other words, it is not possible to detect the airflow around the substrate W during substrate processing. Therefore, the state of the airflow at the four points P1 to P4 around the substrate W during substrate processing is estimated by the machine learning model M.

[0048] While the substrate processing is being performed in the substrate processing apparatus 1, the control values Cv1 to Cv4 and the detection values from the permanent sensors Sp0 to Sp9 are constantly input to the data acquisition unit 91 of the computer 90 and then passed to the airflow prediction unit 93. The airflow prediction unit 93 uses the control values Cv1 to Cv4, the wind speed data D10 to D19, azimuth data D20 to D29, and elevation angle data D30 to D39 respectively detected by the permanent sensors Sp0 to Sp9 as input variables, and the estimated wind speeds E11 to E14, estimated azimuths E21 to E24, and estimated elevation angles E31 to E34 at the four points P1 to P4 which are the first inference result R1, the estimated control value EC1 obtained by estimating the control value Cv1 which is the second inference result R2, and the estimated wind speed E15 obtained by estimating the wind speed data D19 of the permanent sensor Sp9 as output.

[0049] Then, the airflow prediction unit 93 outputs the first inference result R1 to the substrate processing apparatus 1 and passes the second inference result R2 to the evaluation unit 94.

[0050] After the operation of the machine learning model M starts, when trying to confirm the validity of the estimated wind speeds E11 to E14, estimated azimuths E21 to E24, and estimated elevation angles E31 to E34 which are the first inference result R1, it is difficult to compare with the actual measurement results because the dummy sensors St1 to St4 have been removed. Therefore, the following evaluation method is used to evaluate the validity of the inference results output by the machine learning model.

[0051] <3. Evaluation Method of Machine Learning Model> Hereinafter, the evaluation method of the machine learning model M will be described while referring to FIG. 5. FIG. 5 is a flowchart showing the flow of the evaluation method of the machine learning model M.

[0052] A machine learning model that outputs inference results of values that cannot be actually measured inside an operating device, such as the state of the airflow around the substrate W in the substrate processing apparatus 1 described above, plays an important role in knowing the state inside the operating device.

[0053] However, if the equipment is operated under new operating conditions that were not anticipated during the training of the machine learning model, the accuracy of the machine learning model may decrease. However, the number of possible operating conditions for the equipment is enormous, making it difficult to acquire training data that covers all operating conditions in advance. On the other hand, it is difficult to evaluate how well the machine learning model can produce reasonable inference results for operating conditions that are not included in the training data and have not been trained.

[0054] Therefore, in this evaluation method, the output variables from the machine learning model M include not only the inference results for the target, unmeasurable values, but also the inference results for values ​​whose true values ​​are known as confirmation inference results. By comparing the confirmation inference results with their true values, the validity of the output of the machine learning model M is evaluated.

[0055] In the evaluation method shown in Figure 5, first, the airflow prediction unit 93 inputs the control values ​​Cv1 to Cv4, which are input variables, and the detected values ​​from the permanent sensors Sp0 to Sp9 into the machine learning model M, and the machine learning model M outputs a first inference result R1 and a second inference result R2 (step S1). Then, the airflow prediction unit 93 hands over the first inference result R1 to the substrate processing device 1.

[0056] The airflow prediction unit 93 then passes the second inference result R2 to the evaluation unit 94. The data acquisition unit 91 also passes the control value Cv1, which is the true value of the second inference result R2, and the wind speed data D19 from the permanent sensor Sp9 to the evaluation unit 94. As a result, the evaluation unit 94 acquires the second inference result R2 and the true value of the second inference result R2 (step S2).

[0057] Next, the evaluation unit 94 compares the second inference result R2 with the true value of the second inference result R2 to determine the validity of the inference of the first inference result R1 (step S3). Specifically, for example, the evaluation unit 94 calculates the RMSE between each value of the second inference result R2 and the true value of that value. RMSE stands for root-mean-square error. If the calculated RMSE is smaller than a predetermined threshold, the evaluation unit 94 determines that the first inference result R1 is valid. On the other hand, if the calculated RMSE is larger than a predetermined threshold, the evaluation unit 94 determines that the first inference result R1 is not valid.

[0058] Furthermore, the value calculated to evaluate the validity of the inference is not limited to RMSE. Also, the evaluation result output from the evaluation unit 94 may be binary information indicating whether it is valid or not, or it may be an evaluation value indicating the degree of validity.

[0059] <4. Modifications> Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment.

[0060] In the above embodiment, the second inference result R2, which is the verification inference result, included both the inference result of the control quantity of the substrate processing apparatus 1 and the inference result of the measured value of the actual sensor that can be measured within the substrate processing apparatus 1. However, the present invention is not limited thereto. The verification inference result may consist only of the inference result of the control quantity of the apparatus, or only of the inference result of the measured value of the actual sensor that can be measured within the apparatus. Furthermore, the verification inference result may contain only one value, or three or more values.

[0061] Furthermore, in the above embodiment, the machine learning model M to be evaluated was for estimating the airflow near the substrate W in a substrate processing apparatus. However, the present invention is not limited to this. The machine learning model to be evaluated may output inference results of values ​​that cannot be measured in other devices such as printing apparatuses or image processing apparatuses. Values ​​that cannot be measured in reality are, for example, measured values ​​of virtual sensors in locations where it is difficult to install a temperature sensor, such as the temperature at a location where it is difficult to install a temperature sensor. In addition, values ​​that cannot be measured in reality may be, for example, values ​​for which accurate measurement itself is difficult. Furthermore, the machine learning model to be evaluated does not necessarily have to be used in any device.

[0062] Furthermore, the elements that appear in the above embodiments and modifications may be combined as appropriate, to the extent that no contradictions arise.

[0063] 1: Substrate processing unit 90: Computer 91: Data acquisition unit 92: Learning unit 93: Airflow prediction unit 94: Evaluation unit Cv1, Cv2, Cv3, Cv4: Control values ​​D10-D19: Wind speed data (permanent sensor) D20-D29: Azimuth angle data (permanent sensor) D30-D39: Depression angle data (permanent sensor) D41-D44: Wind speed data (temporary sensor) D51-D54: Azimuth angle data (temporary sensor) D61-D64: Depression angle data (temporary sensor) E11-E14: Estimated wind speed E21-E24: Estimated azimuth angle E31-E34: Estimated depression angle M: ​​Machine learning model M1: Input layer M2: LSTM layer M3, M4: Fully connected layers Pg: Computer program R1: First inference result R2: Second inference result Sp0-Sp9: Permanent sensors St1-St4: Temporary sensors

Claims

1. An evaluation method for evaluating the validity of the inference results of a machine learning model that outputs inference results of measured values ​​of a virtual sensor within a device, wherein the output variables of the machine learning model include: the inference results of the measured values ​​of the virtual sensor and a verification inference result, the verification inference result being an inference result of a control variable of the device or an inference result of a measured value of a real sensor that can actually be measured by the device, and the evaluation method comprising: a) a step of acquiring the output results of the machine learning model; b) a step of comparing the verification inference result with a true value to determine the validity of the inference results of the measured values ​​of the virtual sensor included in the output results.

2. An evaluation method according to claim 1, wherein the machine learning model includes LSTM (Long Short-Term Memory).

3. An evaluation method according to claim 1 or claim 2, wherein the machine learning model includes two output layers.

4. An evaluation method according to any one of claims 1 to 3, wherein the input variables of the machine learning model include the control variables of the device.

5. An evaluation method according to claim 4, wherein in step a), the control quantity within the range of unlearned data is input to the machine learning model.

6. An evaluation method according to any one of claims 1 to 5, wherein in step b), the validity is determined based on the root mean square error between the confirmation inference result and the true value.

Images