Image machine learning device, image processing system provided with image machine learning device, and image machine learning method
Patent Information
- Application Number
- PCT/JP2025/009001
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-03-11
- Publication Date
- 2026-09-17
Smart Images
Figure JP2025009001_17092026_PF_FP_ABST
Abstract
Description
Image machine learning device, image processing system equipped with image machine learning device, image machine learning method
[0001] The present invention relates to an image machine learning device, an image processing system equipped with an image machine learning device, and an image machine learning method.
[0002] In semiconductor inspection equipment, when measuring the length of manufactured semiconductor integrated circuits, images captured by electron microscopes or similar devices are processed using machine learning models to increase their resolution. For example, images of semiconductor integrated circuits created by semiconductor manufacturing equipment are used as a dataset to train a machine learning model, and the trained machine learning model is then incorporated into the semiconductor inspection equipment for image processing during operation.
[0003] However, the trends (patterns) of the dataset can change between the training and operational phases, which can lead to performance degradation of the machine learning model. Therefore, continuing to use the same machine learning model for extended periods could compromise the reliability of semiconductor testing. To address this, when performance degradation is observed in a machine learning model, measures such as retraining or switching to a different model are taken to maintain performance.
[0004] Conventionally, an example of this type of invention is described in International Publication No. 2021 / 084623 (Patent Document 1). Specifically, Patent Document 1 describes "an information processing device comprising: a generation unit that generates a plurality of learning models with different properties based on each learning data included in a first learning dataset to which labels indicating correct answer information are assigned; a generation unit that generates a second learning dataset including a plurality of learning data using the estimation results of a learning model whose estimation accuracy is equal to or greater than the predetermined standard when the estimation accuracy of the labels for the input data to be estimated by any of the plurality of learning models falls below a predetermined standard; and an execution unit that retrains the learning model whose estimation accuracy has fallen below the predetermined standard using the second learning dataset." (Claim 10).
[0005] International Publication No. 2021 / 084623
[0006] In the invention described in Patent Document 1, performance degradation of a machine learning model is detected and retrained. However, when updating to a model with corrected performance degradation after detecting the degradation, it was not always easy to confirm robustness against the performance degradation factors.
[0007] To solve the above-mentioned problems, the machine learning device of the present invention includes a dataset storage unit storing a first dataset, a first model trained using the first dataset, a degradation detection unit that detects degradation of the inference results obtained by the first model on an image to be inferred from an external device, a dataset processing unit that processes the first dataset to obtain a second dataset based on the degradation detection results from the degradation detection unit, a retraining dataset storage unit that stores the second dataset, and a second model trained using the second dataset. The device obtains first and second inference results obtained by inferring data from the dataset storage unit and the retraining dataset storage unit using the first model, and third and fourth inference results obtained by inferring data from the dataset storage unit and the retraining dataset storage unit using the second model, and presents an evaluation result that evaluates the change between the first and second inference results and the change between the third and fourth inference results.
[0008] According to the machine learning device of the present invention, when a performance degradation of a machine learning model is detected and the model is updated with a corrected version, two types of verification are performed: normal verification and differential verification. This ensures that robustness against performance degradation factors can be reliably confirmed.
[0009] Other issues, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention.
[0010] This is a diagram illustrating the configuration of an image processing system equipped with a machine learning device in this embodiment. This is a diagram illustrating the functional configuration of the machine learning device in this embodiment. This is a diagram illustrating the robustness verification method. This is a diagram illustrating the robustness verification method. This is a diagram illustrating the robustness verification method. This is a diagram illustrating the relationship between inferred images before and after model modification. This is a flowchart of the overall processing of the machine learning device in this embodiment. This is a flowchart of the degradation detection process of the machine learning device in this embodiment. This is a flowchart of the data processing process of the machine learning device in this embodiment. This is a flowchart of the model retraining process of the machine learning device in this embodiment. This is a flowchart of the normal verification process of the machine learning device in this embodiment. This is a flowchart of the difference verification process of the machine learning device in this embodiment. This is a flowchart of the model update process of the machine learning device in this embodiment.
[0011] The embodiments (examples) for carrying out the present invention will be described in detail below with reference to the drawings as appropriate. The embodiments are illustrative examples for explaining the present invention, and have been omitted and simplified as appropriate for clarity of explanation. The present invention can also be carried out in various other forms. Unless otherwise specified, each component may be singular or plural.
[0012] The position, size, shape, and extent of each component shown in the drawings may not represent the actual position, size, shape, and extent in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, and extent disclosed in the drawings. Also, for explanatory purposes, the same component may be described using different reference numerals in different drawings.
[0013] Examples of various types of information may be described using terms such as "table" and "list," but these types of information may also be represented by other data structures. For example, various types of information such as "XX table" and "XX list" may also be referred to as "XX information." When describing identification information, terms such as "identification information," "identifier," "name," "ID," and "number" are used, but these terms are interchangeable.
[0014] In the following explanation, when describing processing by programs, the focus may be on programs and functional units, but the main hardware component is the processor, or the information processing unit (computer) that includes such a processor. The information processing unit executes processing according to the program read into memory, using resources such as memory and communication interfaces as appropriate, through the processor. In addition to the CPU, a GPU (Graphical Processing Unit) or DSP (Digital Signal Processor) may also be used as the processor. Furthermore, the processing to realize the function is not limited to software program processing; it can also be implemented with dedicated circuits. Applicable dedicated circuits include FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits).
[0015] Figure 1A is a diagram of the configuration of an image processing system equipped with an image machine learning device in this embodiment, comprising an inspection device 180 that processes and inspects captured images of the microstructure of manufactured semiconductor integrated circuits, etc., using a machine learning model, and a machine learning device 100 that trains a machine learning model using captured images from the inspection device 180 as a dataset, provides the machine learning model to the inspection device 180, and, when performance degradation of the machine learning model is detected, provides a retrained and corrected machine learning model to the inspection device 180. The machine learning device 100 includes a main memory 110, an auxiliary memory 120, an arithmetic processing unit 130, an input / output interface 140, a communication interface 150, and bus signal lines 160 connecting them.
[0016] The main memory 110 includes a learning unit 111, an inference execution unit 112, a degradation detection unit 113, a dataset processing unit 114, a normal verification unit 115, a difference verification unit 116, a model update unit 117, and a result display unit 118. The degradation detection unit 113 also includes a shape change detection unit 113a, a brightness contrast detection unit 113b, a noise detection unit 113c, and a sharpness detection unit 113d.
[0017] Each of these parts in the main memory 110 is an application program (software), and these parts function when the arithmetic processing unit 130, which will be described later, reads and executes these programs.
[0018] The auxiliary storage device 120 includes a dataset storage unit 121, a retraining dataset storage unit 122, an inference target image storage unit 123, a model storage unit 124, a verification result storage unit 125, and a degradation detection result storage unit 126. The degradation detection result storage unit 126 also includes an image correlation value 126a, a brightness difference value 126b, a noise value 126c, and a sharpness value 126d, corresponding to each part of the degradation detection unit 113.
[0019] The arithmetic processing unit 140 is a processor unit of a computer device, and in addition to reading and executing each program stored in the main memory 110 described above, it also performs other controls on the machine learning device 100 itself based on control programs (not shown) and performs processing based on predetermined programs. In the processing of the machine learning device 100 in this embodiment, in cases where a specific processing entity is not specified in the description, the arithmetic processing unit 130 performs the processing by executing a predetermined program.
[0020] The input / output interface 140 is an interface with input / output devices such as a display device (not shown) and a keyboard / mouse, and provides a user interface function for inputting predetermined information to the machine learning device 100 and displaying processing results from the machine learning device 100.
[0021] The communication interface 150 provides wired or wireless interface functions for acquiring inference target images etc. from inspection equipment (not shown), and for connecting to other devices or the cloud to obtain and upload information.
[0022] Next, the overall processing in the image processing analysis apparatus 100 will be described in detail mainly with reference to FIG. 1B. FIG. 1B is a functional configuration diagram illustrating the overall processing, data flow, and the like in the machine learning apparatus 100 of the present embodiment. In the following description, the explanation will be given with FIG. 1B as the main focus, and with reference to FIG. 1A and other drawings as appropriate.
[0023] A dataset storage unit 121 stores, as in-operation model data, learning data 121a and verification data 121b used when generating the in-operation model (Y) 124a. A learning unit 111 performs learning (111a) using the learning data 121a, thereby generating the in-operation model (Y) 124a, which is stored in a model storage unit 124.
[0024] An image output from an inspection apparatus 180 is stored in an inference target image storage unit 123, and as an inference target image (B0), inference (112c) is performed by an inference execution unit 112 using the in-operation model (Y) 124a stored in the model storage unit 124, and an inference result (Y(B0)) 171 is output. For the inference result (Y(B0)) 171, a degradation detection unit 113 detects inference performance degradation, and detects the content and extent of the degradation.
[0025] The performance degradation determination in the degradation detection unit 113 is performed from the viewpoints of shape change, luminance contrast, noise, and sharpness. For these items, a shape change detection unit 113a, a luminance contrast detection unit 113b, a noise detection unit 113c, and a sharpness detection unit 113d of the degradation detection unit 113 respectively perform degradation detection processing, and the results (presence / absence of degradation and the extent (numerical value) of degradation) are stored in a degradation detection result storage unit 126 (image correlation value 126a, luminance difference value 126b, noise value 126c, sharpness value 126d).
[0026] For example, the luminance contrast detection unit 113b measures the luminance contrast of an inferred image, and determines that "there is performance degradation" when a difference (luminance difference) obtained by comparison with a separately acquired correct image is larger than a predetermined threshold, and stores the determination result and the numerical value of the luminance difference in the luminance difference numerical value 126b of the degradation detection result storage unit 126. That is, in this case, as a factor of the performance degradation, the luminance contrast of the inference target image (B0) changes in tendency from the luminance contrast of the dataset at the time of learning and increases, and the in-operation model (Y) 124a is in a state where it cannot cope with the inference target image having high luminance contrast.
[0027] When the degradation detection unit 113 detects performance degradation of the inference result (Y(B0)) 171, the dataset processing unit 114 reads the degree of the performance degradation from the degradation detection result storage unit 126, processes the in-operation model data in the dataset storage unit 121 accordingly, and stores the processed data in the retraining dataset storage unit 122. For example, when "decrease in luminance contrast" is detected as performance degradation, and the degree of the performance degradation is a deficiency of (30 units) in luminance contrast, the in-operation model data in the dataset storage unit 121 is processed to increase the luminance contrast of the image data by (30 units), and then stored in the retraining dataset storage unit 122.
[0028] The learning unit 111 performs learning (111b) using the training data 122a in the retraining dataset storage unit 122, thereby generating a modified model (X) 124b compatible with high luminance contrast, and stores the generated modified model in the model storage unit 124.
[0029] The robustness of the generated modified model (X) 124b is verified by the normal verification unit 115 and the difference verification unit 116. That is, the normal verification unit 115 causes the degradation detection unit 113 to check whether there is any performance degradation (whether the performance degradation has been eliminated) with respect to the inference result (X(B0)) 172 obtained by the inference execution unit 112 performing inference (112d) on the inference target image (B0) from the inference target image storage unit 123 using the modified model (X) 124b.
[0030] Meanwhile, the difference verification unit 116 uses four types of inference results, appropriately selecting and discarding them: the inference result (Y(A0)) 173 obtained by the inference execution unit 112 using the operational model (Y) 124a to infer (112a) data (A0) from the verification data 121b of the dataset storage unit 121; the inference result (Y(A1)) 174 obtained by inferring (112b) data (A1) from the verification data 122b of the retraining dataset storage unit 122; the inference result (X(A0)) 175 obtained by inferring (112e) data (A0) using the modified model (X) 124b; and the inference result (X(A1)) 176 obtained by inferring (112f) data (A1).
[0031] Figure 2A illustrates an example of the verification method, in which the relationship between the inference results Y(A0) and Y(A1) obtained by the operational model (Y) and the inference results X(A0) and X(A1) obtained by the modified model (X) is confirmed for the unprocessed data (low brightness) A0 and the processed data A1 (high brightness) obtained by processing (increasing) the brightness of A0 by (+n1).
[0032] First, in a simplified manner, if X(A0) and X(A1) satisfy the relationship f(x) = mx + k ... (Equation 1) (Figure 2A (1)) with respect to a predetermined slope coefficient "m", the modified model X can be judged as "passing" (Verification 1-1). Here, the coefficient "m" may be a value assumed in advance, or a value estimated from past performance data may be used. Furthermore, even if the relationship is not strictly as in (Equation 1), it may be judged as "passing" if it falls within a certain range (-e0 to +e0) that deviates from the straight line shown by (Equation 1).
[0033] Furthermore, since the slope "n" between Y(A0) and Y(A1) should be different from "m" (n ≠ m), we can calculate the slope "m'" between X(A0) and X(A1). If "m'" is equal to "n" (m' = n), the modified model X is judged to be "failure," and if (m' ≠ n), the modified model X is judged to be "not failure" (Verification 1-2). In other words, although judgment is possible with (Verification 1-1), the accuracy of the judgment can be improved by adding (Verification 1-2).
[0034] The above determination method has the advantage that the number of determinations is small, that is, the number of processes is small and the method is simple. On the other hand, by chance, the slope of Y(A0) and Y(A1) becomes "m", and there may be a contradictory result that although the result obtained in "Verification 1-1" is "pass", the result obtained in "Verification 1-2" is "fail".
[0035] Therefore, as the data (A1) from the verification data 122b in the retraining dataset storage unit 122, instead of a single piece of data, a plurality of pieces of data (A1 １ , A1 ２ , ... A1 ｎ ) are used, whereby the verification accuracy can be improved.
[0036] That is, as shown in FIG. 2B, for a plurality of points (A0, A1 １ , A1 ２ ), it is determined whether X(A0), X(A1 １ ), X(A1 ２ ) are on the same straight line. Specifically, for X(A1 １ ), X(A1 ２ ), the slopes (m1, m2) with respect to X(A0) are obtained respectively, and it is checked whether they are equal to the coefficient "m" in formula 1 (m1=m2=m). If they are equal, it can be determined as "pass" (Verification 2-1).
[0037] Also, similarly to (Verification 1-2), regarding the slope "n" between Y(A0) and Y(A1 １ ) and the slope "m" between X(A0) and X(A1 １ ), when m≠n, the corrected model (X) 124b can be determined as "not failed". However, even if m=n by chance, regarding the slope "k" between Y(A0) and Y(A1 ２ ), the probability that m=k by chance is extremely low. Therefore, if the slopes "m1" and "m2" (=m) are not equal to either of the slopes "n" and "k" ((m1(=m2))≠n, and (m1(=m2))≠k), the accuracy of the determination that the corrected model X is "not failed" can be improved (Verification 2-2).
[0038] Furthermore, in the above-described determination method, if the inputs (A0, A1) to the models (operational model (Y) 124a, modified model (X) 124b) are not single images but have a distribution, then the output will also have a distribution. Therefore, since the relationship (slope) between the input and output cannot be simply compared, statistics (representative values) may be calculated from the distribution to compare the relationship. For example, statistics such as the mean, maximum value, minimum value, median, and variance can be used as representative values.
[0039] Furthermore, as a method for confirming the relationship (a method for determining failure), instead of using the slope of the output image values as described above, one may compare the difference in inference values between the two models before and after retraining. That is, as shown in Figure 2C, the same inference target image (A0, A1 １ A1 ２ The inference result for ) is the difference between Y(A0) and X(A0) of 200, and Y(A1 １ ) and X (A1 １ The difference between ) is 201, and Y(A1 ２ ) and X (A1 ２ If the difference between the two values (202) is equal, the result may be judged as "failure." In this case as well, the more inference results to compare, the higher the accuracy of the judgment can be.
[0040] Furthermore, the change in output in response to input is not limited to a linear expression as shown in (Equation 1), but may also be expressed by a more complex mathematical formula.
[0041] In the above explanation, we described the case where "reduction in brightness contrast" is detected as performance degradation by the degradation detection unit 113. However, Figure 3 shows an example of how to confirm the relationship with the corrected model after retraining when performance degradation is detected from other perspectives (shape, noise, sharpness). For example, if the inference result of the operational model 124a detects degradation in that the shape is incorrect, and the cause of this degradation is that the training data 121a of the operational model 124a is not compatible with large image data, then the processed data by the dataset processing unit 114 will be the image size of the training data 121a enlarged. Therefore, the size of the input image to the corrected model (X) 124b will be larger, and the size of the output image will also be larger, and the difference verification unit 116 should confirm that the measured length value will also be larger as a result.
[0042] In this way, the modified model (X) 124b, whose robustness has been confirmed by the normal verification unit 115 and the differential verification unit 116, is updated by the model update unit 117 as a new operational model (Y) 124a and provided to the inspection device 180 as the modified machine learning model. Also, if necessary, the contents of the retraining dataset storage unit 122 are replaced in the dataset storage unit 121 with operational model data.
[0043] If the degradation detection unit 113 detects multiple degradation factors, the dataset processing unit 114 will create multiple datasets of processed data for each degradation factor, and multiple modified models (X) 124b will be generated for each degradation factor. The model update unit 117 will then select one of the multiple modified models and replace it as the model currently in operation. The model update unit 117 may select one of the multiple modified models by choosing the one with the best inference performance, selecting the most relevant modified model based on a predetermined priority of inference content, or presenting the contents of multiple modified models to the user via a display device or the like for selection.
[0044] Furthermore, the results display unit 118 may display the verification results (Figures 2A to 2C, etc.) to a display device (not shown) via the input / output interface 140 and present them to the user.
[0045] Next, the processing of the machine learning device 100 and its various components will be explained using flowcharts. First, the overall processing flowchart is shown in Figure 4A.
[0046] In step S410, the degradation detection unit 113 performs degradation detection on the inference result data Y(B0)171 from the operational model (Y)124a. In step S420, the dataset processing unit 114 processes the data in the dataset storage unit 122 based on the degradation detection result from the degradation detection unit 113 and stores it in the retraining dataset storage unit 122. In step S430, the learning unit 111 performs retraining based on the data in the retraining dataset storage unit 122 and stores the modified model (X)124b in the model storage unit 124. In step S440, the normal verification unit 115 performs normal verification on the modified model (X)124b. In step S450, the difference verification unit 116 performs difference verification on the modified model (X)124b. In step S480, the model update unit 117 determines whether the modified model (X)124b is updatable, performs the update process, and the entire flow ends (S490).
[0047] Next, the processing of each of the above steps will be explained in more detail. Figure 4B is a flowchart detailing the degradation detection process in step S410 in Figure 4A. The main processing unit in this step is the degradation detection unit 113. In step S411, inference is performed on the inferred image B0 from the inferred image storage unit 123 using the operational model (Y) 124a. In step S412, performance degradation is attempted on the inferred image Y (B0) 171 using one of the four degradation detection units (shape change detection unit 113a, brightness contrast detection unit 113b, noise detection unit 113c, sharpness detection unit 113d) of the degradation detection unit 113.
[0048] If performance degradation is detected in step S413 (Yes), in step S414, the result that performance degradation was detected and the degradation level (numerical value, etc.) are stored in the corresponding storage unit of the degradation detection result storage unit 126, and the process proceeds to step S415. If performance degradation is not detected in step S413 (No), the process proceeds to step S415. In step S415, it is checked whether performance degradation detection has been performed by all four types of degradation detection units of the degradation detection unit 113. If there are any unprocessed degradation detection units (No), the process returns to step S412, and the degradation detection process using other different degradation detection units (steps S412 to S415) is repeated. If all processes have been completed (Yes in step S415), the degradation detection process ends (S416).
[0049] Figure 4C is a flowchart detailing the data processing steps in step S420 of Figure 4A. The main processing unit in this step is the dataset processing unit 114. In step S421, the dataset processing unit 114 reads the performance degradation detection result from the degradation detection result storage unit 126 for one performance degradation (in terms of perspective). In step S422, it checks whether performance degradation has been detected for the read performance degradation. If performance degradation has been detected (Yes), the process proceeds to step S423 to read image data from the dataset storage unit. If performance degradation has not been detected (No in step S422), the process proceeds to step S426.
[0050] In step S424, the read image data is processed based on the performance degradation level included in the performance degradation detection result. In step S425, it is checked whether processing has been completed for all image data, and until it is completed (No in step S425), the process returns to step S423 and repeats processing of the remaining data. If processing of all data is completed in step S425 (Yes), in step S426, it is checked whether processing from all performance degradation perspectives has been completed, and if there are any unprocessed performance degradation perspectives (No), the process returns to step S422 and repeats the subsequent processing. If processing from all performance degradation perspectives is completed in step S426 (Yes), the data processing process ends (S427).
[0051] Figure 4D is a flowchart detailing the model retraining process in step S430 in Figure 4A. The main processing unit in this step is the learning unit 111. In step S431, one dataset is read from the retraining dataset storage unit 122. For example, if degradation detection is performed by the four degradation detection units (113a, 113b, 113c, 113d) of the degradation detection unit 113, four types of datasets processed accordingly are stored in the retraining dataset storage unit 122, and one of these datasets is read.
[0052] In step S432, training is performed using the read dataset to generate a modified model (X) 124b, which is then stored in the model storage unit 124. In step S433, it is checked whether training has been performed on all datasets. If there are any untrained datasets (No in step S433), the process returns to step S432, training is performed on the untrained datasets, and the subsequent processing is repeated. If all datasets have been trained (Yes in step S433), multiple types of modified models (X) 124b are generated corresponding to each of the multiple datasets, and the model retraining process is completed (S434).
[0053] Figure 4E is a flowchart detailing the normal verification process in step S440 in Figure 4A. The main processing unit in this step is the normal verification unit 115. In step S441, the retrained modified model (X) 124b is read from the model storage unit 124. If multiple types of retrained modified models (X) 124b have been generated, one of them is read.
[0054] In step S442, the inference target image B0, which has a different trend from the one used when the operational model (Y) 124a was trained, is read from the inference target image storage unit 123. In step S443, the inference execution unit 112 performs inference using the read inference target image B0 and the modified model (X) 124b, and checks for performance degradation of the inference result X(B0) using one of the four degradation detection units (113a, 113b, 113c, 113d) provided in the degradation detection unit 113.
[0055] In step S444, if performance degradation is detected (Yes), the process proceeds to step S445, where a "fail" mark (flag) is assigned to the modified model (X) 124b stored in the model storage unit 124. If performance degradation is not detected (No), the process proceeds to step S446, where a "pass" mark is assigned to the modified model (X) 124b. Here, assigning a "pass" or "fail" mark means, for example, assigning characters or symbols that mean "pass" or "fail" to the file name of the application program of the modified model (X) 124b stored in the model storage unit 124, or writing "pass" or "fail" in a predetermined area within the program file.
[0056] In step S447, it is checked whether performance degradation detection has been performed for all four degradation detection units (113a, 113b, 113c, 113d) of the degradation detection unit 113. If there are any unprocessed degradation detections (No), the process returns to step S443 and repeats the subsequent processing until performance degradation detection has been performed for all four degradation detection units (113a, 113b, 113c, 113d). If performance degradation detection has been performed for all four degradation detection units (Yes), the process proceeds to step S448.
[0057] In step S448, if multiple types of retrained models (X) 124b have been generated, it is checked whether the performance degradation detection process described above has been performed on all of them. If there are any unprocessed modified models (X) 124b (No), the process returns to step S441 and repeats the subsequent processes until processing is completed for all modified models (X) 124b. Once processing is completed for all modified models (X) 124b (Yes), the normal verification process ends (S449).
[0058] Figure 4F is a flowchart detailing the difference verification process in step S450 in Figure 4A. The main processing unit in this step is the difference verification unit 116, and it is a difference verification process flow that uses both (verification 2-1) and (verification 2-2) as explained above using Figure 2B.
[0059] In step S451, the retrained model (X) 124b is read from the model storage unit 124. If there are multiple retrained models (X) 124b, one of them is read. In step S452, the operational model (Y) 124a, whose performance degradation has been detected, is read from the model storage unit 124. In step S453, the corresponding image (A0) is read from the validation data 121b, which corresponds to the training data of the operational model (Y) 124a, from the dataset storage unit 121. In step S454, the corresponding image (A1) is read from the validation data 122b, which corresponds to the training data of the modified model (X) 124b, from the retraining dataset storage unit 122. １ ) and (A1 ２ In step S455, when reading the corresponding image (A0), the inference execution unit 112 obtains the inference result X(A0) 175 by inferring the corresponding image (A1) using the model (X) 124b. In step S456, the inference execution unit 112 obtains the inference result X(A0) 175 by inferring the corresponding image (A1) １ By inferring ) using model (X) 124b, the inference result X (A1 １ ) 176 is obtained. In step S457, the inference execution unit 112 obtains the corresponding image (A1 ２ By inferring ) using model (X) 124b, the inference result X (A1 ２ ) Obtain 176'.
[0060] In step S458, the inference execution unit 112 obtains the inference result Y(A0) 173 by inferring the corresponding image (A0) using the model (Y) 124a. In step S459, the inference execution unit 112 obtains the inference result Y(A0) 173 by inferring the corresponding image (A1 １ By inferring ) using model (Y) 124a, the inference result Y(A1 １ ) 174 is obtained. In step S460, the inference execution unit 112 obtains the corresponding image (A1 ２ By inferring ) using model (Y) 124a, the inference result Y(A1 ２ ) Obtain 174'.
[0061] In step S461, the inference results from model (X) 124b are (A0, X(A0)) and (A1 １ , X (A1 １)) and are mapped onto the XY coordinate system as shown in Figure 4A, and their inclination "m1" is obtained. Similarly, in step S462, (A0, X(A0)) and (A1 ２ , X (A1 ２ The slope "m2" is obtained from the model (Y) 124a, and in step S463, the inference result (A0, Y(A0)) and (A1 １ , Y(A1 １ The slope of )) is "n", and in step S464, (A0, Y(A0)) and (A1 ２ , Y(A1 ２ Get the slope "k" of ).
[0062] In step S465, X(A0), X(A1) obtained as described above, １ ), X (A1 ２ ), Y(A0), Y(A1 １ ), Y (A1 ２ The result display unit 118 maps each point and the inclinations "m1", "m2", "n", and "k" onto the XY coordinate system as shown in Figure 2B, and displays them on an external display device via the input / output interface 140. This step S465 may be omitted.
[0063] In step S466, it is checked whether the slopes "m1" and "m2" obtained as described above are equal to a predetermined slope "m". If they are equal (Yes), the process proceeds to step S467; if they are not equal (No), the process proceeds to step S469 and the model (X) is marked as "failed".
[0064] In step S467, it is checked whether the slope "m1" (=m2) is equal to either the slope "n" or the slope "k". If it is not equal to either (Yes), the process proceeds to step S468 to mark the modified model (X) as "pass". If it is equal to at least one of them (No), the process proceeds to step S469 to mark the modified model (X) as "fail".
[0065] In step S470, if multiple modified models (X) were read in step S451, it is checked whether the above processing has been completed for all of them. If it has been completed for all of them, the difference verification process ends (S471). If there are any unprocessed modified models (X) (No), the process returns to step S451 and is repeated until the subsequent processing is completed for all modified models (X).
[0066] Figure 4G is a flowchart detailing the model update process in step S480 in Figure 4A. The main processing unit in this step is the model update unit 117. In step S481, the model storage unit 124 checks for the presence of a corrected model (X) that has been marked as "pass" and not "fail" (hereinafter referred to as "certified pass corrected model (X)"), and if there is one, the number of such models. In step S482, the presence of a certified pass corrected model (X) is checked, and if there is one (Yes), the number of certified pass corrected models (X) is checked in step S483. If there is one (Yes), the process proceeds to step S486, where the current model (operating model (Y)) used for inference of images from the inspection device is replaced with the certified pass corrected model (X). The data stored in the retraining dataset storage unit 122, which was used to generate the certified pass corrected model (X), is replaced in the dataset storage unit 121.
[0067] On the other hand, if there are multiple (No.) approved correction models (X) in step S483, one is selected from the multiple models in step S484, and the process proceeds to step S486 to replace the model. Here, as a method for selecting one of the multiple approved correction models (X), as described above, the one with the best inference performance (for example, the one with the smallest degree of degradation detected by the degradation detection unit 113 in the normal verification process) may be selected, or the correction model with the highest relevance (greatest impact) may be selected based on the priority of predetermined inference content (image quality content), or the content of the multiple approved correction models (X) may be presented to the user on a display device or the like for selection.
[0068] Furthermore, if the number of approved corrected models (X) in step S482 is "None" (No), then in step S485, a message (alarm) is displayed to the user on a display device or the like indicating that the robustness of the corrected model (X) could not be confirmed, and therefore the model update was unsuccessful, and the model update process is terminated (S485).
[0069] The updated model is provided to the inspection device 180 and used as an operational model in the inspection device 180.
[0070] As explained above, according to this embodiment, when a performance degradation of a machine learning model is detected and the model is updated with a corrected version, two types of verification are performed: normal verification and differential verification. Therefore, robustness against performance degradation factors can be reliably confirmed.
[0071] This embodiment can be modified in various ways to the extent possible. For example, the degradation detection unit 113 performed degradation detection using four detection units: a shape change detection unit 113a, a brightness contrast detection unit 113b, a noise detection unit 113c, and a sharpness detection unit 113d. However, it is not limited to these, and performance degradation may be detected from other aspects of image quality. Furthermore, instead of using all four of these detection units, at least one specific unit may be used.
[0072] Furthermore, in this embodiment, the robustness of the modified model is confirmed by using both the normal verification unit 115 and the differential verification unit 116, but it is also possible to confirm it using only the differential verification unit 116.
[0073] Furthermore, in this embodiment, if the robustness of the modified model could not be confirmed, the model was not updated. However, the user may be shown the pass / fail score (how far it fell short of the passing level) from the normal verification unit 115 or the differential verification unit 116 for the modified model, and the model may be updated (or considered as a candidate for model update) even if the model is marked as "failed" by the user.
[0074] 100: Machine learning device 110: Main memory 111: Learning unit 112: Inference execution unit 113: Degradation detection unit 113a: Shape change detection unit 113b: Brightness contrast detection unit 113c: Noise detection unit 113d: Sharpness detection unit 114: Dataset processing unit 115: Normal verification unit 116: Difference verification unit 117: Model update unit 118: Result display unit 120: Auxiliary memory 121: Dataset storage unit 122: Dataset storage unit for retraining 123: Image storage unit for inference 124: Model storage unit 125: Verification result storage unit 126: Degradation detection result storage unit 126a: Image correlation value 126b: Brightness difference value 126c: Noise value 126d: Sharpness value 130: Processing unit 140: Input / output interface 150: Communication interface 160: Bus signal line
Claims
1. An image machine learning device comprising: a dataset storage unit storing a first dataset; a first model trained using the first dataset; a degradation detection unit that detects degradation of the inference results obtained by the first model on an image to be inferred from an external device; a dataset processing unit that processes the first dataset to obtain a second dataset based on the degradation detection results from the degradation detection unit; a retraining dataset storage unit that stores the second dataset; and a second model trained using the second dataset; and a difference verification unit that obtains first and second inference results obtained by inferring data in the dataset storage unit and the retraining dataset storage unit using the first model, and third and fourth inference results obtained by inferring data in the dataset storage unit and the retraining dataset storage unit using the second model, and presents evaluation results that evaluate the changes between the first and second inference results and the changes between the third and fourth inference results.
2. An image machine learning apparatus according to claim 1, wherein the difference verification unit determines whether the second model is acceptable or unacceptable based on the evaluation result.
3. An image machine learning apparatus according to claim 2, wherein the difference verification unit determines the second model to be acceptable when the rate of change between the third inference result and the fourth inference result is equal to a predetermined rate and different from the rate of change between the first inference result and the second inference result.
4. An image machine learning apparatus according to claim 2, characterized in that the difference verification unit obtains, as a second inference result, a plurality of different second inference results obtained by inferring a plurality of different data in the retraining dataset storage unit; obtains, as a fourth inference result, a plurality of different fourth inference results obtained by inferring a plurality of different data in the retraining dataset storage unit; and determines the second model to be acceptable when the rate of change between the third inference result and each of the plurality of different fourth inference results is equal to a predetermined rate and is different from the rate of change between the first inference result and each of the plurality of different second inference results.
5. An image machine learning apparatus according to claim 2, wherein the difference verification unit obtains the first inference result and the third inference result by performing inference on the same image data, obtains the second inference result and the fourth inference result by performing inference on other identical image data, and determines that the second model is unacceptable if the difference between the first inference result and the third inference result is equal to the difference between the second inference result and the fourth inference result.
6. An image machine learning apparatus according to claim 4, characterized in that the degradation detection unit detects degradation of the inference result from a plurality of different viewpoints, and the dataset processing unit processes the first dataset from the viewpoints detected by the degradation detection unit.
7. An image machine learning apparatus according to claim 6, wherein the degradation detection unit detects degradation of inference performance using at least one of shape change, brightness contrast, noise, and sharpness as characteristics of the inference image.
8. An image machine learning apparatus according to claim 6, comprising an inference target image storage unit for storing the inference target image from the external device, and a normal verification unit for verifying the appropriateness of the inference result obtained by inferring the inference target image in the inference target image storage unit using the second model, with the degradation detection unit.
9. An image machine learning apparatus according to claim 8, characterized in that it has a model update unit that replaces the current model before the update with the second model which has been determined to pass by the normal verification unit and the difference verification unit as the updated model.
10. An image machine learning apparatus according to claim 9, wherein the model update unit, if there are multiple second models that have been determined to be acceptable, selects one of them and replaces the current model before the update with the updated model.
11. An image processing system comprising an inspection device for inspecting images using a machine learning model, and an image machine learning device according to claim 10, wherein the inspection device is provided with the machine learning model from the machine learning device, and the machine learning device provides the inspection device with the updated model when it detects a performance degradation of the current model being used by the inspection device.
12. A machine learning method for images in a machine learning apparatus having a dataset storage unit, a first model, a degradation detection unit, a dataset processing unit, a retraining dataset storage unit, a second model, and a difference verification unit, wherein the dataset storage unit stores a first dataset, the first model learns using the first dataset, the degradation detection unit detects degradation of the inference results performed by the first model on an inference target image from an external device, the dataset processing unit obtains a second dataset by processing the first dataset based on the degradation detection results from the degradation detection unit, the retraining dataset storage unit stores the second dataset, and the second model learns using the second dataset. The difference verification unit obtains a first inference result and a second inference result obtained by inferring data from the dataset storage unit and the retraining dataset storage unit using the first model, and a third inference result and a fourth inference result obtained by inferring data from the dataset storage unit and the retraining dataset storage unit using the second model, and presents an evaluation result that evaluates the change between the first inference result and the second inference result, and the change between the third inference result and the fourth inference result.