Information processing method, computer program, and information processing device

Abstract

Provided are an information processing method, a computer program, and an information processing device that can be expected to make it possible to accurately perform various types of processing that use a captured image of a substrate that has undergone substrate processing.　According to one embodiment, an information processing method involves an information processing device acquiring a captured image of a substrate that has undergone substrate processing, performing first processing that uses a trained super-resolution model to upscale the acquired captured image and thereby generate a high-resolution image, and performing second processing that uses a trained learning model on the super-resolution image generated by the first processing.

Description

Information Processing Method, Computer Program, and Information Processing Apparatus

[0007] ,

[0006] , ,

[0005] ,

[0001] The present disclosure relates to an information processing method, a computer program, and an information processing apparatus.

[0002] In Patent Document 1, a pattern inspection system has been proposed that inspects an image of an inspection target pattern of an electronic device using an identifier configured by machine learning based on the image of the inspection target pattern of the electronic device and the data used to manufacture the inspection target pattern. This pattern inspection system includes a storage unit that stores a plurality of pattern images of an electronic device and pattern data used to manufacture the pattern of the electronic device, and an image selection unit that selects a learning pattern image for machine learning from the plurality of pattern images based on the pattern data and pattern images stored in the storage unit.

[0003] Japanese Patent Application Laid-Open No. 2020-35282

[0004] The present disclosure provides an information processing method, a computer program, and an information processing apparatus that can be expected to accurately perform various processes using a captured image of a substrate on which substrate processing has been performed.

[0005] An information processing method according to an embodiment includes an information processing apparatus acquiring a captured image of a substrate on which substrate processing has been performed, performing a first process of generating a high-resolution image obtained by upscaling the acquired captured image using a learned super-resolution model, and performing a second process using a learned learning model for the high-resolution image generated in the first process.

[0006] According to the present disclosure, it can be expected to accurately perform various processes using a captured image of a substrate on which substrate processing has been performed.

[0007] This is a schematic diagram illustrating the overview of the information processing system according to this embodiment. This is a schematic diagram illustrating the overview of the processing performed by the information processing device according to this embodiment. This is a block diagram illustrating an example configuration of the information processing device according to this embodiment. This is a flowchart illustrating the procedures for high-resolution processing and subsequent processing performed by the information processing device according to this embodiment. This is a schematic diagram showing an example of high-resolution processing and subsequent processing by the information processing device according to this embodiment. This is a schematic diagram showing an example of segmented imaging. This is a flowchart illustrating the procedures for retraining processing performed by the information processing device according to this embodiment. This is a schematic diagram illustrating the retraining processing of the super-resolution model performed by the information processing system according to this embodiment. This is a schematic diagram illustrating an example of the information processing device performing multiple processes as subsequent processing. This is a schematic diagram illustrating the retraining processing of the super-resolution model performed by the information processing system according to Modification 2.

[0008] Specific examples of information processing systems according to the embodiments of this disclosure will be described below with reference to the drawings. However, this disclosure is not limited to these examples and is intended to include all changes within the meaning and scope of the claims as indicated by the claims.

[0009] <System Overview> Figure 1 is a schematic diagram illustrating the overview of the information processing system according to this embodiment. The information processing system according to this embodiment is configured to include an information processing device 1, a substrate processing device 101, and an electron microscope 102, etc. The substrate processing device 101 is a device such as a process chamber that performs processes such as etching on semiconductor wafers (substrates). Processed substrates that have undergone processes such as etching, sputtering, film deposition, coating, or development in the substrate processing device 101 are photographed by the electron microscope 102 to capture images of their cross-sectional shape or surface shape, etc. The electron microscope 102 is a device that observes an object by, for example, irradiating the object with an electron beam and detecting secondary electrons or transmitted electrons emitted from the object, and outputs captured images such as so-called SEM (Scanning Electron Microscope) images or TEM (Transmission Electron Microscope) images of the object. These captured images output by the electron microscope 102 are provided to the information processing device 1.

[0010] Various structures are formed on the surface of a substrate processed by the substrate processing apparatus 101. The formed structures may vary in size or shape due to processes such as film formation, etching, or exposure. If the size or shape of the formed structures deviates significantly from the size or shape of the target structure, it is determined that the substrate or the structure on the substrate is defective, or that an abnormality has occurred in the substrate processing. In this embodiment, the information processing system determines whether the size or shape of the structures formed on the substrate by the substrate processing is correct. Based on images of the substrate taken by the electron microscope 102, the information processing apparatus 1 performs processing such as segmentation, classification, or abnormality detection regarding the structures formed on the substrate.

[0011] In this embodiment, the substrate processing apparatus 101, the electron microscope 102, and the information processing apparatus 1 are described as separate devices, but this is not limited to them. For example, the substrate processing apparatus 101 and the electron microscope 102 may be a single device, the electron microscope 102 and the information processing apparatus 1 may be a single device, or the substrate processing apparatus 101, the electron microscope 102, and the information processing apparatus 1 may be a single device. Furthermore, each of the devices of the substrate processing apparatus 101, the electron microscope 102, and the information processing apparatus 1 may be configured by combining multiple devices.

[0012] In order to accurately perform processing such as segmentation, classification, anomaly detection, or length measurement of structures formed on a substrate based on images captured by the electron microscope 102, it is preferable that the resolution of the captured images used for processing be as high as possible. However, in order to capture the structure to be processed at high resolution, the electron microscope 102 needs to get close to the structure to take the image, which narrows the range of the captured image. If the structure to be processed is too large to fit in a single captured image, it is necessary to capture the structure in multiple stages and combine the obtained images for subsequent processing, and errors during image combining may cause errors in subsequent processing. Conversely, if the electron microscope 102 is used to capture such a large structure in a single captured image, the resulting image will have low resolution for the structure, which may reduce the accuracy of subsequent processing.

[0013] Therefore, in this embodiment, the information processing system takes an image of the structure to be processed, formed on a substrate, using an electron microscope 102 so that it fits within a single image. The information processing device 1 uses super-resolution technology to increase the resolution of the image (upscale) to the low-resolution image taken by the electron microscope 102, and uses it for subsequent processing such as segmentation, classification, anomaly detection, or length measurement.

[0014] In the information processing system according to this embodiment, the substrate processed by the substrate processing apparatus 101 is photographed using an electron microscope 102. The substrate processing includes, for example, multiple processes such as film deposition, etching, or exposure, and these multiple processes are executed in a predetermined order. The photographing of the substrate by the electron microscope 102 may be performed at any timing before or after these multiple processes included in the substrate processing.

[0015] Figure 2 is a schematic diagram illustrating the overview of the processing performed by the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment includes a pre-trained super-resolution model 5 and a learning model 6 that have been pre-machine-trained. The super-resolution model 5 is a learning model that has been machine-trained to output an input image with increased resolution, and existing deep learning models of neural networks such as ESRGAN (Enhanced Super-Resolution Generative Adversarial Network) or SwinIR (Image Restoration Using Swin Transformer) can be used. The information processing device 1 may acquire and use an existing pre-trained super-resolution model 5, or it may use a super-resolution model 5 that has been fine-tuned to perform high-resolution enhancement suitable for images related to substrate processing.

[0016] Furthermore, the learning model 6 is a learning model that has been trained through machine learning to perform predetermined processing on the input image, and a neural network learning model such as a CNN (Convolutional Neural Network) may be used. In this embodiment, predetermined processing performed using the learning model 6 may include, for example, segmentation, classification, or anomaly detection related to structures formed on the substrate. The learning model 6 is generated in advance by machine learning using training data that associates, for example, captured images of the substrate with correct data resulting from predetermined processing.

[0017] The machine learning processing for generating the super-resolution model 5 and the learning model 6 may be performed on the information processing device 1, or on a device other than the information processing device 1.

[0018] The information processing device 1 according to this embodiment acquires an image captured by an electron microscope 102 so as to include the structure to be processed, inputs this image to a super-resolution model 5, and obtains a high-resolution image output by the super-resolution model 5. Next, the information processing device 1 inputs the obtained high-resolution image to a learning model 6 and obtains the processing result output by the learning model 6. For example, the information processing device 1 inputs the high-resolution image to a learning model 6 that performs segmentation of a structure formed on a substrate, and obtains the segmentation result (a collection of flags classifying what is depicted in each pixel of the input image) output by the learning model 6. Based on this segmentation result, the information processing device 1 can, for example, measure the length and width of the target structure and determine whether the target structure is good or bad depending on whether the measured length is within a predetermined range. The processing performed by the information processing device 1 using the learning model 6 is not limited to segmentation processing; various processing such as classification or anomaly detection can be employed.

[0019] In this embodiment, the information processing device 1 "acquires" the processing results of the super-resolution model 5 and the learning model 6. In this embodiment, "acquisition" includes the information processing device 1 obtaining information from an external device and the information processing device 1 obtaining information from an internal (i.e., its own) unit or module.

[0020] Furthermore, in the information processing system according to this embodiment, for example, each time substrate processing is performed by the substrate processing apparatus 101, imaging by the electron microscope 102 and processing by the information processing apparatus 1 are performed, and the processing results are stored in a database or the like. The information processing apparatus 1 according to this embodiment can improve processing accuracy by periodically performing machine learning (fine-tuning, retraining) of the super-resolution model 5 and / or learning model 6 based on the information stored in this way.

[0021] For example, if a user checks the processing results such as segmentation, classification, or anomaly detection output by the learning model 6 and determines that there are errors, the user provides the corrected processing results to the information processing device 1. The information processing device 1 can store the corrected processing results as ground truth data in a database, associating them with the captured image that was the source of these processing results. The information processing device 1 can perform machine learning on the super-resolution model 5 and the learning model 6 by associating the captured image of the circuit board stored in the database with the ground truth data from the segmentation, classification, or anomaly detection processing based on this captured image as training data (teaching data).

[0022] In this embodiment, an electron microscope 102 such as an SEM or TEM is used as the device for photographing the substrate, but it is not limited to this. The imaging device may be a camera using a device such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), or any other device that performs imaging.

[0023] Furthermore, although the object to be photographed in this embodiment is a substrate, it is not limited to this. The object to be photographed may be various things such as a part of the substrate processing apparatus 101, a part of the substrate processing apparatus 101, or consumables used in the substrate processing apparatus 101. The information processing system according to this embodiment can similarly process images taken of, for example, a part of the substrate processing apparatus 101 with a CCD camera.

[0024] <Device Configuration> Figure 3 is a block diagram showing an example configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment can be realized by installing a predetermined application program on a general-purpose information processing device such as a personal computer or a server computer. However, the information processing device 1 may be a dedicated information processing device for controlling a substrate processing device 101 or an electron microscope 102. The information processing device 1 according to this embodiment is configured to include a processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, and an operation unit 15, etc. In this embodiment, the explanation will be given assuming that processing is performed by one information processing device 1, but the processing of the information processing device 1 may be distributed among multiple devices.

[0025] The processing unit 11 is composed of an arithmetic processing unit such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Network Processing Unit), or quantum processor, and a storage device such as ROM (Read Only Memory) and RAM (Random Access Memory). The processing unit 11 reads and executes a program 12a stored in the storage unit 12, thereby performing various processes such as increasing the resolution of captured images, and subsequent processes such as segmentation, classification, anomaly detection, or length measurement based on the high-resolution images.

[0026] The functions realized by the processing unit 11 can be implemented using any circuit or processing circuitry. For example, the circuit or processing circuitry can be implemented using a general-purpose processor, a special-purpose processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a conventional circuit. The functions realized by the processing unit 11 can be programmed using one or more programs stored in one or more memories, or configured in other ways to execute the disclosed functions. The functions of the processing unit 11 can be implemented using circuit or processing circuitry including combinations of these.

[0027] The storage unit 12 is configured using a large-capacity storage device such as a hard disk or an SSD (Solid State Drive). The storage unit 12 stores various programs executed by the processing unit 11, and various data necessary for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores the program 12a executed by the processing unit 11. The storage unit 12 is also provided with a model information storage unit 12b that stores information about the pre-trained super-resolution model 5 and the learning model 6 that have undergone machine learning, and a learning data storage unit 12c that stores learning data for performing machine learning on these models.

[0028] In this embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or optical disc, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may also be written to the storage unit 12 during the manufacturing stage of the information processing device 1, for example. Alternatively, the program 12a may be distributed by a remote server device or the like and acquired by the information processing device 1 via communication. For example, the program 12a may be read from the recording medium 99 by a writing device and written to the storage unit 12 of the information processing device 1. The program 12a may be provided by distribution via a network, or it may be provided in a form recorded on the recording medium 99.

[0029] The model information storage unit 12b stores information about various pre-generated learning models. The information about the learning models may include, for example, information indicating the configuration of the learning model and information such as predetermined values ​​of internal parameters. In this embodiment, the model information storage unit 12b stores information such as the configuration and parameters of a super-resolution model 5 such as ESRGAN or SwinIR. Note that the super-resolution model 5 is not limited to ESRGAN or SwinIR, but various models may be used. In this embodiment, the model information storage unit 12b also stores information such as the configuration and parameters of a learning model 6 that performs processing such as segmentation, classification, or anomaly detection on a high-resolution image of a substrate.

[0030] The training data storage unit 12c stores training data used when performing machine learning on the super-resolution model 5 and / or the training model 6. In this embodiment, if the user determines that there is an error in the result of processing the captured image using the super-resolution model 5 and the training model 6, the information processing device 1 receives input of the correct processing result from the user and stores the data as training data in the training data storage unit 12c, as data associating the captured image with the correct processing result. The information processing device 1 can read the training data from the training data storage unit 12c and retrain the super-resolution model 5 and / or the training model 6, for example, once a week or once a month, or when the amount of accumulated training data exceeds a threshold.

[0031] The communication unit 13 is connected to the electron microscope 102 via a cable such as a communication line or signal line, and transmits and receives data with the electron microscope 102 via this cable. In this embodiment, the communication unit 13 receives data of the captured image of the substrate transmitted from the electron microscope 102 and provides the received data to the processing unit 11. In this embodiment, the target image is exchanged between the electron microscope 102 and the information processing device 1 via communication, but this is not the only way, and the target image may be exchanged via a recording medium such as a memory card.

[0032] The display unit 14 is configured using a liquid crystal display or the like, and displays various images and characters based on the processing of the processing unit 11. In this embodiment, the display unit 14 displays, for example, images captured from an electron microscope 102, high-resolution images of these captured images, and the results of subsequent processing of these high-resolution images.

[0033] The operation unit 15 receives user input and notifies the processing unit 11 of the received input. For example, the operation unit 15 receives user input via a mechanical button or an input device such as a touch panel provided on the surface of the display unit 14. Alternatively, the operation unit 15 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing device 1.

[0034] The storage unit 12 may be an external storage device connected to the information processing device 1. The information processing device 1 may be a multicomputer comprising multiple computers, or it may be a virtual machine virtually constructed by software. Furthermore, the information processing device 1 is not limited to the above configuration, and for example, it may not have a display unit 14 and an operation unit 15. If the display unit 14 and operation unit 15 are not provided, the information processing device 1 may, for example, use the display unit and operation unit of a terminal device used by the user to display information and receive user operations.

[0035] Furthermore, in the information processing device 1 according to this embodiment, the processing unit 11 reads and executes the program 12a stored in the storage unit 12, thereby realizing the image acquisition unit 11a, the high-resolution processing unit 11b, the subsequent processing unit 11c, the display processing unit 11d, the learning data collection unit 11e, and the learning processing unit 11f, etc., as software-based functional units in the processing unit 11. In this figure, the functional units of the processing unit 11 that relate to the process of increasing the resolution of captured images and subsequent processing of high-resolution images are shown, while functional units related to other processes are omitted from the illustration.

[0036] The image acquisition unit 11a performs the process of acquiring images such as SEM images or TEM images taken by the electron microscope 102 of the substrate processed by the substrate processing apparatus 101. The image acquisition unit 11a communicates with the electron microscope 102 via the communication unit 13 to acquire the image of the substrate taken by the electron microscope 102 as the image to be processed. Note that the image acquisition unit 11a may acquire more than one image as the image to be processed.

[0037] The high-resolution processing unit 11b performs high-resolution processing to increase the resolution of the captured image of the substrate acquired by the image acquisition unit 11a. The high-resolution processing is, for example, the process of generating a 20-pixel x 20-pixel image by doubling the size of a 10-pixel x 10-pixel image both vertically and horizontally. The size of the input and output images and the magnification of the high-resolution processing are predetermined by the designer of the information processing system. In this embodiment, the high-resolution processing unit 11b performs high-resolution processing of the captured image using a super-resolution model 5 that has been pre-machine-trained.

[0038] When a super-resolution model 5 such as ESRGAN is used, the super-resolution model 5 can increase the resolution of the input image and remove noise contained in the image. In this case, the super-resolution model 5 accepts the noise reduction intensity as a setting. The high-resolution processing unit 11b accepts, for example, the setting value of the noise reduction intensity from the user as input, provides the image of the substrate acquired by the image acquisition unit 11a and the set noise reduction intensity to the super-resolution model 5, and acquires a high-resolution image output by the super-resolution model 5 accordingly.

[0039] The subsequent processing unit 11c performs predetermined subsequent processing on the high-resolution image obtained by the high-resolution processing unit 11b. Various subsequent processing methods may be employed, such as segmentation, classification, anomaly detection, or length measurement. Length measurement is a process that measures the length of structures formed on a substrate from the image, and can be performed, for example, based on the results of segmentation processing performed as a subsequent process. In this embodiment, the subsequent processing unit 11c also performs subsequent processing using a pre-trained model 6.

[0040] When segmentation is performed as a subsequent process, the learning model 6 may employ an existing model such as SAM (Segment Anything Model), or it may employ a model such as CNN generated by machine learning by the designer of the information processing system according to this embodiment. The segmentation process is a process of classifying which of the objects depicted in the image each pixel constituting the image belongs to, and outputs a mask image showing the classification result of each pixel for the input image. The subsequent processing unit 11c can perform pixel-level classification of structures formed on a substrate depicted in a high-resolution image by performing segmentation processing using the learning model 6. The information processing device 1 can further perform processes such as measuring the length of the target structure formed on the substrate based on this segmentation result, and determining whether the measured length is within a predetermined range.

[0041] When performing anomaly detection as subsequent processing, a model such as a CNN that has been pre-trained through machine learning to output information indicating the presence or absence of anomalies in response to the input of an image can be adopted as the learning model 6. The designer or the like of the information processing system according to the present embodiment creates learning data in which an image of a target structure is associated with a flag indicating the presence or absence of anomalies in the structure depicted in the image, and generates the learning model 6 by performing machine learning processing using the created learning data.

[0042] When performing classification as subsequent processing, a model such as a CNN that has been pre-trained through machine learning to output information indicating the classification result in response to the input of an image can be adopted as the learning model 6. The designer or the like of the information processing system according to the present embodiment creates learning data in which an image of a target structure is associated with a flag indicating what kind of structure is depicted in the image, and generates the learning model 6 by performing machine learning processing using the created learning data.

[0043] Note that the processing performed by the subsequent processing unit 11c is not limited to the above-mentioned segmentation, anomaly detection, classification, or length measurement, and can be any kind of processing. Also, the subsequent processing unit 11c may perform a plurality of processes in parallel, such as performing segmentation and classification processes in parallel. Further, the subsequent processing unit 11c may perform a plurality of processes in series, such as performing anomaly detection based on the result of segmentation.

[0044] Also, in the present embodiment, both the super-resolution model 5 used by the high-resolution processing unit 11b for high-resolution processing and the learning model 6 used by the subsequent processing unit 11c for subsequent processing are models that adopt the configuration of a neural network. The data output by the super-resolution model 5 can be directly input into the learning model 6, and by backpropagating the error between the output value of the learning model 6 and the correct value of the learning data from the learning model 6 to the super-resolution model 5 by the error backpropagation method, machine learning of the super-resolution model 5 and the learning model 6 can be performed simultaneously.

[0045] The display processing unit 11d performs the processing of displaying various characters and images on the display unit 14. In this embodiment, the display processing unit 11d displays the captured image of the substrate acquired by the image acquisition unit 11a, displays the high-resolution image that has been enhanced by the high-resolution processing unit 11b, and displays the results of subsequent processing by the subsequent processing unit 11c. For example, the display processing unit 11d superimposes a mask image of the segmentation result onto the high-resolution image of the substrate and displays it. Alternatively, the display processing unit 11d may associate the captured image of the substrate with the presence or absence of abnormalities, and may also superimpose an image indicating the location of the abnormality onto the captured image. The display processing unit 11d may display various other images on the display unit 14.

[0046] The training data collection unit 11e performs the process of collecting training data for machine learning (retraining, fine tuning) of the super-resolution model 5 and / or the training model 6. The training data collection unit 11e accepts corrections to the segmentation results from the user, for example, when the user determines that there is an error in the segmentation results displayed by the display processing unit 11d, and associates the original captured image with the corrected segmentation results to create training data. The training data collection unit 11e stores the training data, which associates the captured image with the segmentation results, in the training data storage unit 12c.

[0047] The learning processing unit 11f performs machine learning (re-learning, fine-tuning) of the super-resolution model 5 and / or the learning model 6 using the learning data collected by the learning data collection unit 11e in the learning data storage unit 12c. The learning processing unit 11f reads the learning data from the learning data storage unit 12c at a cycle such as once a week or once a month, or when the accumulated amount of learning data exceeds a threshold, and performs machine learning processing on the super-resolution model 5 and / or the learning model 6 stored in the model information storage unit 12b to update the parameters. Note that the learning processing unit 11f does not automatically perform machine learning processing according to a predetermined cycle or the like, but may perform machine learning processing, for example, when an instruction is given from a user. Also, the machine learning processing of the super-resolution model 5 and / or the learning model 6 may be performed by a device different from the information processing device 1 instead of being performed by the information processing device 1.

[0048] <High-Resolution Processing and Subsequent Processing> FIG. 4 is a flowchart showing the procedures of high-resolution processing and subsequent processing performed by the information processing device 1 according to the present embodiment. FIG. 5 is a schematic diagram showing an example of high-resolution processing and subsequent processing by the information processing device 1 according to the present embodiment. In FIGS. 4 and 5, the case where the information processing device 1 performs segmentation processing as subsequent processing is shown. The image acquisition unit 11a of the processing unit 11 of the information processing device 1 according to the present embodiment acquires the data of the captured image of the substrate photographed by the electron microscope 102 after the substrate processing is performed by the substrate processing device 101 by communicating with the electron microscope 102 via the communication unit 13 (step S1). An example of the captured image of the substrate photographed by the electron microscope 102 is shown at the uppermost stage of FIG. 5.

[0049] The high-resolution processing unit 11b of the processing unit 11 reads the information stored in the model information storage unit 12b to construct the super-resolution model 5 and inputs the captured image acquired in step S1 to the super-resolution model 5 (step S2). At this time, the high-resolution processing unit 11b inputs, for example, a preset noise reduction intensity to the super-resolution model 5 along with the captured image. The high-resolution processing unit 11b acquires the high-resolution image output by the super-resolution model 5 in response to the input of the captured image in step S2 (step S3). An example of a high-resolution image obtained by increasing the resolution of a captured image is shown in the center of Figure 5.

[0050] The subsequent processing unit 11c of the processing unit 11 reads the information stored in the model information storage unit 12b and configures a learning model 6 that performs segmentation processing, and inputs the high-resolution image acquired in step S3 to the learning model 6 (step S4). The subsequent processing unit 11c acquires the segmentation result output by the learning model 6 in response to the input of the high-resolution image in step S4 (step S5). The segmentation result output by the learning model 6 is, for example, data in which the result of classifying which of the objects depicted in the high-resolution image each pixel belongs to is stored in an array of the same size as the original image, a so-called mask image. The bottom row of Figure 5 shows an example of a mask image of the segmentation result for a high-resolution image.

[0051] The processing unit 11 performs a length measurement process based on the segmentation result obtained in step S5, for example, to measure the length of a structure formed on the substrate (step S6). The display processing unit 11d of the processing unit 11 displays the processing results from steps S1 to S6 on the display unit 14 (step S7). Here, the display processing unit 11d can, for example, display the captured image obtained in step S1 and the high-resolution image obtained in step S3 side by side, superimpose the mask image obtained in step S5 onto the high-resolution image, and display the measured value obtained from the length measurement process in step S6 near these images.

[0052] In this embodiment, the user can check the processing result displayed by the information processing device 1 in step S7, and if the user determines that there is an error in the processing result, the user can correct the processing result. The information processing device 1, for example, provides a button on the processing result display screen for transitioning to the processing result correction screen, and transitions to the correction screen by accepting user operation on this button. For example, if segmentation processing is performed as a subsequent process, the information processing device 1 displays a high-resolution image and a mask image on the correction screen, and accepts correction operations on the mask image to correct the processing result.

[0053] After displaying the processing result in step S7, the learning data collection unit 11e of the processing unit 11 determines whether or not to modify the processing result, for example, depending on whether or not the user has operated on the modification button provided on the processing result display screen (step S8). If the processing result is not to be modified (S8: NO), the learning data collection unit 11e terminates the process. If the processing result is to be modified (S8: YES), the learning data collection unit 11e displays a modification screen for modifying the processing result on the display unit 14, for example, and accepts an operation from the user to modify the processing result on the operation unit 15 (step S9). The learning data collection unit 11e associates the processing result reflecting the modification received in step S9 with the captured image acquired in step S1 to create learning data, stores this learning data in the learning data storage unit 12c (step S10), and terminates the process.

[0054] The subsequent processing using the learning model 6 can be performed not only with a high-resolution image obtained by enhancing the resolution of an image taken of the target structure on the substrate in a single shot using the super-resolution model 5, but also with a concatenated image obtained by concatenating multiple images taken in multiple shots at a close distance to the target structure. Figure 6 is a schematic diagram showing an example of performing segmented imaging. The information processing device 1 may accept a selection from the user, for example, whether to perform subsequent processing by enhancing the resolution of the captured image or by concatenating multiple images taken in segments, and perform processing according to the selection. If the selection is made to perform subsequent processing based on a concatenated image, the information processing device 1 acquires multiple images of the substrate, generates a concatenated image by concatenating the acquired multiple images, and inputs the generated concatenated image to the learning model 6. Subsequent processing is the same as when a high-resolution image is input to the learning model 6.

[0055] In Figures 5 and 6, images of the cross-section of the substrate are shown as examples of captured images, but the captured image is not limited to the cross-section of the substrate. The captured image may be, for example, an image of the top surface of the substrate. The image of the top surface of the substrate may be, for example, taken from vertically above the substrate, or from diagonally above the substrate. The object may be photographed from any direction.

[0056] <Retraining Process> In the information processing system according to this embodiment, the super-resolution model 5 is retrained using the training data stored in the training data storage unit 12c. Figure 7 is a flowchart showing the procedure for the retraining process performed by the information processing device 1 according to this embodiment. Figure 8 is a schematic diagram illustrating the retraining process of the super-resolution model 5 performed by the information processing system according to this embodiment. The super-resolution model 5 and the training model 6 used in the information processing system according to this embodiment employ neural network models. The super-resolution model 5 is a model that enhances the resolution of an input image and outputs a high-resolution image, while the training model 6 is a model that performs subsequent processing such as segmentation on the input high-resolution image.

[0057] In this embodiment, the information processing device 1 performs retraining of the super-resolution model 5 with the output of the super-resolution model 5 connected to the input of the learning model 6. The learning processing unit 11f of the processing unit 11 of the information processing device 1 reads the learning data stored in the learning data storage unit 12c (step S21). The learning processing unit 11f inputs the captured images included in the learning data read in step S21 to the super-resolution model 5 (step S22). The learning processing unit 11f obtains the processing result of the segmentation process output by the learning model 6 in response to the input of the captured images in step S22 (step S23).

[0058] The learning processing unit 11f calculates the error between the processing result obtained in step S23 and the correct value contained in the training data read in step S21 (step S24). The learning processing unit 11f propagates the error calculated in step S24 backward to the neurons constituting the neural network from the output of the learning model 6 to the input layer of the super-resolution model 5 (step S25). Based on the error propagated backward in step S25, the learning processing unit 11f updates the parameters of each neuron constituting the neural network of the super-resolution model 5 (step S26), and terminates the retraining process.

[0059] In this embodiment, the information processing device 1 updates the parameters of the super-resolution model 5 in step S26, but does not update the parameters of the learning model 6. However, the information processing device 1 may update the parameters of both the super-resolution model 5 and the learning model 6, or it may update only the parameters of the learning model 6. By performing machine learning processing by combining the super-resolution model 5 and the learning model 6, the information processing device 1 can be expected to train the super-resolution model 5 to output high-resolution images that improve the accuracy of subsequent processing by the learning model 6.

[0060] <Other Subsequent Processing> Figures 4 to 8 above show an example of segmentation processing as a subsequent process using the learning model 6. Subsequent processing using the learning model 6 is not limited to segmentation processing; various processes may be employed. For example, the information processing device 1 may perform classification processing as a subsequent process using the learning model 6. The learning model 6 is pre-generated by machine learning to accept an image of the substrate (high-resolution image) as input and output a classification result that classifies what is depicted in this image. The information processing device 1 inputs the image of the substrate taken by the electron microscope 102 into the super-resolution model 5 to obtain a high-resolution image, and inputs this high-resolution image into the learning model 6 to obtain a classification result. If the information processing device 1 receives a request from the user to correct the classification result, it associates the captured image and the corrected classification result and stores them as training data in the training data storage unit 12c. The information processing device 1 can update the parameters of the super-resolution model 5 by performing machine learning processing with the super-resolution model 5 and the learning model 6 connected, using the training data stored in the training data storage unit 12c.

[0061] For example, the information processing device 1 may perform anomaly detection processing as a subsequent process using the learning model 6. The learning model 6 is pre-generated by machine learning to receive an image (high-resolution image) of the substrate as input and output information indicating whether or not there is an anomaly in the substrate as captured in the image as a detection result. Alternatively, the learning model 6 may output information such as a bounding box surrounding the anomaly on the substrate as a detection result. The information processing device 1 inputs the image of the substrate captured by the electron microscope 102 into the super-resolution model 5 to obtain a high-resolution image, and inputs this high-resolution image into the learning model 6 to obtain anomaly detection results. If the information processing device 1 receives a request from the user to correct the anomaly detection result, it associates the captured image and the corrected anomaly detection result and stores them as training data in the training data storage unit 12c. The information processing device 1 can update the parameters of the super-resolution model 5 by performing machine learning processing with the super-resolution model 5 and the learning model 6 connected, using the training data stored in the training data storage unit 12c.

[0062] Furthermore, for example, the information processing device 1 may perform multiple processes as subsequent processes using the learning model 6. Figure 9 is a schematic diagram illustrating an example in which the information processing device 1 performs multiple processes as subsequent processes. In this example, the information processing device 1 performs two types of subsequent processes using two learning models 6. As a first subsequent process, the information processing device 1 performs segmentation processing on the captured image of the substrate using a segmentation model 6a that has been pre-trained to perform segmentation processing. As a second subsequent process, the information processing device 1 performs classification processing on the captured image of the substrate using a classification model 6b that has been pre-trained to perform classification processing.

[0063] In this example, the information processing device 1 inputs an image of the substrate captured by the electron microscope 102 into the super-resolution model 5 to obtain a high-resolution image, and inputs this high-resolution image into the segmentation model 6a and the classification model 6b, respectively. The information processing device 1 obtains the segmentation result output by the segmentation model 6a and the classification result output by the classification model 6b. Based on the acquired segmentation result and classification result, the information processing device 1 can perform various further processing.

[0064] Furthermore, when the information processing device 1 receives a request from the user to modify the segmentation result or classification result, it stores the captured image and the modified segmentation result and classification result in association with each other in the learning data storage unit 12c. If only one of the segmentation result or classification result is modified, the information processing device 1 stores the modified result and the unmodified result in association with each other in the captured image.

[0065] The information processing device 1 performs machine learning processing on the super-resolution model 5 with the segmentation model 6a and classification model 6b connected to it. The information processing device 1 inputs captured images included in the training data stored in the training data storage unit 12c to the super-resolution model 5 and obtains the segmentation results output by the segmentation model 6a and the classification results output by the classification model 6b. The information processing device 1 calculates the error between the obtained segmentation results and classification results and the correct values ​​for segmentation and classification included in the training data. The information processing device 1 propagates the error in the segmentation results back from the segmentation model 6a to the super-resolution model 5, and propagates the error in the classification results back from the classification model 6b to the super-resolution model 5, and updates the parameters of the super-resolution model 5 based on these errors, thereby enabling machine learning processing of the super-resolution model 5.

[0066] In this example, we have described a case where segmentation and classification processing are performed as multiple subsequent processes, but the information processing device 1 is not limited to these. Furthermore, the information processing device 1 may perform three or more subsequent processes.

[0067] <Modification 1> The information processing device 1 according to Modification 1 stores multiple types of models as super-resolution models 5 in the model information storage unit 12b. The multiple super-resolution models 5 may be multiple types of models with different configurations, such as ESRGAN and SwinIR, or they may be multiple models with the same configuration but different parameters, such as models before and after retraining. Furthermore, these multiple super-resolution models 5 according to Modification 1 have a function to remove noise contained in the image, such as the ESRGAN model, and it is possible to set the noise reduction intensity as a decimal value from 0.0 to 1.0.

[0068] The information processing device 1 according to Modification 1 accepts from the user the setting of which super-resolution model 5 to use and the setting of the noise reduction intensity before starting processing using the super-resolution model 5 and the learning model 6. The information processing device 1 displays information such as the names of the available super-resolution models 5 in a list on the display unit 14 and accepts from the user the selection of the model to be used from among them. The information processing device 1 also accepts numerical input from the user or numerical selection using a slider, and uses this numerical value as the setting of the noise reduction intensity. The information processing device 1 reads the information of the selected super-resolution model 5 from the model information storage unit 12b to construct the model, and by providing the set noise reduction intensity to the super-resolution model 5 along with the captured image of the substrate, it obtains a high-resolution image with noise reduction at the set intensity. The information processing device 1 can perform subsequent processing by inputting the acquired high-resolution image to the learning model 6.

[0069] The information processing device 1 according to Modification 1 accepts from the user the selection of the super-resolution model 5 to be used for processing and the setting of the noise reduction intensity. This allows the user to appropriately change the combination of the type of super-resolution model 5 and the noise reduction intensity to consider a combination suitable for subsequent processing. The information processing device 1 may be configured to accept only one of the following, rather than accepting both the selection of the super-resolution model 5 and the setting of the noise reduction intensity.

[0070] Furthermore, in the modified example 1, the information processing device 1 may automatically determine the settings for the super-resolution model 5 used in processing and its noise reduction intensity, rather than accepting them from the user. The information processing device 1 calculates the accuracy of the processing result for multiple combinations of the super-resolution model 5 and noise reduction intensity used in processing, using the training data stored in the training data storage unit 12c. For example, if three types of super-resolution models 5 are available and the noise reduction intensity is changed in increments of 0.2, such as (0.0, 0.2, ..., 1.0), then 3 × 6 = 18 combinations of the super-resolution model 5 and noise reduction intensity can be created. For these 18 combinations, the information processing device 1 inputs the captured images included in the training data into the super-resolution model 5 and calculates the prediction accuracy of the model for each combination by comparing the results output by the training model 6 with the correct values. Appropriate evaluation metrics such as accuracy or precision may be used for prediction accuracy. The information processing device 1 presents the user with the combination that offers the highest accuracy and then uses this combination to perform processing.

[0071] <Modification 2> Figure 10 is a schematic diagram illustrating the retraining process of the super-resolution model 5 performed by the information processing system according to Modification 2. The information processing system according to Modification 2 performs machine learning of the super-resolution model 5 using training data that associates a captured image of the substrate, a ground truth image (ground truth data) of a high-resolution image obtained by upscaling this captured image, and ground truth values ​​(ground truth data) for processing results such as segmentation obtained by the super-resolution model 5 and the learning model 6 based on this captured image.

[0072] The information processing device 1 in the modified version 2 inputs captured images included in the training data to the super-resolution model 5 and calculates the error (first error) between the high-resolution image output by the super-resolution model 5 and the correct high-resolution image included in the training data. The information processing device 1 also inputs the high-resolution image output by the super-resolution model 5 to the training model 6 and calculates the error (second error) between the processing result, such as segmentation, output by the training model 6 and the correct value of this processing included in the training data.

[0073] The information processing device 1 performs machine learning on the super-resolution model 5 by backpropagating the first error to the super-resolution model 5 and backpropagating the second error from the learning model 6 to the super-resolution model 5, and updating the internal parameters of the super-resolution model 5 based on both of these errors.

[0074] Furthermore, the parameter updates based on the first error and the parameter updates based on the second error do not have to be performed simultaneously. For example, the information processing device 1 may perform the parameter updates based on the first error first, and then the parameter updates based on the second error. Alternatively, the information processing device 1 may perform the parameter updates based on the second error first, and then the parameter updates based on the first error. Alternatively, the information processing device 1 may perform the parameter updates based on the first error and the parameter updates based on the second error alternately.

[0075] Furthermore, if parameter updates based on the first error and parameter updates based on the second error are performed at different times, the training data does not necessarily need to be a representation of the captured image, the ground truth image of the high-resolution image, and the correct value for the processing result. The information processing device 1 may, for example, store in the training data storage unit 12c a first training data representation of the captured image and the ground truth image of the high-resolution image, and a second training data representation of the captured image and the correct value for the processing result separately. The information processing device 1 can individually perform parameter updates based on the first error using the first training data and parameter updates based on the second error using the second training data.

[0076] Furthermore, if the training data includes both captured images and correct images, the information processing device 1 may, for example, acquire a high-resolution image of the circuit board, generate a low-resolution image by downscaling this high-resolution image, use the generated low-resolution image as the captured image for the training data, and use the original high-resolution image as the correct image for the training data.

[0077] <Summary> In the information processing system according to this embodiment with the above configuration, the information processing device 1 acquires an image taken by the electron microscope 102 of a substrate that has been processed by the substrate processing device 101, performs a first process in which it generates a high-resolution image by upscaling (increasing the resolution of) the acquired image using a trained super-resolution model 5, and performs a second process on the generated high-resolution image using a trained learning model 6. As a result, the information processing system according to this embodiment is expected to be able to perform various processes such as segmentation processing, classification processing, or anomaly detection processing based on the captured image with high accuracy, even if the image taken by the electron microscope 102 does not have sufficient resolution.

[0078] Furthermore, in the information processing system according to this embodiment, the information processing device 1 performs machine learning processing on the super-resolution model 5 using training data that associates the captured image of the substrate obtained from the electron microscope 102 with the correct data relating to the results of the second processing on this captured image. As a result, the information processing system according to this embodiment is expected to perform machine learning processing on the super-resolution model 5 to perform high-resolution processing suitable for the second processing.

[0079] Furthermore, in the information processing system according to this embodiment, the information processing device 1 performs machine learning processing on the learning model 6 using training data that associates the captured image of the substrate obtained from the electron microscope 102 with the correct data relating to the results of the second processing on this captured image. As a result, the information processing system according to this embodiment is expected to perform machine learning processing on the learning model 6 so that the super-resolution model 5 performs the second processing suitable for the high-resolution image that has been enhanced to high resolution.

[0080] Furthermore, in the information processing system according to this embodiment, the information processing device 1 displays (outputs) the result of the second processing using the learning model 6 and accepts requests from the user to modify this processing result. The information processing device 1 associates the modified processing result with the captured image and stores it as training data in the training data storage unit 12c, and performs machine learning processing using the stored training data. As a result, the information processing system according to this embodiment is expected to improve the accuracy of processing by retraining the super-resolution model 5 and / or the learning model 6.

[0081] Furthermore, in the information processing system according to this embodiment, the super-resolution model 5 and the learning model 6 are neural network models. The information processing device 1 inputs captured images of training data to the super-resolution model 5, inputs the high-resolution images output by the super-resolution model 5 to the learning model 6, and calculates the error between the output data of the learning model 6 and the correct data of the training data. The information processing device 1 performs machine learning processing by backpropagating the calculated error from the learning model 6 to the super-resolution model 5. As a result, the information processing system according to this embodiment is expected to perform machine learning processing on the super-resolution model 5 to increase the resolution of captured images to a level suitable for subsequent processing performed by the learning model 6.

[0082] Furthermore, in the information processing system according to this embodiment, the information processing device 1 accepts the selection of a super-resolution model 5 and the setting of the noise reduction intensity by the super-resolution model 5, and uses the selected super-resolution model 5 to perform high-resolution enhancement at the set noise reduction intensity. As a result, the information processing system according to this embodiment is expected to allow the user to verify the combination of the type of super-resolution model 5 and the noise reduction intensity, and to have the information processing device 1 perform processing based on the captured image of the substrate using a suitable combination.

[0083] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of this disclosure is indicated by the claims and not in the sense described above, and all modifications within the meaning and scope equivalent to the claims are intended.

[0084] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used.

[0085] 1 Information Processing Device (Computer) 5 Super-Resolution Model 6 Learning Model 6a Segmentation Model 6b Classification Model 11 Processing Unit 11a Image Acquisition Unit 11b High-Resolution Processing Unit 11c Successor Processing Unit 11d Display Processing Unit 11e Learning Data Acquisition Unit 11f Learning Processing Unit 12 Storage Unit 12a Program (Computer Program) 12b Model Information Storage Unit 12c Learning Data Storage Unit 13 Communication Unit 14 Display Unit 15 Operation Unit 99 Recording Medium 101 Substrate Processing Device 102 Electron Microscope

Claims

1. An information processing method comprising: an information processing device acquiring a photographic image of a substrate that has undergone substrate processing; performing a first process to generate a high-resolution image by upscaling the acquired photographic image using a trained super-resolution model; and performing a second process on the high-resolution image generated in the first process using the trained trained model.

2. The information processing method according to claim 1, wherein machine learning of the super-resolution model is performed using training data obtained by associating a captured image of a substrate with ground truth data relating to the results of the second processing on the captured image.

3. The information processing method according to claim 1, wherein machine learning of the learning model is performed using training data obtained by associating a captured image of a substrate with correct data relating to the results of the second processing on the captured image.

4. The information processing method according to claim 2 or 3, comprising outputting the result of the second processing, accepting a request for correction of the result, associating the corrected result with the captured image and storing it as training data, and performing machine learning using the stored training data.

5. The information processing method according to claim 2 or 3, wherein the super-resolution model and the learning model are neural network models, and the information processing device inputs the captured images of the learning data to the super-resolution model to calculate the error between the output data of the learning model obtained from the super-resolution model and the correct data of the learning data, and performs machine learning by backpropagating the calculated error from the learning model to the super-resolution model.

6. The information processing method according to claim 1, wherein machine learning of the super-resolution model is performed using training data that associates a captured image of a substrate with ground truth data of a high-resolution image obtained by upscaling the captured image and ground truth data relating to the results of the second processing on the captured image.

7. The information processing method according to claim 1, comprising: receiving the selection of a super-resolution model and the setting of the noise reduction intensity by the super-resolution model, and performing high-resolution processing with the set noise reduction intensity using the selected super-resolution model.

8. The information processing method according to claim 1, wherein, as a second process, segmentation processing is performed on an object formed on the substrate based on the high-resolution image generated in the first process.

9. The information processing method according to claim 8, wherein machine learning of the super-resolution model is performed using training data obtained by associating an image of a substrate with the correct data obtained from the segmentation process for the image.

10. The information processing method according to claim 1, wherein, as a second process, an anomaly detection process is performed on an object formed on the substrate based on the high-resolution image generated in the first process.

11. The information processing method according to claim 10, wherein machine learning of the super-resolution model is performed using training data obtained by associating an image of a substrate with the correct data of the anomaly detection process for the image.

12. The information processing method according to claim 1, wherein, as the second process, classification processing is performed on the object formed on the substrate based on the generated high-resolution image.

13. The information processing method according to claim 12, wherein machine learning of the super-resolution model is performed using training data obtained by associating an image of a substrate with the correct data of the classification process for the image.

14. The information processing method according to claim 1, wherein, as the second process, based on the high-resolution image generated in the first process, a segmentation process for the object formed on the substrate using a trained segmentation model and a classification process for the object formed on the substrate using a trained classification model are performed.

15. The information processing method according to claim 14, wherein machine learning of the super-resolution model is performed using training data obtained by associating a captured image of a substrate with the correct data obtained from the segmentation process for the captured image and the correct data obtained from the classification process for the captured image.

16. The information processing method according to claim 1, wherein the learning model can be input either the high-resolution image based on an image of the object on the substrate taken in a single shot, or a concatenated image obtained by concatenating multiple images of the object taken in multiple shots at a distance closer to the substrate than the single shot, and the super-resolution model is input to the image obtained in the single shot.

17. A computer program that causes a computer to perform a first process of acquiring a photographic image of a circuit board that has undergone circuit board processing, and generating a high-resolution image by upscaling the acquired photographic image using a trained super-resolution model, and a second process of performing a trained learning model on the high-resolution image generated in the first process.

18. An information processing device comprising a processing unit, wherein the processing unit acquires an image of a substrate that has undergone substrate processing, performs a first process of generating a high-resolution image by upscaling the acquired image using a trained super-resolution model, and performs a second process on the high-resolution image generated in the first process using the trained model.

Images