Computer program, information processing method, and information processing device

Abstract

Provided are a computer program, an information processing method, and an information processing device that can be expected to detect various patterns from a target image without preliminarily performing machine learning of a detection target pattern.　A computer program according to an embodiment of the present invention causes a computer to execute a process for detecting, from a target image obtained by imaging a substrate subjected to substrate processing, a prescribed pattern formed on the substrate. The computer: acquires condition information related to a detection target pattern; extracts one or more pattern candidates from the target image on the basis of the acquired condition information; deletes a pattern candidate from the extracted one or more pattern candidates on the basis of comparison between the condition information and each of the pattern candidates; and outputs one or more pattern candidates that have not been deleted as a detection result of the pattern.

Description

Computer Program, Information Processing Method, and Information Processing Apparatus

[0001] The present disclosure relates to a computer program, an information processing method, and an information processing apparatus.

[0002] In Patent Document 1, a pattern inspection system has been proposed that inspects an image of an inspection target pattern of an electronic device using a discriminator configured by machine learning based on the image of the inspection target pattern of the electronic device and the data used to manufacture the inspection target pattern. This pattern inspection system includes a storage unit that stores a plurality of pattern images of an electronic device and pattern data used to manufacture the pattern of the electronic device, and an image selection unit that selects a learning pattern image for machine learning from the plurality of pattern images based on the pattern data and the pattern images stored in the storage unit.

[0003] Japanese Patent Application Laid-Open No. 2020-35282

[0004] The present disclosure provides a computer program, an information processing method, and an information processing apparatus that can be expected to detect various patterns from a target image without previously machine-learning the pattern to be detected.

[0005] A computer program according to an embodiment causes a computer to execute a process of detecting a predetermined pattern formed on a substrate from a target image obtained by photographing the substrate on which substrate processing has been performed, the computer obtaining condition information regarding the pattern to be detected, extracting one or more pattern candidates from the target image based on the obtained condition information, deleting pattern candidates from the extracted one or more pattern candidates based on a comparison between the condition information and each pattern candidate, and outputting the one or more pattern candidates that have not been deleted as a detection result of the pattern.

[0006] According to the present disclosure, it can be expected to detect various patterns from a target image without previously machine-learning the pattern to be detected.

[0007] This is a schematic diagram illustrating the overview of the information processing system according to this embodiment. This is a schematic diagram illustrating the overview of the pattern detection process performed by the information processing device according to this embodiment. This is a block diagram illustrating an example configuration of the information processing device according to this embodiment. This is a flowchart illustrating an example of the procedure for the pattern detection process performed by the information processing device according to this embodiment. This is a schematic diagram showing an example of an input screen. This is a schematic diagram illustrating the first algorithm for the pattern candidate extraction process. This is a schematic diagram illustrating the second algorithm for the pattern candidate extraction process. This is a schematic diagram illustrating the first algorithm for the deletion process. This is a schematic diagram illustrating the second algorithm for the deletion process. This is a schematic diagram showing an example of a display of the pattern detection result. This is a schematic diagram showing an example of a screen for modifying the conditions for pattern detection. This is a schematic diagram illustrating the overview of the pattern detection process performed by the information processing system according to Embodiment 2.

[0008] Specific examples of information processing systems according to embodiments of this disclosure will be described below with reference to the drawings. However, this disclosure is not limited to these examples and is intended to include all changes within the meaning and scope of the claims as indicated by the claims.

[0009] <System Overview> Figure 1 is a schematic diagram illustrating the overview of the information processing system according to this embodiment. The information processing system according to this embodiment is configured to include an information processing device 1, a substrate processing device 101, and an electron microscope 102, etc. The substrate processing device 101 is a device such as a process chamber that performs processes such as etching, sputtering, film deposition, coating, or development on semiconductor wafers (substrates). After processing such as etching in the substrate processing device 101, the processed substrate is photographed by the electron microscope 102 to capture its cross-sectional shape or surface shape, etc. The electron microscope 102 is a device that observes an object by, for example, irradiating the object with an electron beam and detecting secondary electrons or transmitted electrons emitted from the object, and outputs so-called SEM (Scanning Electron Microscope) images or TEM (Transmission Electron Microscope) images of the object. These images output by the electron microscope 102 are provided to the information processing device 1 as target images.

[0010] Various structures are formed on the surface of a substrate processed by the substrate processing apparatus 101, and structures of different shapes may be formed for each product. Also, multiple structures of the same shape may be formed on the surface of the substrate, as shown in the target image in the center of Figure 1. In this embodiment, each of these structures of the same shape will be referred to as a pattern. However, in reality, even if the intention is to form patterns of the same shape, the size or shape of each pattern may differ due to processes such as film deposition, etching, or exposure. Patterns whose size or shape deviates beyond a predetermined threshold are judged to be abnormal. In order to determine such abnormal patterns, the information processing system according to this embodiment performs a process to detect multiple patterns formed on the target substrate based on an image of the target substrate taken by the electron microscope 102.

[0011] In this embodiment, the substrate processing apparatus 101, the electron microscope 102, and the information processing apparatus 1 are described as separate devices, but this is not limited to them. For example, the substrate processing apparatus 101 and the electron microscope 102 may be a single device, the electron microscope 102 and the information processing apparatus 1 may be a single device, or the substrate processing apparatus 101, the electron microscope 102, and the information processing apparatus 1 may be a single device. Furthermore, each of the devices of the substrate processing apparatus 101, the electron microscope 102, and the information processing apparatus 1 may be configured by combining multiple devices.

[0012] The information processing device 1 acquires SEM images, etc., of the target substrate captured by the electron microscope 102 as the target image. In this embodiment, the information processing device 1 also acquires a template image showing the pattern to be detected, and / or text information related to the pattern to be detected, as condition information for pattern detection. The information processing device 1 performs a process to detect patterns that match the condition information from the acquired target image. The lower part of Figure 1 shows the state in which the pattern detected from the target image is surrounded by a bounding box. For example, the information processing device 1 outputs information such as the position and size of the bounding box relative to the target image for each detected pattern as the pattern detection result.

[0013] Figure 2 is a schematic diagram illustrating the outline of the pattern detection process performed by the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment is provided with a target image captured by an electron microscope 102 and conditional information such as a template image or text information set by the user as input information for the pattern detection process. In the example shown in Figure 2, a template image is provided to the information processing device 1 as conditional information. Based on the provided target image and conditional information, the information processing device 1 performs a pattern candidate extraction process to extract image regions from the target image that satisfy the conditional information (similar to the template image in this figure) as pattern candidates. Details of the pattern candidate extraction process will be described later, but the information processing device 1 according to this embodiment extracts pattern candidates from the target image, for example, by using a template matching algorithm or by using an existing trained deep learning model that performs object detection.

[0014] The pattern candidate extraction process extracts one or more pattern candidates from the target image, but some of these may be unsuitable (inappropriate, unsuitable, or incompatible) as patterns. The five pattern candidates shown in the center of Figure 2 include two unsuitable ones. Therefore, the information processing device 1 according to this embodiment performs a deletion process to remove unsuitable candidates from the pattern candidates extracted by the pattern candidate extraction process. The details of the deletion process will be described later, but the information processing device 1 according to this embodiment determines whether each pattern candidate is appropriate or unsuitable by comparing the feature quantities of each pattern candidate with the feature quantities of the condition information, and deletes those determined to be unsuitable from the candidates. The information processing device 1 outputs the one or more pattern candidates that remain after deleting the unsuitable candidates as the final pattern detection result.

[0015] The information output by the information processing device 1 as a pattern detection result is information indicating the position where the pattern is captured in the target image, and can be, for example, the position information of a rectangular frame (so-called bounding box) surrounding the pattern. Based on the pattern detection result of the information processing device 1, the information processing system can perform processing such as measuring the length or area of ​​each pattern and determining the quality of each pattern based on the measurement results.

[0016] <Device Configuration> Figure 3 is a block diagram showing an example configuration of the information processing device 1 according to this embodiment. The information processing device 1 according to this embodiment can be realized by installing a predetermined application program on a general-purpose information processing device such as a personal computer or a server computer. However, the information processing device 1 may be a dedicated information processing device for controlling a substrate processing device 101 or an electron microscope 102. The information processing device 1 according to this embodiment is configured to include a processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, and an operation unit 15, etc. In this embodiment, the explanation will be given assuming that processing is performed by one information processing device 1, but the processing of the information processing device 1 may be distributed among multiple devices.

[0017] The processing unit 11 is composed of an arithmetic processing unit such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Network Processing Unit), or quantum processor, and a storage device such as ROM (Read Only Memory) and RAM (Random Access Memory). The processing unit 11 reads and executes a program 12a stored in the storage unit 12, thereby performing various processes such as acquiring a target image and condition information, extracting pattern candidates from the target image, and deleting unsuitable pattern candidates.

[0018] The functions realized by the processing unit 11 can be implemented using any circuit or processing circuitry. For example, the circuit or processing circuitry can be implemented using a general-purpose processor, a special-purpose processor, an integrated circuit, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a conventional circuit. The functions realized by the processing unit 11 can be programmed using one or more programs stored in one or more memories, or configured in other ways to execute the disclosed functions. The functions of the processing unit 11 can be implemented using circuit or processing circuitry including combinations of these.

[0019] The storage unit 12 is configured using a large-capacity storage device such as a hard disk or an SSD (Solid State Drive). The storage unit 12 stores various programs executed by the processing unit 11, and various data necessary for the processing of the processing unit 11. In this embodiment, the storage unit 12 stores the program 12a executed by the processing unit 11. The storage unit 12 is also provided with a condition information storage unit 12b that stores and accumulates condition information related to the target pattern to be detected from the target image.

[0020] In this embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or optical disc, and the information processing device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may also be written to the storage unit 12 during the manufacturing stage of the information processing device 1, for example. Alternatively, the program 12a may be distributed by a remote server device or the like and acquired by the information processing device 1 via communication. For example, the program 12a may be read from the recording medium 99 by a writing device and written to the storage unit 12 of the information processing device 1. The program 12a may be provided by distribution via a network, or it may be provided in a form recorded on the recording medium 99.

[0021] The condition information storage unit 12b stores and accumulates condition information such as template images and text information used for pattern detection. The information processing device 1 can, for example, acquire a template image generated by a user by cropping a part of a target image as condition information and store it in the condition information storage unit 12b. Alternatively, the information processing device 1 can acquire text information in which a user expresses the characteristics of a pattern in writing as condition information and store it in the condition information storage unit 12b. The condition information storage unit 12b stores condition information such as template images and text information in association with text information such as the title of the condition information set by the user or a description of the pattern to which this condition information applies, along with information such as the date and time the condition information was acquired. The information processing device 1 may also store information regarding the settings of the board processing (so-called recipe settings) performed on the board for which pattern detection was performed using the condition information, along with the template images and text information, in the condition information storage unit 12b. The information processing device 1 can, for example, display a list of multiple template images stored as condition information in the condition information storage unit 12b and acquire a template image that will be the pattern to be detected by accepting the selection of one template image from among the multiple template images.

[0022] The communication unit 13 is connected to the electron microscope 102 via a cable such as a communication line or signal line, and transmits and receives data with the electron microscope 102 via this cable. In this embodiment, the communication unit 13 receives data of the captured image of the substrate transmitted from the electron microscope 102 and provides the received data to the processing unit 11. In this embodiment, the target image is exchanged between the electron microscope 102 and the information processing device 1 via communication, but this is not the only way, and the target image may be exchanged via a recording medium such as a memory card.

[0023] The display unit 14 is configured using a liquid crystal display or the like, and displays various images and characters based on the processing of the processing unit 11. In this embodiment, the display unit 14 displays, for example, a target image acquired from an electron microscope 102, and displays the results of pattern detection from the target image.

[0024] The operation unit 15 receives user input and notifies the processing unit 11 of the received input. For example, the operation unit 15 receives user input via a mechanical button or an input device such as a touch panel provided on the surface of the display unit 14. Alternatively, the operation unit 15 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing device 1.

[0025] The storage unit 12 may be an external storage device connected to the information processing device 1. The information processing device 1 may be a multicomputer comprising multiple computers, or it may be a virtual machine virtually constructed by software. Furthermore, the information processing device 1 is not limited to the above configuration, and for example, it may not have a display unit 14 and an operation unit 15. If the display unit 14 and operation unit 15 are not provided, the information processing device 1 may, for example, use the display unit and operation unit of a terminal device used by the user to display information and receive user operations.

[0026] Furthermore, in the information processing device 1 according to this embodiment, the processing unit 11 reads and executes the program 12a stored in the storage unit 12, thereby realizing the image acquisition unit 11a, the condition information acquisition unit 11b, the pattern candidate extraction unit 11c, the deletion processing unit 11d, and the display processing unit 11e, etc., as software-based functional units in the processing unit 11. In this figure, the functional units of the processing unit 11 that are related to the process of detecting patterns from a target image are shown, and functional units related to other processes are omitted from the illustration.

[0027] The image acquisition unit 11a performs the process of acquiring target images, such as SEM images or TEM images, of the substrate processed by the substrate processing apparatus 101, which are then captured by the electron microscope 102. The image acquisition unit 11a communicates with the electron microscope 102 via the communication unit 13 to acquire the images of the substrate captured by the electron microscope 102 as target images for pattern detection. Note that the image acquisition unit 11a may acquire more than one image as a target image.

[0028] The condition information acquisition unit 11b performs processing to acquire condition information from the user, such as a template image or text information that defines the pattern to be detected from the target image. The condition information acquisition unit 11b displays the target image acquired from, for example, the electron microscope 102 on the display unit 14, and accepts the user to specify a part of the image region from this target image, thereby acquiring this part of the image region as a template image.

[0029] For example, the condition information acquisition unit 11b displays a text box on the display unit 14 for inputting condition information along with the target image, and acquires condition information by accepting input of text information describing the characteristics of the pattern to be detected. Various forms of expression can be used for inputting text information, such as sentences describing the shape of the pattern to be detected, or bullet points of characteristic dimensions of the pattern (numerical values ​​or aspect ratios, etc.). The condition information acquisition unit 11b may also generate an image of the pattern based on the text information input as condition information, and use the generated image as a template image.

[0030] The condition information acquisition unit 11b also accepts input from the user for information such as a title or description for the acquired condition information, such as template images or text information, and stores the condition information together with the received information in the condition information storage unit 12b. The condition information acquisition unit 11b may also acquire information related to the settings of the substrate processing from, for example, the substrate processing device 101, and store the condition information and the information related to the settings of the substrate processing in association with each other in the condition information storage unit 12b. The condition information acquisition unit 11b displays a list of multiple condition information stored in the condition information storage unit 12b on the display unit 14, and acquires the condition information by accepting the selection of one of the displayed multiple condition information. The user can appropriately select which method to input the condition information, for example, on a menu screen.

[0031] The pattern candidate extraction unit 11c extracts one or more pattern candidates from the target image acquired by the image acquisition unit 11a that match the condition information acquired by the condition information acquisition unit 11b. For example, when the condition information acquisition unit 11b acquires a template image as condition information, the pattern candidate extraction unit 11c extracts one or more pattern candidates by performing template matching between the target image and the template image. In this embodiment, the pattern candidate extraction unit 11c also performs edge detection on the target image and the template image as a preprocessing step, and extracts pattern candidates by performing template matching between the edges of the target image and the edges of the template image.

[0032] For example, the pattern candidate extraction unit 11c extracts pattern candidates using a pre-trained deep learning model that has undergone machine learning. Existing deep learning models such as OV-DINO (Open-Vocabrary self-DIstillation with NO labels), OS2D (One-Stage One-Shot Detection), or QATM (Quality-Aware Template Matching for deep learning) can be used as the deep learning model. OV-DINO is a deep learning model that detects objects from target images based on text information. OS2D and QATM are deep learning models that detect objects from target images based on template images. The pattern candidate extraction unit 11c inputs the target image acquired by the image acquisition unit 11a to the deep learning model, and also inputs the template image or text information acquired as condition information by the condition information acquisition unit 11b to the deep learning model as so-called prompt information. The pattern candidate extraction unit 11c acquires the object detection results output by the deep learning model in response to these inputs and extracts one or more detected patterns as pattern candidates.

[0033] Any of the above methods (algorithms) may be used for the pattern candidate extraction process by the pattern candidate extraction unit 11c. Furthermore, the pattern candidate extraction unit 11c may perform the pattern candidate extraction process using each of the above-mentioned algorithms, and the resulting collection of pattern candidates (union) obtained by each algorithm may be used as the final pattern candidate.

[0034] The deletion processing unit 11d performs the process of deleting unsuitable pattern candidates from one or more pattern candidates extracted by the pattern candidate extraction unit 11c. For example, the deletion processing unit 11d performs template matching between the pattern candidate image and the template image of the condition information to calculate the similarity and determines whether the calculated similarity exceeds a predetermined threshold. If the similarity between the pattern candidate and the template image does not exceed the threshold, the deletion processing unit 11d selects this pattern candidate for deletion. Note that the template matching performed by the pattern candidate extraction unit 11c uses the edges of the target image and the edges of the template image, while the template matching performed by the deletion processing unit 11d uses the original image.

[0035] For example, the deletion processing unit 11d deletes pattern candidates using a pre-trained deep learning model that has undergone machine learning. Existing models such as DINO-v2 or ResNet (Residual Network) can be used as the deep learning model. These deep learning models accept image input and output the feature quantities of the input image. The deletion processing unit 11d uses these deep learning models to obtain the feature quantities of each pattern candidate extracted by the pattern candidate extraction unit 11c and the feature quantities of the template image acquired as condition information by the condition information acquisition unit 11b. When text information is used as condition information, the deletion processing unit 11d can, for example, use a multimodal deep learning model capable of handling both text information and images to acquire the feature quantities of both the image and the text information. The deletion processing unit 11d calculates the similarity between the feature quantities of the pattern candidates and the feature quantities of the condition information (template image or text information), and if the similarity does not exceed a predetermined threshold, this pattern candidate is targeted for deletion.

[0036] The processing unit 11 outputs the remaining pattern candidates as the final pattern detection result after the deletion processing unit 11d has deleted unsuitable pattern candidates from the one or more pattern candidates extracted by the pattern candidate extraction unit 11c.

[0037] In addition, if unsuitable pattern candidates are removed from one or more pattern candidates, it may be possible that all pattern candidates are removed and none remain. In such cases, the processing unit 11 may return to the point where it obtains condition information by the condition information acquisition unit 11b, obtain other condition information from the user, and re-detect pattern candidates.

[0038] The display processing unit 11e performs processing to display various characters and images on the display unit 14. In this embodiment, the display processing unit 11e displays the final pattern detection results obtained by the processing of the pattern candidate extraction unit 11c and the deletion processing unit 11d on the display unit 14. For example, the display processing unit 11e generates an image on the display unit 14 by superimposing bounding boxes surrounding each finally detected pattern onto the target image acquired from the electron microscope 102. The display processing unit 11e also displays an input screen on the display unit 14 for the user to input condition information. The display processing unit 11e may also display various other images on the display unit 14.

[0039] <Pattern Detection Processing> Figure 4 is a flowchart showing an example of the procedure for pattern detection processing performed by the information processing device 1 according to this embodiment. In the information processing system according to this embodiment, for example, after various substrate processing is performed in the substrate processing device 101, the processed substrate is transported to the electron microscope 102 and the surface or cross-section is photographed. The electron microscope 102 transmits image data such as SEM images or TEM images of the substrate to the information processing device 1, and the information processing device 1 stores the images received from the electron microscope 102 in the storage unit 12. The user can start pattern detection processing on the image of the substrate taken by the electron microscope 102 by selecting the pattern detection item on a menu screen displayed on the display unit 14 of the information processing device 1, for example. In order to receive input such as the target and conditions for pattern detection from the user, the display processing unit 11e of the processing unit 11 first displays the input screen on the display unit 14 (step S1).

[0040] Figure 5 is a schematic diagram showing an example of an input screen. In this example, the input screen is divided into four main areas horizontally and vertically. The upper left area is for inputting the target image, the upper right area is for text input, the lower left area is for inputting a template image, and the lower right area is where multiple buttons are placed. The information processing device 1 displays a file selection screen (not shown in the diagram) in response to, for example, a mouse click operation on the "target image input" area, reads the image file selected by the user from the storage unit 12, and displays the target image in this area. In the example shown, one image is shown as the target image, but the information processing device 1 may perform pattern detection processing on multiple images.

[0041] The "Text Input" area and the "Template Image Input" area are areas for inputting conditional information related to patterns to be detected from the target image. The information processing device 1 displays a string entered by the user, for example, using the keyboard, in the "Text Input" area and treats the displayed string as text information for the conditional information. The information processing device 1 also accepts a region selection operation from the user using a mouse on the target image displayed in the "Target Image Input" area, extracts the selected image region from the target image as a template image, and displays the extracted template image in the "Template Image Input" area.

[0042] Furthermore, in the information processing system according to this embodiment, the information processing device 1 stores condition information previously used in pattern detection processing in the condition information storage unit 12b. When the information processing device 1 receives a predetermined operation, for example, in the "text input" area or the "template image input" area, it can display a list of text information or template images stored in the condition information storage unit 12b and accept the selection of condition information to be used for the current pattern detection processing from among them.

[0043] Furthermore, in the lower right area of ​​the input screen, there are buttons labeled "Select Extraction Algorithm," "Select Deletion Algorithm," "Set Algorithm," and "Start Pattern Detection," arranged vertically. The "Select Extraction Algorithm" button accepts the selection of an algorithm for performing the pattern candidate extraction process. The information processing device 1 displays a list of selectable algorithms (not shown in the diagram) in response to clicks on the "Select Extraction Algorithm" button and accepts the user's selection of an algorithm. In this embodiment, two algorithms are selectable: a template matching algorithm or an algorithm utilizing a deep learning model, but other algorithms may also be selectable.

[0044] The "Deletion Algorithm Selection" button is a button for accepting the selection of an algorithm for performing the deletion process of pattern candidates. The information processing apparatus 1 displays a list of selectable algorithms (not shown in the figure) in response to a click operation or the like on the "Deletion Algorithm Selection" button, and accepts the selection of an algorithm from the user. In the present embodiment, it is assumed that two algorithms, an algorithm for template matching or an algorithm using a deep learning model, can be selected, but other algorithms may be selectable.

[0045] The "Algorithm Setting" button is a button for accepting various settings related to the algorithm for pattern candidate extraction processing and the algorithm for deletion processing. When a click operation or the like is performed on the "Algorithm Setting" button, the information processing apparatus 1 displays a setting screen (not shown in the figure) according to the selected algorithm for pattern candidate extraction processing and the algorithm for deletion processing. In the present embodiment, the information processing apparatus 1 accepts, for example, the setting of a threshold value for comparison with similarity in the deletion process of pattern candidates on this setting screen.

[0046] The "Pattern Detection Start" button is a button for giving an instruction to start the process of pattern detection for the target image. When the information processing apparatus 1 receives a click operation or the like on the "Pattern Detection Start" button, it starts the process of pattern detection for the target image input in the "Target Image Input" area based on the condition information input in the "Text Input" area and the "Template Image Input" area.

[0047] In the information processing apparatus 1 that displayed the input screen shown in FIG. 5 in step S1 of the flowchart shown in FIG. 4, the image acquisition unit 11a of the processing unit 11 acquires, as a target image, the image input in the “Target Image Input” area of the input screen from the storage unit 12 (step S2). Further, the condition information acquisition unit 11b of the processing unit 11 acquires the text information input in the “Text Input” area of the input screen and / or the template image input in the “Template Image Input” area as condition information (step S3). Further, the processing unit 11 accepts the selection of an algorithm based on the “Extraction Algorithm Selection” button and the “Deletion Algorithm Selection” button on the input screen (step S4).

[0048] After that, when an operation on the “Pattern Detection Start” button on the input screen is performed, the processing unit 11 starts the processing after step S5. The pattern candidate extraction unit 11c of the processing unit 11 performs a pattern candidate extraction process of extracting one or more pattern candidates that satisfy the condition information acquired in step S3 from the target image acquired in step S2 (step S5). Note that the information processing apparatus 1 according to the present embodiment has a plurality of algorithms for performing the pattern candidate extraction process, and in step S5, the pattern candidate extraction unit 11c extracts pattern candidates using the algorithm selected in step S4.

[0049] FIG. 6 is a schematic diagram for explaining the first algorithm of the pattern candidate extraction process. The first algorithm of the pattern candidate extraction process is an algorithm that uses template matching. Therefore, in the first algorithm of the pattern candidate extraction process, it is necessary to input a template image as condition information. The information processing apparatus 1 according to the present embodiment performs data augmentation processing on the template image given as condition information. The data augmentation process is a method of obtaining a plurality of image data from one image data by generating image data obtained by performing various image processes such as rotation, horizontal movement, vertical movement, left - right inversion, up - down inversion, enlargement, or reduction on the given image data. Since the data augmentation process is an existing technique, detailed explanations such as specific methods of image processing are omitted.

[0050] In the first algorithm of the pattern candidate extraction process, template matching is performed, but the information processing device 1 according to this embodiment performs template matching using edge images. For this purpose, as a preprocessing step for template matching, the information processing device 1 performs edge detection on the input target image and on a plurality of data-enhanced template images, respectively, to generate edge images (contour images). For edge detection, existing methods such as using a Sobel filter may be employed, but any method may be used. The information processing device 1 extracts one or more pattern candidates from the target image by performing template matching using the edge images of the target image and the edge images of each template image.

[0051] The template matching process performed by the information processing device 1 according to this embodiment involves calculating the similarity between a portion of the target image (edge ​​image) and the template image (edge ​​image). If the similarity exceeds a threshold, it is determined that an object depicted in the template image is depicted in a portion of the target image, thereby obtaining information (so-called bounding box information) indicating the location of the portion of the target image that contains an object identical or similar to the object depicted in the template image. By repeatedly comparing the template image with a portion of the target image while moving the position of that portion horizontally and vertically, the information processing device 1 can detect one or more locations from the target image as detection results. The information processing device 1 can extract one or more pattern candidates by extracting the image region corresponding to the detected location from the target image.

[0052] In this embodiment, the information processing device 1 increases the number of template images used for template matching by performing data augmentation processing on the input template image. The information processing device 1 performs template matching between the target image and these multiple template images, removes duplicates as necessary from all pattern candidates obtained based on the multiple template images, and uses the remaining one or more pattern candidates as the final output of the pattern candidate extraction process.

[0053] In this embodiment, the first algorithm of the pattern candidate extraction process performed by the information processing device 1 includes data augmentation processing and edge detection processing, but is not limited to this. The information processing device 1 may perform template matching processing using only the input template image without performing data augmentation processing. Alternatively, the information processing device 1 may perform template matching using the original image without converting the target image and template image into edge images through edge detection processing.

[0054] Figure 7 is a schematic diagram illustrating the second algorithm for pattern candidate extraction processing. The second algorithm for pattern candidate extraction processing is an algorithm that utilizes a deep learning model 110 that has been machine-trained to perform pattern detection. The information processing device 1 is equipped with a pre-trained deep learning model 110 such as OV-DINO, OS2D, or QATM (the storage unit 12 stores information such as the model configuration and internal parameters). The deep learning model 110 may be one of these existing pre-trained models that are widely available, or it may be an existing model that has been further trained with information related to substrate processing. In any case, in this embodiment, the deep learning model 110 is a model that has been machine-trained to accept a target image for detection and condition information of a template image and / or text information as input, and to output position information (bounding box information) of the target image in which a pattern matching the condition information is captured. The deep learning model 110 is not limited to OV-DINO, OS2D, and QATM, but any other model may be adopted.

[0055] The information processing device 1 obtains a target image and condition information from the user, inputs the obtained target image and condition information into the deep learning model 110, obtains the position information of the detection result output by the deep learning model 110 in response to this input, and extracts one or more pattern candidates by extracting the image region corresponding to the obtained position information from the target image. At this time, if text information is input as condition information, the information processing device 1 uses, for example, the OV-DINO model as the deep learning model 110 to extract pattern candidates, and if a template image is input as condition information, it uses, for example, the OS2D or QATM model as the deep learning model to extract pattern candidates. Furthermore, if both text information and a template image are input as condition information, the information processing device 1 performs extraction of pattern candidates based on the text information and extraction of pattern candidates based on the template image, respectively, and removes duplicates from the multiple pattern candidates obtained by both extraction processes as necessary, and the remaining one or more pattern candidates are the final output of the pattern candidate extraction process.

[0056] When the information processing device 1 performs the second algorithm for pattern candidate extraction, it may also perform data augmentation processing on the template image and edge detection processing on the target image and template image, similar to the case of the first algorithm described above. The deep learning model 110 may also be provided in a device different from the information processing device 1. The information processing device 1 receives input from the user and transmits the target image and condition information to the other device, and the other device that receives this transmits the results of pattern candidate extraction using the deep learning model 110 back to the information processing device 1, thereby allowing the information processing device 1 to obtain one or more pattern candidates.

[0057] In step S5 of the flowchart shown in Figure 4, the information processing device 1 that performed the pattern candidate extraction process has a deletion processing unit 11d of the processing unit 11 that performs a deletion process to remove unsuitable pattern candidates from the pattern candidates extracted in step S5 (step S6). The information processing device 1 according to this embodiment has multiple algorithms for performing the deletion process, and in step S6, the deletion processing unit 11d deletes the pattern candidates using the algorithm selected in step S4.

[0058] Figure 8 is a schematic diagram illustrating the first algorithm for the deletion process. The first algorithm for the deletion process is an algorithm that utilizes template matching. Therefore, the first algorithm for the deletion process requires a template image to be input as condition information. In the first algorithm for the deletion process, the information processing device 1 performs template matching between one or more pattern candidate images obtained as a result of the pattern candidate extraction process and the template image. The procedure for the template matching process is the same as the template matching process performed in the first algorithm for the pattern candidate extraction process, and the similarity between the pattern candidate image and the template image is obtained during this process. If the size of the pattern candidate image and the size of the template image are different, multiple similarity values ​​may be obtained, but the information processing device 1 adopts, for example, the maximum value of the multiple similarity values ​​(however, the minimum value or average value may also be used).

[0059] The information processing device 1 calculates the similarity between each pattern candidate and the template image, and determines whether the calculated similarity exceeds a predetermined threshold. Based on the results of this threshold determination, the information processing device 1 removes pattern candidates whose similarity does not exceed the threshold as unsuitable. The information processing device 1 outputs the one or more pattern candidates remaining after removing the unsuitable candidates as the final pattern detection result.

[0060] Figure 9 is a schematic diagram illustrating the second algorithm for the deletion process. The second algorithm for the deletion process utilizes a deep learning model 120 that has been machine-trained to convert input image or text information into features and output them. For example, when a template image is used as conditional information, the information processing device 1 can use a model such as DINO-v2 or ResNet as the deep learning model 120 that converts pattern candidates and template images into features. Also, when text information is used or used in combination as conditional information, the information processing device 1 can use a multimodal model capable of handling both image and text information as the deep learning model 120 that converts pattern candidates and text information into features. The deep learning model 120 may be one of these widely available existing trained models, or it may be an existing model that has been further trained with information related to substrate processing. Note that the deep learning model 120 is not limited to those mentioned above, and any model may be used.

[0061] The information processing device 1 inputs one or more pattern candidates obtained as a result of the pattern candidate extraction process into the deep learning model 120, and acquires the feature quantities output by the deep learning model 120, thereby obtaining the feature quantities for each pattern candidate. The information processing device 1 also inputs template image or text information input as condition information into the deep learning model 120, and acquires the feature quantities output by the deep learning model 120, thereby obtaining the feature quantities for the condition information. The information processing device 1 calculates the similarity (e.g., cosine similarity) between the feature quantities of each pattern candidate and the feature quantities of the condition information, and determines whether the calculated similarity exceeds a predetermined threshold. Based on the result of this threshold determination, the information processing device 1 deletes pattern candidates whose similarity does not exceed the threshold as unsuitable. The information processing device 1 outputs the one or more pattern candidates remaining after deleting the unsuitable candidates as the final pattern detection result.

[0062] In step S6 of the flowchart shown in Figure 4, the information processing device 1, which performed the deletion process, uses the pattern candidates that were extracted in step S5 but were not deleted in the deletion process in step S6 as the pattern detection result from the target image, and the display processing unit 11e of the processing unit 11 displays the pattern detection result on the display unit 14 (step S7).

[0063] Figure 10 is a schematic diagram showing an example of how pattern detection results are displayed. In the detection result display screen of the information processing device 1 according to this embodiment, for example as shown in Figure 10, a "Pattern Detection Results" area is provided at the top of the screen, a "Magnified Image" area is provided at the bottom of the screen, and "Edit" and "Complete" buttons are provided in the bottom right corner. The information processing device 1 displays one or more target images on which pattern detection has been performed in the "Pattern Detection Results" area, and displays the pattern detection results by overlaying a bounding box surrounding the detected pattern for each target image.

[0064] In the "Enlarged Image" area of ​​the detection results display screen, the information processing device 1 displays enlarged versions of the patterns within the bounding boxes shown in the "Pattern Detection Results" area. Below each enlarged image of a pattern, the information processing device 1 displays the similarity between the detected pattern image and the conditional information (template image or text information) used in the detection process. This similarity is the same as that calculated during the deletion process.

[0065] In the information processing system according to this embodiment, if a user who has checked the pattern detection results by the information processing device 1 determines that there are false detections, such as detecting something that is not a pattern as a pattern or failing to detect a pattern that should be detected, they can correct the conditions and re-detect the pattern. When the "Correct" button is pressed on the detection result display screen, the information processing device 1 displays a correction screen that accepts corrections to the conditions for re-detecting the pattern. On the other hand, the "Complete" button is pressed when the user determines that there are no errors in the pattern detection results. When the "Complete" button is pressed, the information processing device 1 stores the pattern detection results in the storage unit 12 and stores the condition information used for pattern detection in the condition information storage unit 12b. The condition information stored in the condition information storage unit 12b can be reused when performing subsequent pattern detection processing.

[0066] Figure 11 is a schematic diagram showing an example of a pattern detection condition modification screen. The condition modification screen displayed by the information processing device 1 according to this embodiment includes, for example, a "Pattern Detection Results" area at the top of the screen, a "Template Settings" area at the bottom of the screen, and a "Re-detect" button in the lower right corner of the screen. The information processing device 1 displays the same content in the "Pattern Detection Results" area of ​​the condition modification screen as in the "Pattern Detection Results" area of ​​the detection result display screen shown in Figure 10. However, in the condition modification screen, the information processing device 1 accepts the selection of one target image from among the one or more target images displayed in the "Pattern Detection Results" area. In the illustrated example, the second target image displayed from the left is selected by the user, and the information processing device 1 indicates that it is selected by displaying a rectangular frame surrounding this target image.

[0067] The information processing device 1 displays the target image selected in the "Pattern Detection Result" area, enlarged, in the "Template Settings" area. The information processing device 1 accepts area specification operations on the target image displayed in the "Template Settings" area, for example, using the mouse on the operation unit 15, and sets the image of the area specified by the user as the new template image. In the illustrated example, the user has specified four areas on the target image, and rectangular frames surrounding the specified areas are displayed superimposed on the target image.

[0068] After the information processing device 1 receives the setting of a new template image in the "Template Settings" area, if it receives an operation on the "Re-detect" button, it adds the new template image to the previous condition information and performs pattern detection on one or more target images again. However, the information processing device 1 may discard the previous condition information and perform re-detection using the new template image. In this example, the information processing device 1 accepts the setting of a template image as condition information for performing re-detection, but is not limited to this, and may also accept input of text information to perform re-detection, and may even accept a change in the algorithm used.

[0069] In step S7 of the flowchart shown in Figure 4, the information processing device 1, which displays the pattern detection result, determines whether or not an operation has been performed on the "Modify" button on the detection result display screen (step S8). If an operation has been performed on the "Modify" button (S8: YES), the display processing unit 11e displays the condition modification screen shown in Figure 11 on the display unit 14, the condition information acquisition unit 11b acquires new condition information (step S9), and the process returns to step S5 to re-detect the pattern.

[0070] If no action is taken on the "Modify" button (S8: NO) and an action is taken on the "Complete" button, the processing unit 11 stores the pattern detection results obtained in the processing up to this point in the storage unit 12 (step S10). The processing unit 11 also stores the condition information of the template image and text information entered by the user in steps S3 and S9 in the condition information storage unit 12b (step S11), and terminates the processing.

[0071] <Summary> In the information processing system according to this embodiment with the above configuration, the information processing device 1 performs the process of detecting a predetermined pattern formed on a substrate from a target image obtained by an electron microscope 102 that photographs a substrate processed by the substrate processing device 101. The information processing device 1 acquires condition information regarding the pattern to be detected and extracts one or more pattern candidates from the target image based on the acquired condition information. From the extracted one or more pattern candidates, the information processing device 1 deletes unsuitable pattern candidates based on a comparison between the condition information and each pattern candidate, and outputs the pattern candidates that were not deleted as the pattern detection result. As a result, the information processing system according to this embodiment is expected to be able to detect various patterns from a target image without having to perform machine learning on the pattern to be detected in advance.

[0072] Furthermore, in the information processing system according to this embodiment, the information processing device 1 acquires a template image in which the pattern to be detected is depicted as conditional information, and extracts pattern candidates by performing template matching using the template image and the target image. As a result, the information processing system according to this embodiment is expected to be able to extract pattern candidates from the target image without having to perform machine learning on the pattern to be detected in advance.

[0073] Furthermore, in the information processing system according to this embodiment, the information processing device 1 performs edge detection on the template image and the target image to generate edge images (contour images), and extracts pattern candidates by performing template matching using the contour images of the template image and the target image. As a result, the information processing system according to this embodiment is expected to improve the accuracy of pattern candidate extraction compared to the case where template matching is performed using the original image without using edge images.

[0074] Furthermore, in the information processing system according to this embodiment, the information processing device 1 acquires a template image on which the pattern to be detected is depicted or text information relating to the pattern to be detected as condition information. The information processing device 1 accepts the condition information and the target image as input and uses a pre-trained deep learning model 110 that has been pre-machine-trained to detect objects that satisfy the condition information from the target image. The acquired condition information and the target image are input to the deep learning model 110, and the detection results output by the deep learning model 110 are obtained. Pattern candidates are extracted based on the detection results. Thus, the information processing system according to this embodiment is expected to be able to extract pattern candidates from the target image without performing any additional machine learning, etc., by utilizing a widely publicized or provided pre-trained deep learning model 110. Furthermore, by using a multimodal deep learning model 110, it becomes possible to use not only template images but also text information relating to the pattern to be detected as condition information.

[0075] Furthermore, in the information processing system according to this embodiment, the information processing device 1 acquires a template image in which the pattern to be detected is depicted as conditional information, calculates the similarity between the template image and the pattern candidates, and deletes unsuitable pattern candidates based on the calculated similarity. As a result, the information processing system according to this embodiment is expected to obtain appropriate pattern detection results by deleting unsuitable ones from the pattern candidates extracted from the target image without having to perform machine learning on the pattern to be detected in advance.

[0076] Furthermore, in the information processing system according to this embodiment, the information processing device 1 uses a pre-trained deep learning model 120, which has been pre-machine-trained to output the features of input information, to acquire the features of condition information and the features of pattern candidates, respectively. It then calculates the similarity between the features of condition information and the features of pattern candidates, and deletes unsuitable pattern candidates based on the calculated similarity. As a result, the information processing system according to this embodiment can acquire the features of condition information and pattern candidates without performing any additional machine learning, by utilizing a widely available or provided pre-trained deep learning model 120, and is expected to obtain appropriate pattern detection results by deleting unsuitable pattern candidates based on the similarity of the features.

[0077] Furthermore, in the information processing system according to this embodiment, the information processing device 1 accepts the selection of an algorithm for extracting pattern candidates from a target image or an algorithm for deleting unsuitable pattern candidates, and performs processing using the selected algorithm. As a result, the information processing system according to this embodiment is expected to allow the user to switch between various algorithms and verify the results of pattern detection.

[0078] Furthermore, in the information processing system according to this embodiment, the information processing device 1 overlays the pattern detection result onto the target image, for example, as a bounding box, and displays it. This allows the information processing system according to this embodiment to provide the pattern detection result in a way that is easy for the user to understand.

[0079] Furthermore, in the information processing system according to this embodiment, the information processing device 1 displays the pattern detection result and the similarity between the detection result and the condition information in correspondence. Through these means, the information processing system according to this embodiment is expected to allow the user to easily understand how well the detection result matches the condition information.

[0080] Furthermore, in the information processing system according to this embodiment, the information processing device 1 accepts input of conditional information based on the target image output along with the pattern detection result, and performs re-detection of the pattern from the target image based on the accepted conditional information. As a result, the information processing system according to this embodiment allows the user to check the pattern detection result and perform re-detection to correct any false detections.

[0081] Furthermore, in the information processing system according to this embodiment, the information processing device 1 stores the acquired condition information in the condition information storage unit 12b, displays the stored condition information and accepts a selection from it, and acquires the selected condition information to perform pattern detection. As a result, the information processing system according to this embodiment is expected to facilitate the input of condition information by the user when detecting a pattern similar to that detected in the past.

[0082] <Embodiment 2> Figure 12 is a schematic diagram illustrating the outline of the pattern detection process performed by the information processing system according to Embodiment 2. The information processing device 1 according to Embodiment 2 performs the pattern candidate extraction process and deletion process in the pattern detection process using multiple algorithms. The information processing device 1 according to Embodiment 2 receives input of the target image and condition information for pattern detection and performs both a pattern candidate extraction process using a first algorithm that uses template matching as shown in Figure 6, and a pattern candidate extraction process using a second algorithm that uses the deep learning model 110 shown in Figure 7. These two types of algorithmic pattern candidate extraction processes can be performed in parallel. The information processing device 1 obtains the final pattern candidate by appropriately deleting duplicates from the pattern candidates obtained by the two types of algorithmic pattern candidate extraction processes and integrating (merging) them.

[0083] After obtaining pattern candidates, the information processing device 1 according to Embodiment 2 sequentially performs a deletion process using a first algorithm that employs template matching as shown in Figure 8, and a deletion process using a second algorithm that employs the deep learning model 120 as shown in Figure 9. These two types of deletion processes can be performed in series. That is, the information processing device 1 deletes unsuitable pattern candidates from the pattern candidates obtained by the pattern candidate extraction process using the first algorithm, and then deletes unsuitable pattern candidates from the remaining pattern candidates using the second algorithm. The information processing device 1 outputs the remaining pattern candidates as the final pattern detection result.

[0084] In this example, pattern candidate extraction and deletion are performed using two types of algorithms each, but this is not limited to this, and pattern candidate extraction and deletion may each be performed using three or more types of algorithms. Furthermore, the number of algorithms for pattern candidate extraction and deletion do not need to be the same; for example, pattern candidate extraction may be performed using one algorithm and deletion may be performed using three algorithms, and various combinations of patterns may be used for pattern detection.

[0085] In the information processing system according to Embodiment 2 with the above configuration, the information processing device 1 performs the process of extracting pattern candidates from the target image using multiple different algorithms, and integrates the multiple extraction results from the multiple algorithms (union) as the final pattern candidate, deleting unsuitable pattern candidates from the pattern candidate. As a result, the information processing system according to this embodiment is expected to be able to extract more pattern candidates from the target image.

[0086] Furthermore, in the information processing system according to Embodiment 2, the information processing device 1 performs the process of deleting unsuitable pattern candidates from one or more pattern candidates using multiple different algorithms, and outputs the one or more pattern candidates that were not deleted by the multiple algorithms as the final pattern detection result. As a result, the information processing system according to this embodiment is expected to delete unsuitable pattern candidates from a large number of pattern candidates with greater accuracy.

[0087] Since the other components of the information processing system according to Embodiment 2 are the same as those of the information processing system according to Embodiment 1, the same reference numerals are used for the same parts, and detailed explanations are omitted.

[0088] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of this disclosure is indicated by the claims, not in the sense described above, and all modifications within the meaning and scope equivalent to the claims are intended.

[0089] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used.

[0090] 1 Information Processing Device (Computer) 11 Processing Unit 11a Image Acquisition Unit 11b Condition Information Acquisition Unit 11c Pattern Candidate Extraction Unit 11d Deletion Processing Unit 11e Display Processing Unit 12 Storage Unit 12a Program (Computer Program) 12b Condition Information Storage Unit 13 Communication Unit 14 Display Unit 15 Operation Unit 99 Recording Medium 101 Substrate Processing Device 102 Electron Microscope 110 Deep Learning Model 120 Deep Learning Model

Claims

1. A computer program that causes a computer to perform a process to detect a predetermined pattern formed on a circuit board from a target image of a circuit board that has undergone circuit board processing, wherein the computer obtains condition information relating to the pattern to be detected, extracts one or more pattern candidates from the target image based on the obtained condition information, deletes pattern candidates from the extracted one or more pattern candidates based on a comparison between the condition information and each pattern candidate, and outputs the one or more pattern candidates that were not deleted as the pattern detection result.

2. The computer program according to claim 1, wherein the program obtains a template image in which the pattern to be detected is depicted as the condition information, generates contour images of the template image and the target image, and extracts the pattern candidates by matching the contour image of the template image and the contour image of the target image.

3. The computer program according to claim 1, wherein the program obtains a template image in which the pattern to be detected is depicted or text information relating to the pattern to be detected as conditional information, inputs the obtained conditional information and the target image into a trained deep learning model that accepts the conditional information and the target image as input and detects objects that satisfy the conditional information from the target image, obtains the detection result output by the deep learning model, and extracts the pattern candidates based on the detection result.

4. The computer program according to claim 1, which acquires a template image in which the pattern to be detected is depicted as condition information, calculates the similarity between the acquired template image and the pattern candidate, and deletes a pattern candidate from the one or more pattern candidates based on the calculated similarity.

5. A computer program according to claim 1, which uses a trained deep learning model that outputs features of input information to obtain features of the condition information and features of the pattern candidates, respectively, calculates the similarity between the obtained features of the condition information and features of the pattern candidates, and deletes a pattern candidate from the one or more pattern candidates based on the calculated similarity.

6. The computer program according to claim 1, which extracts the pattern candidates from the target image using a plurality of different algorithms, and deletes the pattern candidates from the union of the plurality of extraction results from the plurality of algorithms.

7. The computer program according to claim 1, wherein one or more pattern candidates are removed from the aforementioned pattern candidates using a plurality of different algorithms, and the one or more pattern candidates that were not removed by the plurality of algorithms are output as the pattern detection result.

8. The computer program according to claim 1, which accepts selection of an algorithm for extracting pattern candidates from the target image, or an algorithm for deleting pattern candidates.

9. The computer program according to claim 1, which superimposes the detection result of the pattern onto the target image and outputs it.

10. The computer program according to claim 1, which outputs the detection result of the pattern and the similarity between the detection result and the condition information in association.

11. A computer program according to claim 1, which receives input of condition information based on the target image output along with the detection result, and re-detects the pattern from the target image based on the received condition information.

12. A computer program according to claim 1, which stores the acquired condition information in a storage unit, outputs one or more condition information stored in the storage unit, accepts a selection from the output one or more condition information, and retrieves the selected condition information from the storage unit to perform pattern detection.

13. An information processing method comprising an information processing device detecting a predetermined pattern formed on a substrate from a target image of a substrate that has undergone substrate processing, wherein the information processing device acquires condition information relating to the pattern to be detected, extracts one or more pattern candidates from the target image based on the acquired condition information, deletes pattern candidates from the extracted one or more pattern candidates based on a comparison between the condition information and each pattern candidate, and outputs the one or more pattern candidates that were not deleted as the pattern detection result.

14. An information processing apparatus comprising a processing unit that performs a process of detecting a predetermined pattern formed on a substrate from a target image of a substrate that has undergone substrate processing, wherein the processing unit acquires condition information relating to the pattern to be detected, extracts one or more pattern candidates from the target image based on the acquired condition information, deletes pattern candidates from the extracted one or more pattern candidates based on a comparison between the condition information and each pattern candidate, and outputs the one or more pattern candidates that were not deleted as the pattern detection result.

Images