Inspection image relearning system, inspection image relearning method, and program

Abstract

[Problem] To improve the accuracy of remote inspection using inspection images of an inspection subject. [Solution] An inspection image relearning system according to the present invention, which performs remote inspection by using inspection images of an inspection subject, acquires first inspection images, selects one or more training images from among the first inspection images, acquires first evaluations for the training images, configures a training image set on the basis of the first evaluations, sets an abnormality detection area for one or more first inspection images among the first inspection images, performs learning by using the training image set, creates a trained model on the basis of the learning, acquires a second inspection image, estimates an abnormality score or a normality score for the second inspection image by using the trained model, provides abnormality-related information in the set abnormality detection area on the basis of the abnormality score or the normality score, acquires a second evaluation for the abnormality-related information, reconfigures the training image set by reflecting the second evaluation, and performs relearning by using the training image set.

Description

Inspection Image Relearning System, Inspection Image Relearning Method, and Program

[0001] The present invention relates to a technology effective for remote inspection using inspection images of an inspection target.

[0002] In recent years, technologies related to remote inspection using inspection images have attracted attention. For example, Patent Document 1 provides a technology for accurately automatically obtaining the indicated value of an indicating instrument from an image of the indicating instrument. In addition, Patent Document 2 provides a technology for ensuring a necessary inspection system while reducing the burden on maintenance personnel in the inspection of substation equipment within a substation.

[0003] Japanese Patent Application Laid-Open No. 2002-185364, Japanese Patent Application Laid-Open No. 2023-177995

[0004] At the site where inspection work such as equipment maintenance is being carried out, it is necessary to regularly inspect the presence or absence of abnormalities in the inspection target. However, remote inspection using images can perform the inspection work more efficiently than visual inspection, so remote inspection using images is being carried out. However, there is a problem that the inspection accuracy of remote inspection using inspection images of an inspection target at a site where inspection work such as equipment maintenance is being carried out is low and cannot withstand practical use. Therefore, a technology for improving the inspection accuracy of remote inspection using inspection images of an inspection target is required. However, the technologies described in Patent Documents 1 and 2 cannot improve the inspection accuracy of remote inspection using inspection images of an inspection target. Therefore, the inventor of the present invention focused on a mechanism for reflecting the evaluation of the degree of abnormality or the degree of normality estimated from the inspection image, reconstructing the configuration of the learning image set of the inspection image of the inspection target, and relearning.

[0005] In view of such problems, an object of the present invention is to provide an inspection image relearning system, an inspection image relearning method, and a program that can improve the inspection accuracy of remote inspection using inspection images of an inspection target by reflecting the evaluation of the degree of abnormality or the degree of normality estimated from the inspection image, reconstructing the configuration of the learning image set of the inspection image of the inspection target, and relearning.

[0006] The present invention relates to an inspection image retraining system for remote inspection using inspection images of an object to be inspected, comprising: a first inspection image acquisition unit that acquires first inspection images of the object to be inspected taken with a camera; a selection unit that selects one or more training images from the acquired first inspection images; a first evaluation acquisition unit that acquires a first evaluation indicating the degree of abnormality or normality for the selected training images; a configuration unit that constructs a training image set based on the acquired first evaluation; a setting unit that sets an abnormality detection area for one or more of the selected first inspection images; a learning unit that learns using the configured training image set; a model creation unit that creates a trained model based on the learning; a second inspection image acquisition unit that acquires second inspection images of the object to be inspected taken with a camera; an estimation unit that estimates an abnormality score or a normal score for the acquired second inspection image using the created trained model; and a provision unit that provides abnormality-related information within the set abnormality detection area based on the estimated abnormality score or normal score. The present invention provides an inspection image relearning system comprising: a second evaluation acquisition unit that acquires a second evaluation indicating the degree of abnormality or normality for the provided abnormality-related information; a reconstruction unit that reconstructs the learning image set reflecting the acquired second evaluation; and a relearning unit that relearns using the reconstructed learning image set.

[0007] According to the present invention, by reflecting the evaluation of the degree of abnormality or normality estimated from the inspection images, and by reconstructing the configuration of the training image set of inspection images of the subject to be inspected and retraining it, it is possible to improve the inspection accuracy of remote inspections using inspection images of the subject to be inspected.

[0008] Although this invention falls under the category of a system, similar effects and benefits can be achieved with methods and programs.

[0009] According to the present invention, it is possible to improve the accuracy of remote inspections using inspection images of the object being inspected.

[0010] This diagram illustrates the overview of the inspection image relearning system 1. This diagram shows the functional configuration of the inspection image relearning system 1. This diagram shows a flowchart of the learning process performed by the inspection image relearning system 1. This diagram schematically shows the state in which the information terminal 3 has received the selection of learning images. This diagram schematically shows the state in which the information terminal 3 has received the first evaluation. This diagram schematically shows the state in which the computer 10 has configured the learning image set. This diagram schematically shows the state in which the computer 10 has set the anomaly detection area. This diagram shows a flowchart of the reset process performed by the inspection image relearning system 1. This diagram shows a flowchart of the first provision process performed by the inspection image relearning system 1. This diagram schematically shows the state in which the computer 10 has estimated the anomaly score. This diagram schematically shows the screen displayed by the information terminal 3 when the computer 10 provides anomaly-related information. This diagram shows a flowchart of the second provision process performed by the inspection image relearning system 1. This diagram shows a flowchart of the relearning process performed by the inspection image relearning system 1. This diagram schematically shows the state in which information terminal 3 has received the second evaluation.

[0011] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the attached drawings. In the following drawings, the same elements are denoted by the same numbers or reference numerals throughout the description of the embodiments.

[0012] [Overview of Inspection Image Retraining System 1] Figure 1 is a schematic diagram illustrating the overview of the Inspection Image Retraining System 1. Based on Figure 1, the components of the Inspection Image Retraining System 1 will be described. The Inspection Image Retraining System 1 is a system that performs remote inspections using inspection images of the object to be inspected, comprising at least a computer 10 with server functionality. In this embodiment, in addition to the computer 10, the Inspection Image Retraining System 1 is a system that includes an information terminal 3 used by users such as inspectors and system administrators who perform remote inspections, and a camera 4 for photographing the object to be inspected.

[0013] Computer 10 has server functionality and may be implemented using, for example, a single computer, or multiple computers, like a cloud computer. In this specification, a cloud computer may be one that uses any computer scalably to perform a specific function, or one that includes multiple functional modules to implement a certain system and uses those functions in any combination.

[0014] Information terminal 3 is a terminal device used by users such as inspectors and administrators, and is, for example, a terminal device such as a mobile phone, smartphone, tablet device, personal computer, laptop computer, or a wearable device such as a smartwatch, smart glasses, or HMD (Head Mounted Display).

[0015] Camera 4 is a camera that photographs the object to be inspected. Camera 4 is a camera that photographs the object to be inspected, such as equipment, river water levels, or people. Camera 4 may be a fixed camera fixed at any point from which the object to be inspected can be photographed, or it may be attached to a mobile robot or drone. The images that camera 4 captures may be not only still images but also videos. When camera 4 captures video, it may treat the video itself as an image, or it may extract a part of the video and treat it as an image.

[0016] Furthermore, the inspection image relearning system 1 may include other terminals and devices such as external systems, in addition to the information terminal 3, camera 4, and computer 10 described above. The number, types, and functions of these devices are not particularly limited and can be designed as appropriate.

[0017] This section outlines the processing steps involved when the inspection image retraining system 1 performs a remote inspection using inspection images of the subject being inspected.

[0018] Computer 10 acquires a first inspection image of the object to be inspected, captured by camera 4 (step S1). Computer 10 acquires the first inspection image from camera 4.

[0019] Computer 10 selects one or more training images from the acquired first inspection images (step S2). Computer 10 displays the acquired first inspection images on the information terminal 3 and accepts the user's selection of training images. Computer 10 selects the training images accepted by the information terminal 3 as training images.

[0020] Computer 10 obtains a first evaluation indicating the degree of abnormality or normality for the selected training image (step S3). Computer 10 displays the selected training image on the information terminal 3 and receives input of the first evaluation from the user. Computer 10 obtains the first evaluation received by the information terminal 3.

[0021] Computer 10 constructs a training image set based on the acquired first evaluation (step S4). Based on the acquired first evaluation, Computer 10 uses abnormal images evaluated as abnormal as training images and normal images evaluated as normal as training images, and combines them to construct a training image set of abnormal images and a training image set of normal images.

[0022] Computer 10 sets an anomaly detection area for one or more of the selected first inspection images (step S5). Computer 10 displays the selected first inspection images on the information terminal 3 and accepts input from the user regarding the setting of the anomaly detection area. Computer 10 acquires the setting of the anomaly detection area received by the information terminal 3.

[0023] Computer 10 performs training using the configured training image set (step S6). Computer 10 performs training using the training image set according to a predetermined training method.

[0024] Computer 10 creates a trained model based on the learning (step S7). Computer 10 creates a trained model based on the learning results using a predetermined algorithm.

[0025] Computer 10 acquires a second inspection image of the object to be inspected, captured by camera 4 (step S8). Computer 10 acquires the second inspection image from camera 4. If the field of view differs between the first inspection image and the second inspection image, computer 10 performs field of view correction.

[0026] Computer 10 uses the created trained model to estimate an abnormal score or a normal score for the acquired second inspection image (step S9). Based on the degree of agreement and similarity between the acquired second inspection image and the created trained model, Computer 10 estimates whether the inspection object shown in the second inspection image is abnormal or normal, and estimates a score for the second inspection image based on the estimation result. Computer 10 estimates the score of the second inspection image that is estimated to be abnormal as the abnormal score, and the score of the second inspection image that is estimated to be normal as the normal score.

[0027] The computer 10 provides abnormality-related information within the set abnormality detection area based on the estimated abnormality score or normal score (step S10). The computer 10 displays abnormality-related information on the information terminal 3, including whether something is abnormal or normal, abnormality score or normal score on a pixel-by-pixel or image-by-image basis, abnormality label or normal label on a pixel-by-pixel or image-by-image basis based on the abnormality score or normal score, a heat map on a pixel-by-pixel or image-by-image basis based on the abnormality score or normal score on a pixel-by-pixel basis, a grayscale image on a pixel-by-pixel or image-by-image basis obtained by thresholding the abnormality score or normal score on a pixel-by-pixel or image-by-image basis, and provides abnormality-related information.

[0028] Computer 10 obtains a second evaluation indicating the degree of abnormality or normality for the provided abnormality-related information (step S11). Information terminal 3 receives input of the second evaluation from the user for the provided abnormality-related information. Computer 10 obtains the second evaluation received by information terminal 3.

[0029] Computer 10 reconstructs the training image set to reflect the acquired second evaluation (step S12). Computer 10 adds second inspection images that received an abnormal second evaluation as abnormal images to the training image set of abnormal images, and adds second inspection images that received a normal second evaluation as normal images to the training image set of normal images.

[0030] Computer 10 retrains using the reconstructed training image set (step S13). Computer 10 retrains using the reconstructed training image set according to a predetermined training method. Computer 10 recreates the trained model based on the retraining results using a predetermined algorithm.

[0031] The above is an overview of the inspection image retraining system 1. The inspection image retraining system 1 makes it possible to improve the accuracy of remote inspections using inspection images of the subject being inspected.

[0032] [Device Configuration] Figure 2 is a block diagram showing the configuration of the inspection image relearning system 1. Based on Figure 2, the device configuration of the inspection image relearning system 1 will be described. The inspection image relearning system 1 is a system that performs remote inspection using inspection images of the object to be inspected, and is composed of at least a computer 10. In this embodiment, the inspection image relearning system 1 includes an information terminal 3 and a camera 4 in addition to the computer 10. The inspection image relearning system 1 is a system in which the computer 10 is connected to the information terminal 3 and the camera 4 via a network 8 such as a public telephone network, enabling data communication. In addition to the information terminal 3, camera 4 and computer 10, the inspection image relearning system 1 may also include external systems, printing devices, and other terminals and devices, and their number, types, and functions can be designed as appropriate.

[0033] Information terminal 3 is the aforementioned terminal device used by users such as inspectors and administrators. Information terminal 3 includes a terminal control unit comprising a CPU (Central Processing Unit), GPU (Graphics Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and a communication unit comprising devices that enable communication with other terminals and devices. Furthermore, information terminal 3 includes an input / output unit comprising various devices that perform tasks such as receiving predetermined inputs and inputting and outputting various types of data.

[0034] Camera 4 is a camera that is fixed at any point where the object to be inspected is to be photographed, or is attached to a moving object. Camera 4 takes still images or videos of the object to be inspected and transmits the captured images as inspection images to the computer 10 or information terminal 3. Camera 4 may treat the captured still images as inspection images, or it may extract a part of the captured video as a still image and treat the extracted still image as an inspection image, or it may treat the captured video as an inspection image.

[0035] Computer 10 has server functionality and may be implemented as a single computer, or as a cloud computer, as it is implemented as multiple computers. Computer 10 includes a control unit with a CPU, GPU, RAM, ROM, etc., and a communication unit with a device for communicating with other terminals and devices, a first inspection image acquisition unit for acquiring a first inspection image of the inspection target captured by camera 4, a first evaluation acquisition unit for acquiring a first evaluation indicating the degree of abnormality or normality for selected training images, a second inspection image acquisition unit for acquiring a second inspection image of the inspection target captured by camera 4, a provision unit for providing abnormality-related information within a set abnormality detection area based on an estimated abnormality score or normality score, and a second evaluation acquisition unit for acquiring a second evaluation indicating the degree of abnormality or normality for the provided abnormality-related information. The computer 10 includes, as a processing unit, various devices for executing various processes, a selection unit for selecting one or more training images from the acquired first inspection images, a configuration unit for configuring a training image set based on the acquired first evaluation, a setting unit for setting an anomaly detection area for one or more of the selected first inspection images, a learning unit for learning using the configured training image set, a model creation unit for creating a trained model based on the learning, an estimation unit for estimating an anomaly score or a normal score for the acquired second inspection images using the created trained model, a reconstruction unit for reconstructing the training image set reflecting the acquired second evaluation, and a retraining unit for retraining using the reconstructed training image set.

[0036] In computer 10, the control unit loads a predetermined program and, in cooperation with the communication unit, realizes a first inspection image acquisition module, a first evaluation acquisition module, a second inspection image acquisition module, a provision module, a count alert module, a frequency alert module, and a second evaluation acquisition module. Also in computer 10, the control unit loads a predetermined program and, in cooperation with the processing unit, realizes a selection module, a configuration module, a setting module, a learning module, a model creation module, a change determination module, an estimation module, an assignment module, a grayscale image creation module, a map creation module, a graph creation module, a correction module, a count determination module, a frequency determination module, a reconstruction module, a retraining module, and a model recreation module.

[0037] The following describes each process performed by the inspection image relearning system 1, along with the processes performed by each module described above. In this specification, each module may perform its processing as a function of its own, or it may perform it through a predetermined application.

[0038] [Learning Process Executed by Inspection Image Retraining System 1] Based on Figure 3, the learning process executed by the inspection image retraining system 1 will be explained. The same figure is a flowchart of the learning process executed by the computer 10. The learning process consists of the following steps: first inspection image acquisition process (step S1) to acquire first inspection images taken by the camera 4 of the object to be inspected; selection process (step S2) to select one or more training images from the acquired first inspection images; first evaluation acquisition process (step S3) to acquire a first evaluation indicating the degree of abnormality or normality for the selected training images; configuration process (step S4) to construct a training image set based on the acquired first evaluation; setting process (step S5) to set an anomaly detection area for one or more of the selected first inspection images; learning process (step S6) to learn using the configured training image set; and model creation process (step S7) to create a trained model based on the learning.

[0039] The first inspection image acquisition module acquires a first inspection image of the inspection target captured by camera 4 (step S20). The inspection target may include not only various equipment (power receiving equipment, generator equipment, fire prevention equipment, building equipment, air conditioning equipment, etc.), but also river water levels, people for the purpose of counting the number of people, etc., and may include not only goods but also non-goods. The first inspection image is a still image or video taken by camera 4. Furthermore, the first inspection image may be an image extracted from a video taken by camera 4. The first inspection image is an image that shows at least part or all of the specific inspection target. For example, if the inspection target is a pipe that carries rainwater, the first inspection image is an image that shows the inspection target area (pipe, drain, etc.). Camera 4 takes pictures of the inspection target at pre-set intervals (hourly intervals, daily intervals, weekly intervals, etc.), at arbitrary timings (pre-set dates and times, etc.), or at predetermined timings (when a malfunction of equipment is detected by a connected sensor, when the river water level rises, when there is a sudden change in the number of people, etc.). At this time, it is preferable for camera 4 to photograph the inspection target multiple times, as the way light falls and the amount of rain may differ depending on the season, time of day (morning, noon, night, etc.), weather, etc. Camera 4 transmits the images it has taken of the inspection target to computer 10 as the first inspection image. If camera 4 photographs the inspection target only once, it transmits one image to computer 10 as the first inspection image; if it photographs it multiple times, it transmits multiple images to computer 10 as the first inspection image. At this time, camera 4 transmits the first inspection image to computer 10 including various metadata (dimensions, date and time of shooting, timing of shooting, resolution, shooting location, exposure amount, lens focal length, presence or absence of flash, location information, size, etc.). The first inspection image acquisition module receives this first inspection image and acquires the first inspection image of the inspection target taken by camera 4.

[0040] The selection module selects one or more training images from the acquired first inspection images (step S21). The computer 10 transmits the acquired first inspection images to the information terminal 3. The information terminal 3 receives these first inspection images and displays them on its display unit via a predetermined UI (User Interface), etc. The information terminal 3 accepts the user's selection of images to be used as training images from among the displayed first inspection images (see Figure 4). The information terminal 3 accepts the selection of at least one image from among the multiple displayed first inspection images. Here, the information terminal 3 may accept the selection of only images showing an abnormal state (a state in which some problem has occurred with the inspection target), or it may accept the selection of only images showing a normal state (a state in which no problem has occurred with the inspection target), or it may accept the selection of images showing both abnormal and normal states. The information terminal 3 transmits the selection data (image data, metadata, etc.) related to the selected first inspection images to the computer 10. Computer 10 receives this selection data. Based on this selection data, the selection module selects one or more training images from the acquired first inspection images. The selection module may also select training images using AI (Artificial Intelligence) instead of selecting training images selected by the user.

[0041] Referring to Figure 4, the state in which the information terminal 3 accepts the selection of learning images will be explained. This figure schematically shows the state in which the information terminal 3 has accepted the selection of learning images. The information terminal 3 displays four first inspection images 21 (21a to 21d) via the UI 20. The information terminal 3 accepts input (tap, etc.) from the user 22 for the first inspection images 21 that the user 22 desires as learning images. When the information terminal 3 receives input from the user 22 for the first inspection image 21a, it displays a check mark 23 on the first inspection image 21a to indicate that the selection of this first inspection image 21a has been accepted. The information terminal 3 displays check marks 23 on the other first inspection images 21b to 21d that have been accepted for selection, in the same way as the first inspection image 21a. Note that the display method by which the information terminal 3 indicates that the selection of a first inspection image has been accepted is not limited to a check mark; other methods (adding an icon, changing the color, etc.) may also be used. The information terminal 3 transmits the selection data for the first inspection images 21a and 21b, which display the check mark 23, to the computer 10. The computer 10 receives this selection data for the first inspection images 21a and 21b. Based on the selection data for the first inspection images 21a and 21b, the selection module selects the first inspection images corresponding to the first inspection images 21a and 21b from the acquired first inspection images as training images.

[0042] Returning to Figure 3, the learning process will be explained further. The first evaluation acquisition module acquires a first evaluation indicating the degree of abnormality or normality for the selected training images (step S22). The first evaluation may be a label (abnormal, normal, etc.), a numerical representation of each degree (abnormal score (score above a threshold), normal score (score below a threshold), etc.), a comment regarding each degree (abnormality occurring, normal, no problem, etc.), or something else. This first evaluation may be applied to the training image unit, to any pixel unit in the training image, a combination of both, or something else. The computer 10 transmits all or some of the selected training images to the information terminal 3. The information terminal 3 receives these training images and displays them on its display unit via a predetermined UI, etc. The information terminal 3 accepts input from the user for the first evaluation indicating the degree of abnormality or normality for the displayed training images (see Figure 5). Information terminal 3 accepts user input (such as selection or direct input of labels, scores, and comments) and assigns the received labels, scores, and comments as a first evaluation to each displayed training image. Information terminal 3 transmits the assigned first evaluation to computer 10. At this time, information terminal 3 also transmits training data (image data, etc.) related to the training image to which the first evaluation has been assigned to computer 10. The first evaluation acquisition module receives this first evaluation and training data and acquires a first evaluation indicating the degree of abnormality or normality for the selected training image. Note that the first evaluation acquisition module may acquire the first evaluation using AI instead of acquiring the first evaluation entered by the user.

[0043] Referring to Figure 5, the state in which the information terminal 3 accepts the first evaluation will be explained. The figure schematically shows the state in which the information terminal 3 has accepted the first evaluation. The information terminal 3 displays two training images 31 (31a, 31b) via the UI 30. The information terminal 3 accepts input from the user, such as the assignment of labels 32, the assignment of scores 33, and the assignment of comments 34 (selection input, direct input, etc.). The information terminal 3 assigns the content received from the user to the training images 31 (31a, 31b) as labels 32 (32a, 32b), scores 33 (33a, 33b), and comments 34 (34a, 34b), respectively, and displays the assigned content. Training image 31a is assigned "Normal" as label 32a, a score of 3.5 as score, and no comment 34a has been assigned. The training image 31b is assigned the label 32b indicating an abnormality, a score of 78.5, and a comment 34a indicating the presence of a foreign object. The information terminal 3 similarly accepts the first evaluation for each of the other training images and assigns the received first evaluation to each training image. Note that the method by which the information terminal 3 assigns the first evaluation is not limited to the illustrated method, and other methods (such as assigning an icon or changing the color) may also be used. Furthermore, the content of the first evaluation may also be in a manner other than a label, score, or comment. The information terminal 3 transmits the first evaluation (label 32, score 33, and comment 34) and training data assigned to each training image 31 to the computer 10. The first evaluation acquisition module receives this first evaluation and training data and acquires a first evaluation indicating the degree of abnormality or normality for the selected training image.

[0044] Returning to FIG. 3, the continuation of the learning process will be described. Based on the acquired first evaluation, the configuration module constructs a learning image set (step S23). The learning image set is a set including at least one or more learning images to which the first evaluation is given. In other words, it is a set including at least one or more pairs of learning images and the first evaluation. The learning image set can be, for example, a set of learning images that are normal images with a normal label given as the first evaluation, a set of learning images that are abnormal images with an abnormal label given as the first evaluation, or a set of learning images that are normal images with a normal label given as the first evaluation and abnormal images with an abnormal label given as the first evaluation. Based on the acquired first evaluation, the configuration module selects at least one or more learning images to which the same content of the first evaluation (normal (given a normal label, score less than the threshold, positive (no problem, etc.) comment given, no comment given, etc.) is given) or abnormal (given an abnormal label, score greater than or equal to the threshold, negative (abnormal, problem occurred, etc.) comment given, etc.) is given, groups the selected learning images, and constructs them as a learning image set (see FIG. 6). Alternatively, based on the acquired first evaluation, the configuration module selects at least one or more learning images to which different contents of the first evaluation are given (learning images of normal images and learning images of abnormal images), groups the selected learning images, and constructs them as a learning image set. That is, there are patterns of the learning image set including a set of learning images (normal images) evaluated as normal, a set of learning images (abnormal images) evaluated as abnormal, and a set of learning images (normal images) evaluated as normal and learning images (abnormal images) evaluated as abnormal. Regarding the number of learning images selected by the configuration module, it can be a preset number, a number set by the user, or determined by other methods.

[0045] Referring to Figure 6, the state in which the computer 10 constitutes a training image set will be explained. This figure schematically shows the state in which the computer 10 has constituted a training image set. The configuration module selects at least one training image to constitute a training image set from at least one training image to which a first evaluation has been assigned. In this figure, the configuration module has selected two training images 41b and 41d from the four training images 41 (41a to 41d), which are normal images that have been assigned normal labels 42 (42a, 42b, 42c). The training image 41c that has not been selected is an abnormal image that has been assigned an abnormal label 42 (42c). A check mark 43 is shown for the selected training images 41b and 41d to indicate that they have been selected. The configuration module combines these training images 41b and 41d to constitute a training image set 44 of normal images.

[0046] Returning to Figure 3, the learning process will be explained further. The setting module sets an anomaly detection area for one or more of the selected first inspection images (step S24). The anomaly detection area is the area in the image where an anomaly is to be detected in the inspection object. The anomaly detection area may be set on a pixel-by-pixel basis (set in raster format), set using the vertex coordinates of a polygon (set in vector format), or by any other method. In addition, the anomaly detection area may be any area within the first inspection image, or it may be the entire area of ​​the first inspection image. Furthermore, the anomaly detection area may be set in only one location, or in multiple locations (two, three, etc.). The setting module may set an area that has been previously set in the first inspection image as the anomaly detection area, or it may set an area specified by the user as the anomaly detection area. The case where the setting module sets an area specified by the user as the anomaly detection area will be explained below. The computer 10 transmits one or more of the selected first inspection images to the information terminal 3. Information terminal 3 receives the first inspection image and displays it on its display unit via a predetermined UI. Information terminal 3 receives input from the user for an area to be designated as an anomaly detection area for the displayed first inspection image. Information terminal 3 receives input from the user (such as the coordinates of polygon vertices) and accepts input for the anomaly detection area for the displayed first inspection image. Information terminal 3 transmits the received anomaly detection area to computer 10. At this time, information terminal 3 also transmits anomaly detection data (image data, etc.) related to the first inspection image for which the anomaly detection area input was received to computer 10. Computer 10 receives this anomaly detection area and anomaly detection data and acquires the anomaly detection area to be set for the first inspection image. Based on the acquired anomaly detection area and anomaly detection data, the setting module sets anomaly detection area for one or more of the selected first inspection images (see Figure 7). The setting module may also set the anomaly detection area using AI.

[0047] Referring to FIG. 7, the state in which the computer 10 sets an abnormal detection area will be described. This figure is a diagram schematically showing the state in which the computer 10 has set an abnormal detection area. The setting module sets the corresponding area of the first inspection image 50 as the abnormal detection area 52 based on the abnormal detection area input by the user. In this figure, the setting module sets the area surrounded by the dotted line as the abnormal detection area 52 for the inspection target 51. In this case, the information terminal 3 receives the input of the vertices of the polygon along the inspection target 51, and accepts the input with the area surrounded by the straight lines connecting the received vertices as the abnormal detection area 52.

[0048] Returning to FIG. 3, the continuation of the learning process will be described. The learning module learns using the constructed learning image set (step S25). Examples of the learning method executed by the learning module include machine learning such as unsupervised learning, supervised learning, and reinforcement learning, and deep learning using convolutional neural networks, recurrent neural networks, long short-term memory, etc. In the present embodiment, machine learning will be described as an example. When the learning module executes unsupervised learning, it learns the features of normal images using only the learning images given as normal. When the learning module executes supervised learning, it learns to estimate a score (abnormal score, normal score) indicating the degree of abnormality or normality from the image. Also, when the learning module executes supervised learning, it learns to estimate the score obtained by processing the given comment by sentiment analysis or the like from the image. Further, when the learning module executes reinforcement learning, it learns a policy for assigning the degree of abnormality or normality to the images sequentially obtained. Note that the learning method executed by the learning module is merely an example, and the learning module may learn using the learning image set by a learning method other than the above-described learning methods.

[0049] The model creation module creates a trained model based on the learning (step S26). The model creation module creates a trained model based on the learning results using a predetermined algorithm (logistic regression, k-nearest neighbors, SVM (Support Vector Machines), simple Bayesian classifier, etc.).

[0050] The above describes the learning process. The inspection image retraining system 1 uses the trained model created through the learning process to execute the processes described later.

[0051] [Reset process performed by the inspection image relearning system 1] Based on Figure 8, the reset process performed by the inspection image relearning system 1 will be explained. The same figure is a flowchart of the reset process performed by the computer 10. The reset process is performed when camera 4 is a fixed-point camera and when there is a change in the fixed position of camera 4. Detailed explanations of processes similar to those described above will be omitted.

[0052] The change detection module determines whether or not there has been a change in the position of camera 4 (step S30). The change detection module determines whether or not there has been a change in the position of camera 4, such as whether the user has changed the position of camera 4 or whether the position of camera 4 has changed due to some external factor. For example, when the user changes the position of camera 4, the change detection module obtains a notification to that effect from the information terminal 3 and performs this determination based on whether or not this notification has been obtained. Alternatively, the change detection module compares the first inspection image obtained from camera 4 with a first inspection image obtained from camera 4 in the past and performs this determination based on whether or not the field of view is the same. Note that the method by which the change detection module determines a change in the position of camera 4 is not limited to the example described above and can be designed as appropriate. If the change detection module determines that there has been no change in the position of camera 4 (step S30 NO), the computer 10 terminates the reset process.

[0053] On the other hand, if the change detection module determines that there has been a change in the position of camera 4 (step S30 YES), the first inspection image acquisition module reacquires the first inspection image taken again by camera 4 with the inspection target re-fixed in the changed position (step S31). With camera 4 re-fixed in the changed position, it photographs the inspection target using the same process as in step S20 and retransmits the first inspection image to the computer 10. The first inspection image acquisition module receives this first inspection image again and reacquires the first inspection image taken again by camera 4 with the inspection target re-fixed in the changed position.

[0054] The selection module re-selects one or more training images to be used for training from the re-acquired first inspection images (step S32). The selection module uses the re-acquired first inspection images to perform the same processing as in step S21, and re-selects one or more training images to be used for training from the re-acquired first inspection images.

[0055] The first evaluation acquisition module reacquires a first evaluation indicating the degree of abnormality or normality for the re-selected training image (step S33). The first evaluation acquisition module uses the re-selected training image to perform the same processing as in step S22 and reacquires a first evaluation indicating the degree of abnormality or normality for the re-selected training image.

[0056] The configuration module reconstructs the training image set based on the reacquired first evaluation (step S34). The configuration module uses the reacquired first evaluation to perform the same processing as in step S23 and reconstructs the training image set based on the reacquired first evaluation.

[0057] The setting module resets the anomaly detection area for one or more of the re-selected first inspection images (step S35). The setting module uses the re-selected first inspection images to perform the same processing as in step S24, and resets the anomaly detection area for one or more of the re-selected first inspection images.

[0058] The learning module retrains itself using the reconstructed training image set (step S36). The learning module performs the same processing as in step S25 using the reconstructed training image set and retrains itself using the reconstructed training image set.

[0059] The model creation module recreates the trained model based on the retraining (step S37). The model creation module performs the same process as in step S26 based on the retraining and recreates the trained model based on the retraining.

[0060] The above describes the reset process. If a change occurs in the position of camera 4, the inspection image relearning system 1 will execute the process using the trained model recreated by the reset process.

[0061] [First Provisioning Process Executed by the Inspection Image Retraining System 1] The first provisioning process executed by the inspection image retraining system 1 will be explained based on Figure 9. The same figure is a flowchart of the first provisioning process executed by the computer 10. The first provisioning process is a detailed description of the second inspection image acquisition process (step S8) in which a second inspection image is acquired by the camera 4 of the object to be inspected, an estimation process (step S9) in which an abnormal score or normal score is estimated for the acquired second inspection image using the created trained model, and a provisioning process (step S10) in which abnormality-related information within the set abnormality detection area is provided based on the estimated abnormal score or normal score. The first provisioning process is the process when the camera 4 uses a second inspection image in which the object to be inspected is photographed at the same angle of view as the first inspection image. Detailed explanations of processes similar to those described above will be omitted.

[0062] The second inspection image acquisition module acquires a second inspection image of the inspection target captured by camera 4 (step S40). The second inspection image is a still image or video captured by camera 4, similar to the first inspection image. The second inspection image in the first provision process is an image of the inspection target captured at the same field of view as the first inspection image. The second inspection image may be a newly captured image by camera 4, or an image captured in the past. The second inspection image may be the same image as an image included in the training image set, or a different image. Camera 4 may capture the inspection target using the same process as in step S20 and transmit the second inspection image to computer 10, or it may transmit a second inspection image that was captured in the past using the same process as in step S20. The second inspection image acquisition module receives this second inspection image and acquires the second inspection image of the inspection target captured by camera 4. Computer 10 may also acquire the training images that make up the training image set as the second inspection image.

[0063] The estimation module uses the created trained model to estimate an abnormality score or a normal score for the acquired second inspection image (step S41). The abnormality score is a score indicating the degree of abnormality in the second inspection image, and is a score where the score of the abnormality detection area set by the processing in step S24 is equal to or greater than the threshold. The abnormality score can be expressed as a score, percentage, rank, etc. The normal score is a score indicating the degree of normality in the second inspection image, and is a score where the score of the abnormality detection area set by the processing in step S24 is less than the threshold. The normal score can be expressed as a score, percentage, rank, etc. The estimation module performs image analysis on the acquired second inspection image using the trained model and the abnormality detection area, and estimates whether the inspection object shown in the second inspection image is abnormal or normal based on the degree of agreement, similarity, etc. The estimation module estimates the degree of agreement or similarity between the object being inspected in the second inspection image and the object being inspected in the training image evaluated as abnormal by the trained model, based on the object being inspected in the second inspection image, the trained model, and the anomaly detection region. Alternatively, the estimation module estimates the degree of agreement or similarity between the object being inspected in the second inspection image and the object being inspected in the training image evaluated as normal by the trained model, based on the object being inspected in the second inspection image, the trained model, and the anomaly detection region. Based on this estimation result, the estimation module estimates whether the second inspection image is abnormal or normal. The estimation module estimates the score of the second inspection image estimated to be abnormal as the abnormal score. Alternatively, the estimation module estimates the score of the second inspection image estimated to be normal as the normal score. The estimation module estimates an abnormality score if the first evaluation is performed using a pre-trained model with a training image set in which the first evaluation is abnormal, and estimates a normal score if the first evaluation is performed using a pre-trained model with a training image set in which the first evaluation is normal (see Figure 10). The estimation of abnormality or normal scores performed by the estimation module may be performed on a pixel-by-pixel basis of the second check image, or on an image-by-image basis.When performed on a pixel-by-pixel basis, the estimation module estimates an abnormal score or normal score for each pixel. When performed on an image-by-image basis, the estimation module estimates an abnormal score or normal score for each image. Furthermore, when performed on an image-by-image basis, the estimation module may treat the highest of the abnormal scores or normal scores estimated for each pixel as the abnormal score or normal score for the image-by-image portion of the second inspection image, or it may treat the average value as the abnormal score or normal score for the image-by-image portion of the second inspection image, or it may treat an abnormal score or normal score estimated by any other method as the abnormal score or normal score for the image-by-image portion.

[0064] Referring to Figure 10, the state in which the computer 10 estimates an abnormal score or a normal score will be explained. The figure schematically shows the state in which the computer 10 has estimated an abnormal score. The estimation module performs image analysis on the acquired second inspection image 60 using a trained model and an abnormality detection region 61 set in the second inspection image 60. As a result, in the figure, the estimation module estimates that a foreign object 62 is included in the abnormality detection region 61 in the second inspection image 60 and estimates that some kind of abnormality has occurred. The estimation module uses the trained model to estimate a score for the estimated abnormality as the abnormal score. For example, in the figure, a score of 78.5 is estimated as the abnormality score 63 and is assigned to the second inspection image 60. In addition, when the estimation module estimates an abnormality, it assigns an abnormality label 64 to the second inspection image 60 to indicate that an abnormality has occurred in the second inspection image 60. Furthermore, when the estimation module estimates an anomaly, it adds an anomaly comment 65 indicating the nature of the anomaly to the second inspection image 60.

[0065] Returning to Figure 9, the continuation of the first provision process will be explained. The provision module provides anomaly-related information within the set anomaly detection area based on the estimated anomaly score or normal score (step S42). The anomaly-related information is information regarding the degree of anomaly or normality in the anomaly detection area in the second inspection image, and includes, for example, anomaly scores (maximum value, minimum value, average value, percentile value, etc.) or normal scores (maximum value, minimum value, average value, percentile value, etc.) on a pixel-by-pixel or image-by-image basis, anomaly labels or normal labels on a pixel-by-pixel or image-by-image basis based on the anomaly score or normal score, a heatmap on a pixel-by-pixel or image-by-image basis based on the anomaly score or normal score on a pixel-by-pixel basis, a grayscale image on a pixel-by-pixel or image-by-image basis obtained by thresholding the anomaly score or normal score on a pixel-by-pixel basis, and a graph based on the time-series change of the anomaly score or normal score on a pixel-by-pixel or image-by-image basis.

[0066] Each case will be explained. First, the case in which the providing module provides the abnormal score or normal score of the abnormality detection area as abnormality-related information will be explained. The providing module transmits an image of the second inspection image with the estimated abnormal score or normal score attached to it as abnormality-related information to the information terminal 3. This abnormal score or normal score may be the maximum value, minimum value, average value, percentile value, etc. for each score. The information terminal 3 receives this abnormality-related information and displays the image with the abnormal score or normal score attached to it on its display unit via a predetermined UI. Furthermore, if the abnormal score exceeds the threshold, the providing module may, in addition to providing the abnormal score itself, also provide an image of the second inspection image with a label, comment, etc. indicating that it is abnormal attached to it as abnormality-related information. Alternatively, if the normal score does not exceed the threshold, the providing module may, in addition to providing the normal score itself, also provide an image of the second inspection image with a label, comment, etc. indicating that it is normal attached to it as abnormality-related information.

[0067] Next, we will explain the case where the providing module provides an abnormality label or normal label for the abnormality detection area as abnormality-related information. The providing module transmits an image of the second inspection image with an abnormality label assigned as the estimated abnormal state or a normal label assigned as the estimated normal state to the information terminal 3 as abnormality-related information. This abnormality label or normal label is based on the abnormality score or normal score. The assigning module assigns an abnormality label to the second inspection image if the abnormality score is above a threshold, assigns a normal label if the abnormality score is below the threshold, assigns a normal label if the normal score is above the threshold, and assigns an abnormal score if the normal score is below the threshold. The providing module transmits an image of the second inspection image with an abnormality label or normal label assigned based on the estimated abnormality score or normal score as abnormality-related information to the information terminal 3. The information terminal 3 receives this abnormality-related information and displays the image with the abnormality label or normal label assigned on its display unit via a predetermined UI.

[0068] Next, we will describe the case where the providing module provides a grayscale image of the anomaly detection area as anomaly-related information. The providing module transmits to the information terminal 3 an image converted from the second inspection image to a grayscale image on a pixel-by-pixel or image-by-image basis, after thresholding the pixel-by-pixel anomaly score or normal score. This grayscale image is based on the anomaly score or normal score, and the grayscale image creation module performs thresholding on the pixel-by-pixel anomaly score or normal score, converting the anomaly detection area to grayscale on a pixel-by-pixel or image-by-image basis. The providing module transmits the grayscale image converted to grayscale based on the estimated anomaly score or normal score to the information terminal 3 as anomaly-related information for the second inspection image. The information terminal 3 receives this anomaly-related information and displays the grayscale image on its display unit via a predetermined UI.

[0069] Next, we will explain the case where the providing module provides a heatmap of the anomaly detection area as anomaly-related information. The providing module transmits a heatmap created based on the pixel-level or image-level anomaly score or normal score for the second inspection image to the information terminal 3 as anomaly-related information. This heatmap is based on the anomaly score or normal score, and the map creation module creates a heatmap corresponding to the anomaly detection area using a predetermined color scale for the pixel-level anomaly score or normal score. The map creation module can also be configured to overlay the created heatmap with the second inspection image. Here, even if the created heatmap and the second inspection image have different resolutions, the map creation module does not match their resolutions, but overlays the heatmap and the second inspection image while maintaining their individual resolutions. For example, if the heatmap is low resolution and the second inspection image is high resolution, the map creation module will overlay the low resolution heatmap onto the high resolution second inspection image. As a result, anomalies can be confirmed in the high resolution second inspection image while keeping the training cost of the model that outputs the heatmap low. The providing module transmits a heatmap based on the estimated abnormality score or normality score to the second inspection image as abnormality-related information to the information terminal 3. The information terminal 3 receives this abnormality-related information and displays the heatmap on its display unit via a predetermined UI. When the heatmap is superimposed on the second inspection image, the information terminal 3 displays the heatmap with the second inspection image superimposed on it on its display unit via a predetermined UI.

[0070] Finally, we will explain the case where the providing module provides a graph based on the time-series changes of the anomaly detection area as anomaly-related information. The providing module transmits a graph created based on the anomaly score or normal score at the pixel level or image level for the second inspection images in a time-series sequence to the information terminal 3 as anomaly-related information. This graph is based on the anomaly score or normal score of each of the multiple second inspection images in a time-series sequence, and the graph creation module creates a graph (bar graph, line graph, pie chart, etc.) that summarizes the anomaly score or normal score of each second inspection image in a time-series sequence. The providing module transmits a graph based on the time-series changes of the estimated anomaly score or normal score for the second inspection images to the computer 10 as anomaly-related information. The information terminal 3 receives this anomaly-related information and displays the graph on its display unit via a predetermined UI. By providing a graph based on the time-series changes of the anomaly detection area, the providing module makes it easier for users to understand the time periods and times when anomalies are likely to occur.

[0071] The providing module provides this anomaly-related information by displaying it on the information terminal 3. The providing module may also provide this anomaly-related information individually or in combination with other information (see Figure 11).

[0072] Referring to Figure 11, the state in which the computer 10 provides anomaly-related information will be explained. This figure schematically shows the screen displayed by the information terminal 3 when the computer 10 provides anomaly-related information. In this figure, the providing module provides an anomaly label, an anomaly score, and a heatmap as anomaly-related information. The information terminal 3 displays the anomaly-related information provided by the providing module on its display unit via the UI 70. The information terminal 3 displays an image 72 in which an anomaly detection area 71 is set on the second inspection image, an image 74 in which the created heatmap 73 is overlaid on the second inspection image, an image 75 of the created heatmap 73, an image 76 in which the created heatmap 73 is overlaid on the set anomaly detection area 71, an anomaly label 77, and an anomaly score 78. The information terminal 3 can be configured to display multiple anomaly-related information on its display unit in this manner, or it can be configured to display only one anomaly-related information on its display unit.

[0073] The above is the first provision process.

[0074] [Second Provisioning Process Executed by Inspection Image Retraining System 1] The second provisioning process executed by the inspection image retraining system 1 will be explained based on Figure 12. This figure is a flowchart of the second provisioning process executed by the computer 10. The second provisioning process is that the second inspection image is a photograph of the inspection target taken at a different angle of view than the first inspection image. Detailed explanations of processes similar to those described above will be omitted.

[0075] The second inspection image acquisition module acquires a second inspection image of the object to be inspected, captured by camera 4 (step S50). The process in step S50 is the same as the process in step S40.

[0076] The change detection module determines whether or not there is a change in the position of camera 4 (step S51). The process in step S51 is the same as the process in step S30. If the change detection module determines that there is no change in the position of camera 4 (step S51 NO), the computer 10 terminates the second provisioning process and executes the first provisioning process.

[0077] On the other hand, if the change detection module determines that there is a change in the position of camera 4 (step S51 YES), the correction module corrects the second inspection image to the same field of view as the first inspection image using field of view correction (step S52). The correction module identifies the field of view of the first inspection image and corrects the field of view of the second inspection image to the identified field of view using an existing correction method.

[0078] The estimation module uses the created trained model to estimate an abnormal score or a normal score for the second inspection image after field of view correction (step S53). The estimation module uses the second inspection image after field of view correction to perform the same processing as in step S41, and uses the created trained model to estimate an abnormal score or a normal score for the second inspection image after field of view correction.

[0079] The provided module provides anomaly-related information within the set anomaly detection area based on the estimated anomaly score or normal score (step S54). The process in step S54 is the same as the process in step S42.

[0080] The above describes the second provision process. By correcting the field of view of the second inspection image, the inspection image retraining system 1 can continue operation without initializing the trained model and repeating the training process.

[0081] The inspection image relearning system 1 can also be configured to alert the user if the estimated abnormal score or normal score in the first and second provision processes exceeds a predetermined threshold a predetermined number of times, or if it exceeds the threshold a predetermined number of times within a predetermined time period. Each case will be explained below.

[0082] First, we will explain the case where the estimated abnormal score or normal score exceeds a predetermined threshold after a predetermined number of times. In this case, for example, we focus only on the number of times the abnormal score or normal score exceeds the threshold. The count determination module determines whether the estimated abnormal score or normal score has exceeded a predetermined threshold after a predetermined number of times. If the count determination module determines that it has not exceeded the threshold, the computer 10 does not issue an alert to the user. On the other hand, if the count determination module determines that it has exceeded the threshold, the count alert module alerts the user. The methods by which the count alert module alerts the user include, for example, outputting a message to the information terminal 3 indicating whether the abnormal score or normal score has exceeded the threshold or has not exceeded the threshold, or outputting a second inspection image with labels or comments indicating the abnormal or normal score. The count alert module transmits the message and the second inspection image to the information terminal 3, and the information terminal 3 receives the message and the second inspection image and displays them on its display unit via a predetermined UI. Furthermore, the method by which the count alert module alerts the user is not limited to the examples described above, and can be designed as appropriate.

[0083] Next, we will explain the case where the estimated abnormal score or normal score exceeds a predetermined threshold a predetermined number of times within a predetermined time period. In this case, for example, we focus on the frequency with which the abnormal score or normal score exceeds the threshold. The frequency determination module determines whether the estimated abnormal score or normal score has exceeded a predetermined threshold a predetermined number of times within a predetermined time period. If the frequency determination module determines that it has not exceeded the threshold, the computer 10 does not issue an alert to the user. On the other hand, if the frequency determination module determines that it has exceeded the threshold, the frequency alert module alerts the user. The method by which the frequency alert module alerts the user can be the same as the method by which the count alert module alerts the user as described above. Note that the method by which the frequency alert module alerts the user is not limited to the example described above and can be designed as appropriate.

[0084] [Retraining Process Executed by Inspection Image Retraining System 1] Based on Figure 13, the retraining process executed by the inspection image retraining system 1 will be explained. The same figure is a flowchart of the retraining process executed by the computer 10. The retraining process consists of a second evaluation acquisition process (step S11) which acquires a second evaluation indicating the degree of abnormality or normality for the provided abnormality-related information, a reconstruction process (step S12) which reconstructs the training image set reflecting the acquired second evaluation, and a retraining process (step S13) which retrains using the reconstructed training image set. Detailed explanations of processes similar to those described above will be omitted.

[0085] The second evaluation acquisition module acquires a second evaluation indicating the degree of abnormality or normality for the provided abnormality-related information (step S60). The second evaluation may be a label (abnormal, normal, etc.), similar to the first evaluation, or it may be a numerical representation of each degree (abnormal score (score above a threshold), normal score (score below a threshold), etc.), or it may be a comment regarding each degree (abnormality occurring, normal, no problem, etc.), or it may be something else. This second evaluation may be for an abnormality-related information unit, for a second inspection image unit to which abnormality-related information is attached, or for any pixel unit in this second inspection image, or it may be a combination of both, or it may be something else. The information terminal 3 receives input from the user of the second evaluation indicating the degree of abnormality or normality for the abnormality-related information provided by the processing in step S54 (see Figure 14). Information terminal 3 receives input from the user (such as selection or direct input of labels, scores, and comments) and assigns a second evaluation, such as a label, score, or comment, to the provided anomaly-related information. Information terminal 3 transmits the assigned second evaluation to computer 10. At this time, information terminal 3 also transmits evaluation data (such as image data) related to the anomaly-related information to computer 10. The second evaluation acquisition module receives this second evaluation and evaluation data and acquires a second evaluation indicating the degree of anomaly or normality for the provided anomaly-related information. Note that the second evaluation acquisition module may acquire the second evaluation using AI instead of acquiring the second evaluation entered by the user.

[0086] Referring to Figure 14, the state in which the information terminal 3 accepts the second evaluation will be explained. The figure schematically shows the state in which the information terminal 3 has accepted the second evaluation. The information terminal 3 displays the heat map 81 provided as anomaly-related information via the UI 80. The information terminal 3 accepts input from the user, such as the assignment of labels 82, scores 83, and comments 84 (selection input, direct input, etc.). The information terminal 3 assigns the content received from the user to the heat map 81 as labels 82, scores 83, and comments 84, respectively, and displays the assigned content. The heat map 81 is in a state where an anomaly has been assigned as label 82, a score of 78.5 as score 83, and a comment 84 that says "foreign object is present." The information terminal 3 similarly accepts the second evaluation for each of the other anomaly-related pieces of information and assigns the accepted second evaluation to each piece of anomaly-related information. The method by which the information terminal 3 assigns the second evaluation is not limited to the method shown in the diagram; other methods (such as assigning icons or changing colors) may also be used. Furthermore, the content of the second evaluation may also be other than labels, scores, or comments. The information terminal 3 transmits the second evaluation (labels, scores, and comments, etc.) and evaluation data assigned to each anomaly-related piece of information to the computer 10. The second evaluation acquisition module receives this second evaluation and evaluation data and acquires a second evaluation indicating the degree of anomaly or normality for the provided anomaly-related information.

[0087] Returning to Figure 13, let's continue explaining the learning process. The reconstruction module reconstructs the training image set, reflecting the acquired second evaluation (step S61). The reconstruction module reflects the acquired second evaluation in the training image set configured by the process in step S23. For example, if the score estimated by the process in step S41 differs from the acquired second evaluation score by a predetermined value or more, the reconstruction module adds the second inspection image to the training image set, which is associated with the anomaly information that received this second evaluation, based on the acquired second evaluation score. In this case, if the evaluation based on the estimated score was normal, but the evaluation based on the second evaluation score is abnormal, the reconstruction module adds the second inspection image with this second evaluation as an abnormal image to the training image set. Alternatively, the reconstruction module removes the second inspection image with this second evaluation from the training image set of normal images. Furthermore, if the reconstruction module determines that an image is abnormal based on its estimated score, but normal based on the score of the second evaluation, it adds the second-inspection image to the training image set as a normal image. Alternatively, the reconstruction module removes the second-inspection image with the second evaluation from the training image set of abnormal images. Based on the evaluation obtained using the second evaluation, the reconstruction module deletes or adds training images to the training image set and reconstructs the training image set to reflect the obtained second evaluation.

[0088] The retraining module retrains using the reconstructed training image set (step S62). The retraining module performs the same processing as in step S25 using the reconstructed training image set and retrains using the reconstructed training image set.

[0089] The model recreation module recreates the trained model based on the retraining (step S62). The model recreation module performs the same process as in step S26 based on the retraining and recreates the trained model based on the retraining.

[0090] The above describes the retraining process. The inspection image retraining system 1 can use the trained model recreated as a result of the retraining process to estimate the abnormal score or normal score of inspection images in subsequent inspections. The computer 10 may also reset the abnormality detection area based on the acquired second evaluation. In this case, the computer 10 may perform the same process as in step S24, or it may use AI to reset the abnormality detection area.

[0091] The means and functions described above are realized by a computer (including the CPU, information processing unit, and various terminals) reading and executing a predetermined program. The program may be provided, for example, via a network from the computer (SaaS: Software as a Service) or as a cloud service. Alternatively, the program may be provided in a form recorded on a computer-readable recording medium. In this case, the computer reads the program from the recording medium, transfers it to an internal or external recording device, records it, and executes it. Alternatively, the program may be pre-recorded on a recording device (recording medium) and provided to the computer from that recording device via a communication line.

[0092] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments of the present invention are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments of the present invention.

[0093] A first aspect disclosed in this embodiment is an inspection image retraining system for remote inspection using inspection images of an inspection target, comprising: a first inspection image acquisition unit that acquires first inspection images of the inspection target taken with a camera; a selection unit that selects one or more training images from the acquired first inspection images; a first evaluation acquisition unit that acquires a first evaluation indicating the degree of abnormality or normality for the selected training images; a configuration unit that constructs a training image set based on the acquired first evaluation; a setting unit that sets an abnormality detection area for one or more of the selected first inspection images; a learning unit that learns using the configured training image set; a model creation unit that creates a trained model based on the learning; a second inspection image acquisition unit that acquires second inspection images of the inspection target taken with a camera; an estimation unit that estimates an abnormality score or a normal score for the acquired second inspection image using the created trained model; and a provision unit that provides abnormality-related information within the set abnormality detection area based on the estimated abnormality score or normal score. The present invention provides an inspection image relearning system comprising: a second evaluation acquisition unit that acquires a second evaluation indicating the degree of abnormality or normality for the provided abnormality-related information; a reconstruction unit that reconstructs the learning image set reflecting the acquired second evaluation; and a relearning unit that relearns using the reconstructed learning image set.

[0094] A second aspect disclosed in this embodiment provides the inspection image retraining system described in the first aspect, wherein the providing unit provides the abnormal score or normal score of the abnormality detection area as abnormality-related information.

[0095] A third aspect disclosed in this embodiment provides an inspection image retraining system according to the first aspect, wherein the providing unit provides that an abnormality occurs when the abnormal score or normal score exceeds a threshold, and provides that a normal state occurs when the abnormal score or normal score does not exceed a threshold.

[0096] A fourth aspect disclosed in this embodiment provides an inspection image relearning system according to the first aspect, further comprising a graph creation unit that creates a graph based on the time-series changes of the abnormality-related information, wherein the providing unit provides the created graph.

[0097] A fifth aspect disclosed in this embodiment provides an inspection image retraining system according to the first embodiment, further comprising a map creation unit that creates a heat map based on the abnormality-related information, wherein the providing unit provides the created heat map.

[0098] A sixth aspect disclosed in this embodiment provides an inspection image retraining system according to the fifth aspect, wherein the providing unit provides the heat map and the second inspection image superimposed.

[0099] A seventh aspect disclosed in this embodiment provides an inspection image retraining system according to the sixth aspect, wherein, if the resolution of the heat map and the second inspection image are different, the providing unit provides the heat map and the second inspection image superimposed on each other without matching the resolution.

[0100] An eighth aspect disclosed in this embodiment provides an inspection image retraining system according to the first aspect, wherein the first inspection image acquisition unit re-acquires a first inspection image re-captured with the camera after the camera has been re-fixed to the position of the inspection target when the camera position changes; the selection unit re-selects one or more training images to be retrained from the re-acquired first inspection images; the first evaluation acquisition unit re-acquires the first evaluation for the re-acquired training images; the configuration unit reconstructs the training image set based on the re-acquired first evaluation; the setting unit re-sets the anomaly detection area for one or more of the re-acquired first inspection images; the learning unit retrains using the reconstructed training image set; and the model creation unit re-creates the trained model based on the retraining.

[0101] A ninth aspect disclosed in this embodiment provides an inspection image retraining system according to the first embodiment, further comprising a correction unit that, when the position of the camera changes, corrects the second inspection image to the same field of view as the first inspection image using field of view correction, and the estimation unit estimates the abnormal score or normal score for the second inspection image after field of view correction.

[0102] A tenth aspect disclosed in this embodiment provides an inspection image retraining system according to the first embodiment, further comprising a count alert unit that alerts when the estimated abnormal score or normal score exceeds a set predetermined threshold a predetermined number of times.

[0103] An eleventh aspect disclosed in this embodiment provides an inspection image retraining system according to the first embodiment, further comprising a frequency alert unit that alerts when the estimated abnormal score or normal score exceeds a set predetermined threshold a predetermined number of times within a predetermined time period.

[0104] 1. Inspection Image Retraining System 3. Information Terminal 4. Camera 8. Network 10. Computer 20. UI 21. First Inspection Image 22. User 23. Checkmark 30. UI 31. Training Image 32. Label 33. Score 34. Comment 41. Training Image 42. Label 43. Checkmark 50. First Inspection Image 51. Inspection Target 52. Anomaly Detection Area 60. Second Inspection Image 61. Anomaly Detection Area 62. Foreign Object 63. Anomaly Score 64. Anomaly Label 65. Anomaly Comment 70. UI 71. Anomaly Detection Area 72. Image 73. Heatmap 74. Image 75. Image 76. Image 77. Anomaly Label 78. Anomaly Score

Claims

1. An inspection image retraining system for remote inspection using inspection images of an inspection target, comprising: a first inspection image acquisition unit that acquires first inspection images of the inspection target taken with a camera; a selection unit that selects one or more training images from the acquired first inspection images; a first evaluation acquisition unit that acquires a first evaluation indicating the degree of abnormality or normality for the selected training images; a configuration unit that constructs a training image set based on the acquired first evaluation; a setting unit that sets an abnormality detection area for one or more of the selected first inspection images; a learning unit that learns using the configured training image set; a model creation unit that creates a trained model based on the learning; a second inspection image acquisition unit that acquires second inspection images of the inspection target taken with a camera; an estimation unit that estimates an abnormality score or a normal score for the acquired second inspection image using the created trained model; and a provision unit that provides abnormality-related information within the set abnormality detection area based on the estimated abnormality score or normal score. An inspection image relearning system comprising: a second evaluation acquisition unit that acquires a second evaluation indicating the degree of abnormality or normality for the provided abnormality-related information; a reconstruction unit that reconstructs the learning image set reflecting the acquired second evaluation; and a relearning unit that relearns using the reconstructed learning image set.

2. The inspection image retraining system according to claim 1, wherein the providing unit provides the abnormal score or normal score of the abnormality detection area as abnormality-related information.

3. The inspection image retraining system according to claim 1, wherein the providing unit provides that the abnormal score or normal score is abnormal if it exceeds a threshold, and provides that the abnormal score or normal score is normal if it does not exceed a threshold.

4. The inspection image relearning system according to claim 1, further comprising: a graph creation unit that creates a graph based on the time-series changes of the anomaly-related information, wherein the providing unit provides the created graph.

5. The inspection image relearning system according to claim 1, further comprising: a map creation unit that creates a heat map based on the anomaly-related information, wherein the providing unit provides the created heat map.

6. The inspection image retraining system according to claim 5, wherein the providing unit provides the heat map and the second inspection image superimposed on each other.

7. The inspection image retraining system according to claim 6, wherein, if the resolution of the heat map and the second inspection image are different, the providing unit provides the heat map and the second inspection image superimposed without matching their resolutions.

8. The inspection image retraining system according to claim 1. The first inspection image acquisition unit, when the position of the camera changes, reacquires a first inspection image re-captured with the camera after the inspection target has been re-fixed to the changed position; the selection unit re-selects one or more training images to be retrained from the re-acquired first inspection images; the first evaluation acquisition unit reacquires the first evaluation for the re-acquired training images; the configuration unit reconstructs the training image set based on the re-acquired first evaluation; the setting unit re-sets the anomaly detection area for one or more of the re-acquired first inspection images; the learning unit retrains using the reconstructed training image set; and the model creation unit re-creates the trained model based on the retraining.

9. The inspection image retraining system according to claim 1, further comprising: a correction unit that, when the position of the camera changes, corrects the second inspection image to the same field of view as the first inspection image using field of view correction, and the estimation unit that estimates the abnormal score or normal score for the second inspection image after field of view correction.

10. The inspection image relearning system according to claim 1, further comprising: a count alert unit that alerts when the estimated abnormal score or normal score exceeds a set predetermined threshold a predetermined number of times.

11. The inspection image relearning system according to claim 1, further comprising: a frequency alert unit that alerts when the estimated abnormal score or normal score exceeds a set predetermined threshold a predetermined number of times within a predetermined time period.

12. A method for retraining inspection images performed by a computer that performs remote inspection using inspection images of an object to be inspected, comprising: acquiring a first inspection image taken of the object to be inspected with a camera; selecting one or more training images from the acquired first inspection images; acquiring a first evaluation indicating the degree of abnormality or normality for the selected training images; configuring a training image set based on the acquired first evaluation; setting an anomaly detection region for one or more of the selected first inspection images; training using the configured training image set; creating a trained model based on the training; acquiring a second inspection image taken of the object to be inspected with a camera; estimating an anomaly score or normality score for the acquired second inspection image using the created trained model; providing anomaly-related information within the set anomaly detection region based on the estimated anomaly score or normality score; and acquiring a second evaluation indicating the degree of abnormality or normality for the provided anomaly-related information. A method for retraining inspection images, comprising the steps of: reconstructing the training image set in accordance with the acquired second evaluation; and relearning using the reconstructed training image set.

13. A computer that performs remote inspection using inspection images of an object to be inspected, comprising the steps of: acquiring a first inspection image of the object to be inspected by taking a picture of the object to be inspected with a camera; selecting one or more training images from the acquired first inspection images; acquiring a first evaluation indicating the degree of abnormality or normality for the selected training images; configuring a training image set based on the acquired first evaluation; setting an anomaly detection region for one or more of the selected first inspection images; training using the configured training image set; creating a trained model based on the training; acquiring a second inspection image of the object to be inspected by taking a picture of the object to be inspected with a camera; estimating an anomaly score or normality score for the acquired second inspection image using the created trained model; providing anomaly-related information within the set anomaly detection region based on the estimated anomaly score or normality score; acquiring a second evaluation indicating the degree of abnormality or normality for the provided anomaly-related information; and reconstructing the training image set reflecting the acquired second evaluation. A computer-readable program for performing the step of retraining using the reconstructed set of training images.

Images