Model generation apparatus and welding monitoring apparatus

Abstract

This welding monitoring apparatus comprises: an acquisition unit that acquires, as a training image, an image obtained by capturing a welding spot during welding; a display control unit that causes a display device to display, together with the training image, a user interface for allowing a user to specify a defect occurrence risk indicating the degree of certainty that a defect has occurred in the welding spot shown in the training image; and a training unit that performs machine learning in which the training image is used as an explanatory variable and the defect occurrence risk specified by the user is used as a target variable, thereby generating a determination model that inputs a target image obtained by capturing the welding spot during welding, and outputs the defect occurrence risk of the welding spot shown in the target image.

Description

Model Generation Device and Welding Monitoring Device 【0001】 The present disclosure relates to a model generation device and a welding monitoring device. 【0002】 Techniques for monitoring welding points using a machine learning model are known (for example, Patent Documents 1 to 7). 【0003】 Japanese Unexamined Patent Application Publication No. 2018 - 192524, Japanese Unexamined Patent Application Publication No. 2019 - 195811, Japanese Unexamined Patent Application Publication No. 2021 - 13947, Japanese Unexamined Patent Application Publication No. 2023 - 127337, Japanese Unexamined Patent Application Publication No. 2022 - 105404, Japanese Unexamined Patent Application Publication No. 2020 - 182966, Japanese Unexamined Patent Application Publication No. 2023 - 166121 【0004】 A machine learning model may be constructed using learning images precisely acquired under specific welding environments and imaging conditions. In this case, when monitoring welding, it is necessary to acquire an image to be judged under the same welding environment and imaging conditions as during learning. Therefore, adjustment work such as imaging conditions takes time. 【0005】 The present disclosure describes a model generation device and a welding monitoring device that can reduce the workload for acquiring an image to be judged. 【0006】 A model generation device according to an aspect of the present disclosure includes an acquisition unit that acquires an image of a welding point during welding as a learning image, a display control unit that causes a display device to display, together with the learning image, a user interface for allowing a user to specify a defect occurrence risk indicating the confidence level that a defect has occurred at the welding point shown in the learning image, and a learning unit that generates a determination model that inputs a target image of a welding point during welding and outputs a defect occurrence risk of the welding point shown in the target image by performing machine learning using the learning image as an explanatory variable and the defect occurrence risk specified by the user as a target variable. 【0007】 According to the present disclosure, the workload for acquiring an image to be judged can be reduced. 【0008】Figure 1 is a diagram showing the schematic configuration of a welding system including a welding monitoring device according to one embodiment. Figure 2 is a diagram showing an example of the hardware configuration of the computer constituting the welding monitoring device shown in Figure 1. Figure 3 is a functional block diagram of the welding monitoring device shown in Figure 1. Figure 4 is a flowchart showing an example of a model generation method performed by the welding monitoring device shown in Figure 1. Figure 5 is a diagram showing an example of a user interface. Figure 6 is a flowchart showing an example of a welding monitoring method performed by the welding monitoring device shown in Figure 1. Figure 7 is a diagram for explaining the determination made by the determination unit shown in Figure 3. 【0009】 [1] Outline of Embodiments A model generation device relating to one aspect of the present disclosure includes: an acquisition unit that acquires images of welding locations taken during welding as training images; a display control unit that displays a user interface on a display device together with the training images, allowing the user to specify a defect occurrence risk indicating the degree of confidence that a defect has occurred at the welding location shown in the training image; and a learning unit that generates a judgment model that takes a target image of a welding location taken during welding as input and outputs a defect occurrence risk of the welding location shown in the target image by performing machine learning with the training images as explanatory variables and the defect occurrence risk specified by the user as the objective variable. 【0010】In this model generation device, images of the welding area during welding are used as training images, and a user interface for specifying the defect risk is displayed on the display device along with the training images. Then, a judgment model is generated by performing machine learning with the training images as explanatory variables and the defect risk specified by the user as the objective variable. With this configuration, annotation is performed with a simple operation in which the user specifies the defect risk while referring to the training images. The training images only need to be images that allow the user to judge the defect risk. When using the judgment model generated using the training images to determine the defect risk of the welding area shown in the target image, the target image also only needs to be an image that allows the user to judge the defect risk, similar to the training images. Therefore, the constraints of the welding environment and shooting conditions can be relaxed. As a result, the workload for acquiring the target image can be reduced. 【0011】 The user interface may also include display elements that allow the user to specify whether a training image is an image for which defect risk cannot be determined, or to specify the defect risk if the training image is an image for which defect risk can be determined. In this case, the user performs annotation with a simple task of referring to the training image, specifying whether the training image is an image for which defect risk cannot be determined, and specifying the defect risk if the training image is an image for which defect risk can be determined. Therefore, the workload of annotation can be reduced. 【0012】The learning unit may further generate a classification model that takes a target image as input and outputs a valid flag indicating whether the target image is one in which defect risk cannot be determined or can be determined, by performing machine learning with training images as explanatory variables and a valid flag specified by the user, which indicates whether the image is one in which defect risk cannot be determined or can be determined, as the target variable. In this case, images that are determined to be in which defect risk cannot be determined can be excluded from the defect risk determination target. This makes it possible to improve the accuracy of defect risk determination by the determination model. 【0013】 A welding monitoring device relating to another aspect of this disclosure includes: an acquisition unit that acquires an image of a welding location taken during welding as a target image; a determination unit that determines the defect risk of a welding location shown in a target image using a determination model generated by performing machine learning with training images of the welding location taken during welding as explanatory variables and a defect risk that indicates the degree of confidence that a defect has occurred at the welding location shown in the training image, which is a defect risk specified by the user; and an output unit that outputs information corresponding to the determination result by the determination unit. 【0014】 In this welding monitoring device, images of the welding area during welding are acquired as target images, and a judgment model is used to determine the defect risk of the welding area shown in the target image, and information corresponding to the judgment result is output. The judgment model is generated by performing machine learning with training images of the welding area during welding as explanatory variables and the defect risk specified by the user as the dependent variable. Therefore, the training images only need to be images that allow the user to judge the defect risk. When using the judgment model generated using the training images to determine the defect risk of the welding area shown in the target image, the target image also only needs to be an image that allows the user to judge the defect risk, similar to the training images. Therefore, the constraints of the welding environment and shooting conditions can be relaxed. As a result, the workload for acquiring the target image can be reduced. 【0015】The determination unit may use a classification model generated by machine learning, where the training images are the explanatory variables and the effective flag specified by the user, which indicates whether the training images are images for which defect risk can be determined or not, to determine whether the target image is an image for which defect risk can be determined. If the target image is determined to be an image for which defect risk can be determined, the determination model may be used to determine the defect risk of the welded area shown in the target image. In this case, images determined to be for which defect risk cannot be determined are excluded from the defect risk determination. Therefore, it is possible to improve the accuracy of defect risk determination. 【0016】 The output unit may output the above information as a notification corresponding to the defect risk to the worker performing the welding or the supervisor monitoring the welding. In this case, the worker or supervisor can understand the defect risk and take appropriate action accordingly. 【0017】 The output unit may output a stop instruction to the welding control device, which controls the welding machine, as the above information if the risk of defect occurrence is higher than a preset threshold. With this configuration, welding is stopped when there is a high probability that a defect has occurred at the weld. Therefore, the possibility of welding continuing in the presence of a defect is reduced. As a result, it becomes possible to improve the quality of the weld. 【0018】 [2] Examples of Embodiments Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. 【0019】 First, a welding system including a welding monitoring device according to one embodiment will be described with reference to Figure 1. Figure 1 is a diagram showing the schematic configuration of a welding system including a welding monitoring device according to one embodiment. 【0020】The welding system 1 shown in Figure 1 is a system for assisting welding. The welding system 1 is applicable to, for example, arc welding, laser welding, or electron beam welding. Examples of arc welding include TIG (Tungsten Inert Gas) welding, MIG (Metal Inert Gas) welding, MAG (Metal Active Gas) welding, and carbon dioxide arc welding. In this embodiment, TIG welding is exemplified. The welding system 1 includes a welding machine 2, a welding control device 3, a camera 4, an external device 5, and a welding monitoring device 10. 【0021】 The welding machine 2 is equipment used for performing welding. The welding machine 2 may include a welding torch, a welding cart, a welding robot, and a welding head. The welding torch generates an arc between the electrode and the base material while supplying shielding gas, and melts and joins the base material and welding wire by the heat of the arc. Welding may be performed while a welding robot or welding cart moves the welding torch. An operator (welder or welding technician) may operate the welding torch. The welding machine 2 is controlled by a welding control device 3. 【0022】 Camera 4 photographs the welding area of ​​the base material during welding (hereinafter sometimes simply referred to as the "welding area"). Camera 4 is mounted around the welding work area, for example, with a fixed viewpoint. Camera 4 may also be mounted on a welding cart, welding robot, welding torch, or welding head. In this case, the welding area and the viewpoint of camera 4 are relatively fixed to some extent. Camera 4 may also be mounted on the welding helmet (shading mask) worn by the worker so as to be able to photograph the welding area during welding. In this case, the viewpoint of camera 4 is not fixed, but the welding area is photographed during welding. Camera 4 photographs the welding area, for example, including the tip of the electrode and the tip of the welding wire. The images captured by camera 4 may be videos or still images. Camera 4 transmits the images (image data) to the welding monitoring device 10. 【0023】External device 5 is a device that displays the judgment result from the welding monitoring device 10. External device 5 is, for example, a display device (display). External device 5 may display the judgment result to the operator or to the welding work supervisor. External device 5 is installed in a location appropriate to the destination of the display. 【0024】 The welding monitoring device 10 is a device that monitors welding. Specifically, the welding monitoring device 10 determines the risk of defect occurrence at the weld and provides information according to the determination result. The defect occurrence risk is a value that indicates the degree of confidence that a person with welding knowledge, such as a welder or welding technician, would judge that a defect has occurred at the weld. For example, the higher the defect occurrence risk value, the more likely it is that a person with welding knowledge would judge that a defect has occurred at the weld. The defect occurrence risk can take values ​​from, for example, 0.0 to 1.0. Defects at the weld may include internal defects and surface defects. Examples of internal defects include poor fusion and insufficient penetration. Examples of surface defects include undercuts and overlaps. 【0025】 Next, the welding monitoring device 10 will be described in detail with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of the hardware configuration of the computer that constitutes the welding monitoring device shown in Figure 1. Figure 3 is a functional block diagram of the welding monitoring device shown in Figure 1. 【0026】 As shown in Figure 2, the computer 100 constituting the welding monitoring device 10 includes a processing circuitry 101, memory 102, storage 103, communication interface 104, input device 105, and output device 106 as hardware devices. The computer 100 does not necessarily have to include the input device 105 and the output device 106. 【0027】The processing circuit 101 is also called a processor. Examples of processing circuits 101 include CPUs (central processing units), GPUs (graphics processing units), FPGAs (field programmable gate arrays), ASICs (application-specific integrated circuits), microcontrollers, and DSPs (digital signal processors). The memory 102 is also called main memory. The memory 102 is composed of, for example, RAM (random access memory) and ROM (read-only memory). 【0028】 The storage device 103 is also called an auxiliary storage device. The storage device 103 consists of, for example, a hard disk or flash memory. The communication interface 104 consists of, for example, a network card or a wireless communication module. The input device 105 consists of, for example, a keyboard, a mouse, and a touch panel. The output device 106 consists of, for example, a display and a speaker. 【0029】 The welding monitoring device 10 may be composed of one computer 100 or multiple computers 100. By installing a predetermined computer program (welding monitoring program) on one or more computers 100, one or more computers 100 function as the welding monitoring device 10. When the welding monitoring device 10 is composed of multiple computers 100, the multiple computers 100 are logically connected to each other via a communication network so that they can communicate with one another, thereby functioning as the welding monitoring device 10. 【0030】The welding monitoring program may be provided on a computer-readable, non-transitory recording medium such as a CD-ROM (compact disc read-only memory), DVD-ROM (digital versatile disc read-only memory), or semiconductor memory. The welding monitoring program may also be provided via a communication network as a data signal superimposed on a carrier wave. 【0031】 The welding monitoring device 10 operates in an operating mode set by the user using the welding monitoring device 10. The operating modes include a learning mode and a monitoring mode. The learning mode is an operating mode for training the classification model M1 (see Figure 7) and the judgment model M2 (see Figure 7), which will be described later. The monitoring mode is an operating mode for monitoring welding. As shown in Figure 3, the welding monitoring device 10 includes, as functional elements, an acquisition unit 11, a display control unit 12, a learning unit 13, a judgment unit 14, an output unit 15, an image storage unit 21, a learning data storage unit 22, and a model storage unit 23. The functions (operations) of each functional unit will be described in detail in the model generation method and welding monitoring method described later, so the functions of each functional unit will be briefly described here. 【0032】 The image storage unit 21 stores images captured by the camera 4. The learning data storage unit 22 stores learning data generated by the display control unit 12. The model storage unit 23 stores the classification model M1 and the judgment model M2 generated by the learning unit 13. 【0033】The acquisition unit 11 acquires welding images taken of the welding area during welding. The acquisition unit 11 acquires welding images from images stored in the image storage unit 21. If the images stored in the image storage unit 21 are videos, the acquisition unit 11 acquires images obtained by dividing the video into frames as welding images. For example, the acquisition unit 11 acquires 60 full HD (High Definition) images per second as welding images. When the operating mode of the welding monitoring device 10 is set to learning mode, the acquisition unit 11 outputs the welding images to the display control unit 12 as learning images. When the operating mode of the welding monitoring device 10 is set to monitoring mode, the acquisition unit 11 outputs the welding images to the determination unit 14 as target images. 【0034】 The display control unit 12 displays the user interface AN (see Figure 5) on the display device (display) along with the training image. The user interface AN is a display element that allows the user to specify the valid flag and the defect risk. The valid flag is a value that indicates whether the training image is an image for which the defect risk cannot be determined or an image for which it can be determined. The user who specifies the valid flag and the defect risk is, for example, a person with welding knowledge such as a welder or welding technician. The user may be the same as or different from the operator. 【0035】The learning unit 13 generates a classification model M1 and a judgment model M2. Classification model M1 is a machine learning model that takes a target image as input and outputs a valid flag indicating whether the target image is an image in which defect occurrence risk cannot be determined or is an image in which it can be determined. Judgment model M2 is a machine learning model that takes a target image as input and outputs the defect occurrence risk of the welded area shown in the target image. The learning unit 13 generates classification model M1 by performing machine learning with training images as explanatory variables and a valid flag specified by the user, which indicates whether the training image is an image in which defect occurrence risk cannot be determined or is an image in which it can be determined, as the objective variable. The learning unit 13 generates judgment model M2 by performing machine learning with training images as explanatory variables and a defect occurrence risk specified by the user, which indicates the degree to which the user judges (confidence level) that a defect has occurred in the welded area shown in the training image, as the objective variable. 【0036】 The determination unit 14 determines the risk of defects occurring at the welded areas shown in the target image. Specifically, the determination unit 14 uses the classification model M1 to determine whether the target image is an undeterminable image or a determinable image, and if it is determined that the target image is a determinable image, it uses the determination model M2 to determine the risk of defects occurring at the welded areas shown in the target image. 【0037】 The output unit 15 outputs information corresponding to the determination result by the determination unit 14. The output unit 15 may also output a notification to the worker performing the welding or the supervisor monitoring the welding, according to the defect risk. If the defect risk is higher than a preset threshold, the output unit 15 may output a stop command to the welding control device 3 to stop welding. 【0038】When the operating mode of the welding monitoring device 10 is set to learning mode, the acquisition unit 11, display control unit 12, learning unit 13, image storage unit 21, learning data storage unit 22, and model storage unit 23 function. In this case, the welding monitoring device 10 functions as a model generation device that generates a classification model M1 and a judgment model M2. When the operating mode of the welding monitoring device 10 is set to monitoring mode, the acquisition unit 11, judgment unit 14, output unit 15, image storage unit 21, and model storage unit 23 function. 【0039】 Next, the model generation method performed by the welding monitoring device 10 will be explained with reference to Figures 4 and 5. Figure 4 is a flowchart showing an example of the model generation method performed by the welding monitoring device shown in Figure 1. Figure 5 is a diagram showing an example of a user interface. The series of processes shown in Figure 4 are started, for example, when the operating mode of the welding monitoring device 10 is set to learning mode. Before the model generation method is started, the welding location during welding is photographed in advance by the camera 4, and the captured image is stored in the image storage unit 21. 【0040】 First, the acquisition unit 11 acquires training images (step S11). In step S11, the acquisition unit 11 acquires welding images by dividing the image (video) stored in the image storage unit 21 into frames, and outputs the welding images to the display control unit 12 as training images. The acquisition unit 11 may also assign frame numbers in chronological order to the series of training images obtained by dividing the video into frames. 【0041】 Next, when the display control unit 12 receives the training image from the acquisition unit 11, it displays the user interface AN together with the training image on the display device (step S12). The user interface AN may be displayed on the display device provided by the welding monitoring device 10, or on the display device of a terminal device held by the worker. 【0042】As shown in FIG. 5, the user interface AN is a display element (tool) for attaching a label to the learning image Gl. Attaching a label is also referred to as "annotation". In the present embodiment, one of the labels "undetermined", "normal", "risk", and "high risk" is attached to each learning image Gl. 【0043】 The user interface AN includes a display area R1. The display area R1 is an area for displaying the learning image Gl. To the left of the display area R1, the frame number of the learning image Gl displayed in the display area R1 is displayed. The user interface AN includes a button Bf and a button Bb. The button Bf is a button for advancing the learning image Gl displayed in the display area R1 to the learning image of the next frame. The button Bb is a button for returning the learning image Gl displayed in the display area R1 to the learning image of the previous frame. 【0044】 The user interface AN includes a button Bt0, a button Bt1, a button Bt2, a button Bt3, a button Bsel, and a button Bsv. The button Bt0 is a button for attaching the label "undetermined". The button Bt1 is a button for attaching the label "normal". The button Bt2 is a button for attaching the label "risk". The button Bt3 is a button for attaching the label "high risk". The button Bsel is a button for selecting the target to which the label is to be attached. When the button Bsel is set to "current", a label is attached to the learning image Gl displayed in the display area R1. When the button Bsel is set to "all previous", a label is attached to all of the learning images without labels among the learning image Gl displayed in the display area R1 and the frames before it. The button Bsv is a button for saving the learning image in association with the label attached to the learning image. 【0045】The user interface AN includes a timeline TL. The timeline TL shows the labeling status of a series of training images. The timeline TL is composed of vertical lines corresponding to each training image, arranged horizontally in frame order. Each vertical line is displayed in a color corresponding to the label (or lack thereof) assigned to the training image. A bar indicating the frame position of the training image Gl displayed in the display area R1 is displayed on the timeline TL. The user interface AN further includes video operation buttons. The video operation buttons are for playing the series of training images as a video in the display area R1. 【0046】 The user assigns labels to each training image by operating the user interface AN. The user intuitively assigns a label to the training image Gl while viewing it in the display area R1. For example, if the user determines that the training image Gl is an image in which the risk of defect occurrence cannot be determined, they press button Bt0 to assign the label "Undeterminable" to the training image Gl. If the user determines that the training image Gl is an image in which the risk of defect occurrence can be determined and is in a normal state with no possibility of defects occurring at the welded area, they press button Bt1 to assign the label "Normal" to the training image Gl. 【0047】 If the user determines that the training image Gl is an image in which the risk of defect occurrence can be determined and that there is a possibility of a defect occurring at the weld, they can press button Bt2 to assign the label "Risk Present" to the training image Gl. If the user determines that the training image Gl is an image in which the risk of defect occurrence can be determined and that there is a high probability of a defect occurring at the weld, they can press button Bt3 to assign the label "High Risk" to the training image Gl. Alternatively, the user may assign labels by typing the keys displayed below each button on the keyboard instead of using buttons Bt0 to Bt3. 【0048】By pressing the button Bf (or inputting the "→" key on the keyboard), the user displays the learning image of the next frame in the display area R1 and assigns a label to the learning image. By switching the button Bsel, the user selects whether to assign a label to each learning image of one frame or to assign labels to the learning images of multiple frames collectively. Often, the same label is assigned to the learning images of consecutive frames. Therefore, the user presses the button Bf to check the learning images in order and assigns the same label to the learning images of multiple frames collectively. This reduces the annotation work. 【0049】 Then, when the user finishes assigning labels, the user presses the button Bsv. When the button Bsv is pressed, the display control unit 12 associates each learning image with the label assigned to the learning image and stores it in the learning data storage unit 22 as learning data (step S13). In the learning data, the labels of "unable to determine", "normal", "risk", and "high risk" are expressed by a combination of a valid flag and a defect occurrence risk. For example, when the valid flag is 0, it indicates that the learning image is an image for which it is impossible to determine the defect occurrence risk, and when the valid flag is 1, it indicates that the learning image is an image for which it is possible to determine the defect occurrence risk. 【0050】 The label of "unable to determine" is expressed with a valid flag of 0 and a defect occurrence risk of Null (invalid value). The label of "normal" is expressed with a valid flag of 1 and a defect occurrence risk of 0.0. The label of "risk" is expressed with a valid flag of 1 and a defect occurrence risk of 0.8. The label of "high risk" is expressed with a valid flag of 1 and a defect occurrence risk of 1.0. 【0051】Next, the learning unit 13 generates a classification model M1 and a judgment model M2 (step S14). Specifically, the learning unit 13 reads the learning data from the learning data storage unit 22 and uses it as the correct answer data, and generates the classification model M1 by performing machine learning with the learning images as explanatory variables and the valid flag as the target variable. The learning unit 13 extracts the learning images with the valid flag set to 1 from the learning data and uses them as the correct answer data, and generates the judgment model M2 by performing machine learning with the learning images as explanatory variables and the defect occurrence risk as the target variable. Then, the learning unit 13 stores the classification model M1 and the judgment model M2 in the model storage unit 23. 【0052】 This concludes the series of processes for model generation. 【0053】 Next, the welding monitoring method performed by the welding monitoring device 10 will be described with reference to Figures 6 and 7. Figure 6 is a flowchart showing an example of the welding monitoring method performed by the welding monitoring device shown in Figure 1. Figure 7 is a diagram illustrating the determination made by the determination unit shown in Figure 3. The series of processes shown in Figure 6 are started, for example, when the operating mode of the welding monitoring device 10 is set to monitoring mode. The welding monitoring method may also be performed in real time during welding. In this case, the welding location is photographed by the camera 4 during welding, and the photographed images are sequentially stored in the image storage unit 21. 【0054】 First, the acquisition unit 11 acquires the target image (step S21). In step S21, the acquisition unit 11 acquires welding images by dividing the image (video) stored in the image storage unit 21 into frames, and outputs the welding images as the target image to the determination unit 14. The acquisition unit 11 may assign frame numbers in chronological order to the series of target images obtained by dividing the video into frames. 【0055】Next, the determination unit 14 determines whether the target image is an image for which defect risk can be determined or not (step S22). As shown in Figure 7, the determination unit 14 uses the classification model M1 to determine whether the target image Gt is an image for which defect risk can be determined or not. Specifically, the determination unit 14 inputs the target image Gt to the classification model M1 and obtains a valid flag from the classification model M1. If the target image Gt is an image for which defect risk can be determined, a valid flag with a value of 1 is obtained, and if the target image Gt is an image for which defect risk cannot be determined, a valid flag with a value of 0 is obtained. 【0056】 In step S22, if the target image is determined to be an image in which the risk of defect occurrence can be determined (step S22: YES), the determination unit 14 determines the risk of defect occurrence at the welded area shown in the target image (step S23). As shown in Figure 7, the determination unit 14 uses the determination model M2 to determine the risk of defect occurrence at the welded area shown in the target image Gt. Specifically, the determination unit 14 inputs the target image Gt into the determination model M2 to obtain the risk of defect occurrence from the determination model M2. The risk of defect occurrence output from the determination model M2 can take values ​​from 0.0 to 1.0. The larger the value of the risk of defect occurrence, the higher the probability that a person with welding knowledge, such as a welder or welding technician, will determine that a defect has occurred at the welded area shown in the target image Gt. 【0057】 The determination unit 14 then outputs the valid flag obtained in step S22 and the defect risk obtained in step S23 to the output unit 15. If, in step S22, the target image is determined to be an image for which the defect risk cannot be determined (step S22: NO), the determination unit 14 outputs the valid flag obtained in step S22 to the output unit 15 without performing step S23. 【0058】Next, the output unit 15 outputs the judgment result (step S24). The output unit 15 may output the judgment result to the external device 5. The output unit 15 may output the judgment result to the output device 106 of the welding monitoring device 10. The judgment result uses information generated based on the valid flag and the defect risk, which is recognizable to the user. For example, if the valid flag indicates that the target image is an undeterminable image, the judgment result may use a message (text and audio, etc.) indicating that it is undeterminable. The judgment result may also include a message prompting a change in the shooting conditions of the camera 4. If the number of consecutive undeterminable target images exceeds a predetermined number, the output unit 15 may include a warning message in the judgment result indicating that the number of undeterminable images is increasing, and may output a stop instruction to the welding control device 3 to stop welding. 【0059】 If the valid flag indicates that the target image is a decipherable image, the determination result may use a message corresponding to the defect risk. In this case, messages may be prepared in advance for each range of defect risk values. An example of such a message is, "The welding area has a high defect risk." 【0060】 The output unit 15 may output a stop command to the welding control device 3 to stop welding if the risk of defect occurrence is higher than a preset threshold. When the welding control device 3 receives the stop command, it performs a stop process to stop welding on the welding machine 2. The stop process may include arc stopping, trolley stopping, and crater processing. 【0061】 Next, it is determined whether or not the termination conditions have been met (step S25). Examples of termination conditions include reaching the final frame of the video stored in the image storage unit 21, or the user performing an operation to stop monitoring. If it is determined in step S25 that the termination conditions have not been met (step S25: NO), steps S21 to S25 are performed again. On the other hand, if it is determined in step S25 that the termination conditions have been met (step S25: YES), the series of processes of the welding monitoring method are terminated. 【0062】 As explained above, when the operating mode of the welding monitoring device 10 is set to learning mode, welding images taken of the welding location during welding are acquired as learning images, and a user interface AN for the user to specify the defect risk is displayed on the display device along with the learning images. Then, by performing machine learning with the learning images as explanatory variables and the valid flag specified by the user as the target variable, a classification model M1 is generated. By performing machine learning with the learning images as explanatory variables and the defect risk specified by the user as the target variable, a judgment model M2 is generated. 【0063】 When the operating mode of the welding monitoring device 10 is set to monitoring mode, a welding image taken of the welding area during welding is acquired as the target image, and classification model M1 is used to determine whether the target image is an image in which defect occurrence risk cannot be determined or an image in which it can be determined. If the target image is determined to be an image in which defect occurrence risk can be determined, judgment model M2 is used to determine the defect occurrence risk of the welding area shown in the target image, and information according to the judgment result is output. 【0064】 Traditionally, accuracy and explainability have been emphasized in machine learning models. In such cases, it is necessary to create training data by, for example, having the user refer to welding images and identify features such as the position of the molten pool tip, which requires a considerable amount of time for annotation work. While it is conceivable to use a feature extraction program to assist with annotation work, developing such a program is difficult when using images acquired under various welding environments or shooting conditions for machine learning. For example, it is difficult to develop a feature extraction program that can handle the movement and partial loss of the camera's field of view, as well as fluctuations in image brightness, hue, lightness, saturation, and contrast. 【0065】On the other hand, with the welding monitoring device 10, annotation is performed with a simple operation in which the user specifies the defect risk while referring to training images. Therefore, the user does not need to perform tasks such as identifying features, which reduces the workload of annotation work and allows for the generation of a large amount of training data in a short time. In the welding monitoring device 10, since the user specifies the defect risk while referring to training images, the training images only need to be images that allow the user to judge the defect risk. When using the judgment model M2 generated using the training images to determine the defect risk of the welded area shown in the target image, the target image also only needs to be an image that allows the user to judge the defect risk, similar to the training images. Therefore, the constraints of the welding environment and shooting conditions can be relaxed. As a result, the workload for acquiring the target image can be reduced. 【0066】 In other words, the welding monitoring device 10 can use welding images containing irregularities and disturbances as training images. For example, a high degree of consistency or regularity is not required for the viewpoint and field of view of the camera 4, or the welding conditions of the welding location. Therefore, a classification model M1 and a judgment model M2 with high versatility that are not limited to specific welding environments and shooting conditions can be constructed. As a result, the welding monitoring device 10 can accurately determine the risk of defect occurrence even when welding images containing irregularities and disturbances are used as target images. 【0067】 Here, it is conceivable to generate an integrated model by combining the classification model M1 and the judgment model M2 into a single machine learning model. In this case, each training image is labeled "unclassifiable," "normal," "risky," and "high risk," and machine learning is performed with the training images as explanatory variables and information indicating the type of training image specified by the user as the target variable to generate the integrated model. However, there may be more training images for which the defect risk cannot be classified than training images for which it can be classified. In such cases, the above integrated model may not be able to accurately classify the defect risk. 【0068】A training image is labeled "Normal" if the user is confident that there are no defects in the welded area shown in the training image. A training image is labeled "High Risk" if the user is confident that there are defects in the welded area shown in the training image. On the other hand, a training image is labeled "Risk" if the user is not confident that there are defects in the welded area shown in the training image, but judges that there are probably defects. Therefore, the defect risk for training images labeled "Risk" is considered to be greater than 0.5 (50%). The above integrated model does not take into account that the difference between the defect risk for training images labeled "Risk" and the defect risk for training images labeled "High Risk" is smaller than the difference between the defect risk for training images labeled "Normal" and the defect risk for training images labeled "Risk". Therefore, the above integrated model may not be able to accurately determine the defect risk. 【0069】 On the other hand, in the welding monitoring device 10, the determination unit 14 uses the classification model M1 to determine whether the target image is an image in which defect occurrence risk cannot be determined or an image in which defect occurrence risk can be determined. If the target image is determined to be an image in which defect occurrence risk can be determined, the determination unit uses the determination model M2 to determine the defect occurrence risk of the welded area shown in the target image. With this configuration, images that are determined by the classification model M1 to be in which defect occurrence risk cannot be determined are excluded from the target of defect occurrence risk determination. Therefore, it is possible to improve the accuracy of defect occurrence risk determination by the determination model M2. In other words, by preparing separate machine learning models for determining whether an image is in which defect occurrence risk cannot be determined and for determining defect occurrence risk, it is possible to improve the accuracy of the determination. 【0070】The user interface AN is a display element that prompts the user to specify whether a training image is an image for which defect risk cannot be determined, or to specify the defect risk if the training image is an image for which defect risk can be determined. The user performs annotation with a simple task of referring to the training image and specifying whether the training image is an image for which defect risk cannot be determined, or specifying the defect risk if the training image is an image for which defect risk can be determined. Therefore, it is possible to reduce the workload of annotation work. 【0071】 The output unit 15 may output a notification to the worker performing the welding or the supervisor monitoring the welding, according to the risk of defect occurrence. In this case, the worker or supervisor can understand the risk of defect occurrence and take appropriate action. 【0072】 The output unit 15 may output a stop command to the welding control device 3 if the risk of defect occurrence is higher than a preset threshold. With this configuration, welding is stopped when there is a high probability that a defect has occurred at the welding site. Therefore, the possibility of welding continuing in the presence of a defect is reduced. As a result, it becomes possible to improve the quality of the weld. 【0073】 The model generation apparatus and welding monitoring apparatus described herein are not limited to the embodiments described above. 【0074】 For example, the welding monitoring device 10 may be divided into a model generation device having a model generation function for the welding monitoring device 10, and a welding monitoring device having a monitoring function for the welding monitoring device 10. In other words, the model generation device includes an acquisition unit 11, a display control unit 12, a learning unit 13, an image storage unit 21, a learning data storage unit 22, and a model storage unit 23, but does not include a determination unit 14 and an output unit 15. The welding monitoring device includes an acquisition unit 11, a determination unit 14, an output unit 15, an image storage unit 21, and a model storage unit 23, but does not include a display control unit 12, a learning unit 13, and a learning data storage unit 22. 【0075】At least one of the image storage unit 21, the learning data storage unit 22, and the model storage unit 23 may be located outside the welding monitoring device 10. 【0076】 The learning unit 13 does not need to generate a classification model M1. The determination unit 14 does not need to determine whether the target image is an image in which defect occurrence risk cannot be determined or is an image in which it can be determined. In this case, the determination unit 14 may input the target image to the determination model M2 without determining whether it can be determined or not, and obtain the defect occurrence risk of the welded area shown in the target image from the determination model M2. 【0077】 The classification model M1 and the judgment model M2 may be integrated into a single machine learning model (integrated model). In this case, the integrated model may, for example, determine whether the target image input to the integrated model is "unclassifiable," "normal," "risky," or "high risk." 【0078】 In the above embodiment, training images capable of determining the risk of defect occurrence are assigned three levels of labels: "normal," "risky," and "high risk." The labels assigned to training images capable of determining the risk of defect occurrence are not limited to three levels; four or more levels are also acceptable, as long as the granularity is sufficient for the user to make an immediate judgment. 【0079】 The welding system 1 may include multiple cameras 4. In this case, a classification model M1 and a judgment model M2 may be generated for each camera 4. Alternatively, a common classification model M1 and judgment model M2 may be generated using images captured by multiple cameras 4. 【0080】The judgment model M2 may be generated for each type of defect at the weld site. In this case, the output unit 15 may output a message corresponding to the type of defect. For example, if the risk of fusion failure is higher than a preset threshold, the output unit 15 may output a message such as, "The risk of fusion failure is high." The output unit 15 may also output a message that includes the type of defect and specific instructions accordingly. For example, if the risk of internal defect is higher than a preset threshold, the output unit 15 may output a message such as, "There is a high possibility that internal defects remain, so stop welding immediately and perform repair work." 【0081】 The output unit 15 may output an adjustment instruction to the camera 4 to adjust the parameters of the camera 4 if the number of consecutive target images that cannot be determined exceeds a predetermined number. For example, if changing a predetermined parameter in one direction does not reduce the number of target images that cannot be determined, the output unit 15 may change the parameter in the opposite direction. 【0082】 (Note) [Clause 1] A model generation device comprising: an acquisition unit that acquires images of a welding location taken during welding as training images; a display control unit that displays a user interface on a display device together with the training images, allowing the user to specify a defect occurrence risk indicating the degree of confidence that a defect has occurred at the welding location shown in the training images; and a learning unit that generates a judgment model that takes a target image of a welding location taken during welding as input and outputs a defect occurrence risk of the welding location shown in the target image, by performing machine learning with the training images as explanatory variables and the defect occurrence risk specified by the user as the target variable. 【0083】 [Clause 2] The model generation apparatus according to Clause 1, wherein the user interface is a display element that causes the user to specify that the training image is an image in which the risk of defect occurrence cannot be determined if the training image is an image in which the risk of defect occurrence cannot be determined, and causes the user to specify the risk of defect occurrence if the training image is an image in which the risk of defect occurrence can be determined. 【0084】[Clause 3] The model generation device according to Clause 2, wherein the learning unit further generates a classification model that takes the target image as input and outputs a valid flag indicating whether the target image is an image for which defect risk cannot be determined or is an image for which defect risk can be determined, by performing machine learning with the learning image as an explanatory variable and the valid flag specified by the user, which indicates whether the image is an image for which defect risk cannot be determined or is an image for which defect risk can be determined, as the target variable. 【0085】 [Clause 4] A welding monitoring device comprising: an acquisition unit that acquires an image of a welding location taken during welding as a target image; a determination unit that determines the defect risk of the welding location shown in the target image using a determination model generated by performing machine learning with training images of the welding location taken during welding as explanatory variables and the defect risk, which is a defect risk specified by the user and indicates the degree of confidence that a defect has occurred in the welding location shown in the training image, as the objective variable; and an output unit that outputs information corresponding to the determination result by the determination unit. 【0086】 [Clause 5] The welding monitoring device according to Clause 4, wherein the determination unit uses a classification model generated by performing machine learning with the training image as an explanatory variable and the valid flag specified by the user, which indicates whether the training image is an image in which the defect risk cannot be determined or is an image in which it can be determined, to determine whether the target image is an image in which the defect risk cannot be determined or is an image in which it can be determined, and if the target image is determined to be an image in which the defect risk can be determined, the determination unit uses the determination model to determine the defect risk of the welded area shown in the target image. 【0087】 [Clause 6] The welding monitoring device according to Clause 4 or Clause 5, wherein the output unit outputs a notification as information corresponding to the risk of defect occurrence to the worker performing the welding or the supervisor monitoring the welding. 【0088】[Clause 7] The welding monitoring device according to any one of Clauses 4 to 6, wherein the output unit outputs a stop instruction to a welding control device that controls the welding machine performing the welding, as the information, when the risk of defect occurrence is higher than a preset threshold. 【0089】 1. Welding system 2. Welding machine 3. Welding control device 4. Camera 5. External device 10. Welding monitoring device (model generation device) 11. Acquisition unit 12. Display control unit 13. Learning unit 14. Judgment unit 15. Output unit 106. Output device (display device)

Claims

1. A model generation device comprising: an acquisition unit that acquires images of welding locations taken during welding as training images; a display control unit that displays a user interface on a display device together with the training images, allowing the user to specify a defect risk indicating the degree of confidence that a defect has occurred at the welding location shown in the training images; and a learning unit that generates a judgment model that takes a target image of a welding location taken during welding as input and outputs a defect risk of the welding location shown in the target image by performing machine learning with the training images as explanatory variables and the defect risk specified by the user as the target variable.

2. The model generation apparatus according to claim 1, wherein the user interface is a display element that causes the user to specify that the training image is an image in which the risk of defect occurrence cannot be determined if the training image is an image in which the risk of defect occurrence cannot be determined, and causes the user to specify the risk of defect occurrence if the training image is an image in which the risk of defect occurrence can be determined.

3. The model generation device according to claim 2, wherein the learning unit further generates a classification model that takes the target image as input and outputs a valid flag indicating whether the target image is an image for which defect risk cannot be determined or is an image for which defect risk can be determined, by performing machine learning with the learning image as an explanatory variable and the valid flag specified by the user, which indicates whether the image is an image for which defect risk cannot be determined or is an image for which defect risk can be determined, as the target variable.

4. A welding monitoring device comprising: an acquisition unit that acquires an image of a welding location taken during welding as a target image; a determination unit that determines the defect risk of the welding location shown in the target image using a determination model generated by performing machine learning with training images of the welding location taken during welding as explanatory variables and the defect risk, which is a defect risk specified by the user and indicates the degree of confidence that a defect has occurred at the welding location shown in the training image, as the objective variable; and an output unit that outputs information corresponding to the determination result by the determination unit.

5. The welding monitoring device according to claim 4, wherein the determination unit uses a classification model generated by performing machine learning with the training image as an explanatory variable and the valid flag specified by the user, which indicates whether the training image is an image in which the defect risk cannot be determined or is an image in which it can be determined, to determine whether the target image is an image in which the defect risk cannot be determined or is an image in which it can be determined, and if the target image is determined to be an image in which the defect risk can be determined, the determination unit uses the determination model to determine the defect risk of the welded area shown in the target image.

6. The welding monitoring device according to claim 4 or 5, wherein the output unit outputs a notification as information corresponding to the defect risk to the worker performing the welding or the supervisor monitoring the welding.

7. The welding monitoring device according to claim 4 or 5, wherein the output unit outputs a stop instruction to a welding control device that controls the welding machine performing the welding, as the information, when the defect risk is higher than a preset threshold, to stop the welding.

Images