Robot teaching system and robot teaching method

Abstract

This robot teaching system acquires a relative positional relationship between an actual environment and either a teaching member or a three-dimensional model corresponding to the teaching member, and a display device, detects a teaching member operation performed using the teaching member or an in-air operation on the three-dimensional model displayed on the display device, acquires teaching point data based on the relative positional relationship and the detected teaching member operation or in-air operation, stores teaching point data corresponding to a teaching point used for displaying information regarding a welding operation of a robot, and generates and outputs, to the display device, a display image for displaying the information regarding the welding operation.

Description

Robot teaching system and robot teaching method

[0001] This disclosure relates to a robot teaching system and a robot teaching method.

[0002] Patent Document 1 discloses a welding system comprising a welding robot equipped with a torch and a welding robot control program creation device. The welding system acquires positional information of the welding start point and welding end point for welding a workpiece, and orientation information that can identify the orientation of the torch relative to the welding line at a welding teaching point on the welding line connecting the welding start point and the welding end point. Based on the positional information and orientation information, it creates a welding robot control program that performs welding from the welding start point to the welding end point, and performs welding on the workpiece based on the welding robot control program.

[0003] International Publication No. 2021 / 251087

[0004] This disclosure aims to provide a robot teaching system and a robot teaching method that interpolate teaching content in teaching robot movements using XR equipment.

[0005] This disclosure provides a robot teaching system comprising: a teaching point storage unit that stores teaching point data corresponding to teaching points used to display information about the welding operation of a robot; a display device that is wearable by an operator and displays an image superimposed on an image of the real environment or the real environment itself; a position relationship acquisition unit that acquires the relative position relationship of the real environment, a teaching member used to teach the robot or a three-dimensional model corresponding to the teaching member, and the display device; a detection unit that detects a teaching member operation performed by the operator using the teaching member or an aerial operation, which is an operation performed by the operator in the air away from the display device on the three-dimensional model displayed on the display device; an image generation unit that generates a display image for displaying information about the welding operation of the robot based on the relative position relationship and the teaching member operation or aerial operation detected by the detection unit; and an output unit that outputs the display image to the display device.

[0006] Furthermore, this disclosure provides a robot teaching method performed by a system comprising a display device configured to be wearable by an operator and which displays an image superimposed on an image of the real environment or the real environment itself, and a teaching member used for teaching a robot, wherein the system acquires the relative positional relationship between the real environment, the teaching member or a three-dimensional model corresponding to the teaching member, and the display device, detects a teaching member operation performed by the operator using the teaching member or an aerial operation performed by the operator in the air away from the display device on the three-dimensional model displayed on the display device, acquires and stores teaching point data corresponding to teaching points used for displaying information about the robot's welding operation based on the relative positional relationship and the detected teaching member operation or aerial operation, generates a display image for displaying information about the welding operation, and outputs the display image to the display device.

[0007] According to this disclosure, the content of the instruction can be interpolated when teaching robot movements using XR equipment.

[0008] Figures illustrating an example of a welding teaching system according to an embodiment; Figures illustrating an example of the internal configuration of an MR device and a processing device; Figures illustrating the flow of a series of teaching operations; Figures illustrating a method for calculating the start point of an operation; Figures illustrating a method for calculating the end point of an operation; Figures illustrating a method for interpolating intermediate points in linear motion; Figures illustrating a method for interpolating intermediate points in arc motion; Figures illustrating a method for interpolating intermediate points in weaving motion; Sequence diagram showing an example of the first operation procedure of the welding teaching system in the embodiment; Sequence diagram showing the interpolation procedure of an MR device or processing device in the embodiment; Sequence diagram showing an example of the second operation procedure of the welding teaching system in the embodiment; Figures illustrating an example of an interpolation result screen; Figures illustrating an example of an interpolation result screen.

[0009] (Background to this Disclosure) In recent years, there has been a teaching method that uses Augmented Reality (hereinafter referred to as "AR") equipment, such as the welding system described in Patent Document 1, to teach the position of the teaching point taught by the operator and the orientation of the teaching tool at the teaching point. Compared to the case where a general offline teaching system such as a teach pendant is used, this teaching method allows the operator to directly teach the teaching point, thus reducing the time required to teach the teaching point.

[0010] However, conventional welding teaching techniques using AR equipment only determine the teaching posture for each teaching point, and it is necessary to manually input the proposed movement path or movement speed for these teaching points, which is time-consuming. Therefore, this disclosure describes a robot teaching system and a robot teaching method that interpolate the teaching content in teaching robot movements using XR equipment.

[0011] The following describes in detail embodiments of the robot teaching system and robot teaching method disclosed herein, with reference to the drawings as appropriate. However, unnecessary details may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding by those skilled in the art. The accompanying drawings and the following explanation are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0012] This disclosure describes, as an example, a welding robot being taught welding operations, but is not limited to this example. For example, “welding robot” in this disclosure may be replaced with any robot used for other purposes. Also, “welding operations” in this disclosure may be replaced with any operations performed by the robot.

[0013] <Overview of the Welding Teaching System> First, the welding teaching system 100 according to an embodiment will be described with reference to Figure 1. Figure 1 is a diagram showing an example of the welding teaching system 100 according to an embodiment. Note that the welding teaching system 100 shown in Figure 1 is just an example and is not limited thereto.

[0014] The welding teaching system 100 receives teaching operations from an operator regarding the welding operation of the welding robot RB via the teaching tool TL. The welding teaching system 100 interpolates the teaching content based on the teaching operations to generate a teaching program for the welding operation to be performed by the welding robot RB.

[0015] In this disclosure, the welding teaching system 100 is described, as an example, in which teaching operations are performed using a real teaching tool TL and a real workpiece Wk1, but is not limited to this. Teaching operations may be displayed in a mixed reality space through an MR device DV and received by a virtual welding torch VTC constructed based on 3D model data, etc. Furthermore, the teaching tool may, for example, have buttons that accept operator input that are not physical buttons but virtual buttons, or they may be replaced by the operator's fingers. Similarly, the workpiece used for teaching operations may be a virtual workpiece VWk1 displayed in a mixed reality space through an MR device DV and constructed based on 3D model data, etc.

[0016] Furthermore, the teaching points taught in this disclosure may include not only welding points where the workpiece Wk1 is welded, but also approach points where the workpiece Wk1 is approached, avoidance points where obstacles are avoided, idle points where the welding torch TC is idle, or departure points where the workpiece Wk1 is moved away from.

[0017] The welding teaching system 100 includes at least an MR device DV. The welding teaching system 100 shown in Figure 1 includes a workpiece Wk1, a teaching tool TL, an MR device DV, and a processing device P1. Note that if the MR device DV can perform the functions of the processing device P1, the processing device P1 may be omitted.

[0018] The MR device DV is a so-called head-mounted display and is connected to the processing unit P1 for data communication. The MR device DV is worn on the worker's head and creates a virtual space by superimposing images of virtual production equipment (e.g., a virtual workpiece, a virtual welding robot VRB, a virtual welding torch VTC, or a virtual jig, etc.) onto an image of the real space corresponding to the worker's field of view, and displays this virtual space on the display unit 13, thereby making it visible to the worker as a mixed reality space.

[0019] The MR device DV acquires three-dimensional information including the position of the teaching tool TL during the teaching operation, or six-dimensional information including the position and orientation (hereinafter referred to as "teaching information"). The MR device DV transmits the teaching information of the teaching tool TL during the teaching operation to the processing unit P1. The MR device DV also displays interpolation result screens SC1 and SC2 (see Figures 12 and 13), which are generated by the processing unit P1 and show the results of the teaching operation and the interpolation processing by the processing unit P1.

[0020] The welding robot RB is a six-axis articulated robot with a welding torch TC attached to the end of its wrist. The welding robot RB comprises a base, upper arm, forearm, and wrist, arranged from the mounting surface on which the welding robot RB is installed towards the end (wrist). The welding robot RB drives the upper arm, forearm, wrist, or welding torch TC by rotating the first, second, third, fourth, fifth, or sixth axis. The number of joint axes (number of joints) that the welding robot RB has is not limited to these.

[0021] The welding robot RB has a welding torch TC and is a robot that feeds welding wire from the welding torch TC toward the welding location on the workpiece Wk1 to perform welding. The welding robot RB is controlled by a robot controller (not shown) which is connected to a processing device P1 (described later) in a data communication manner, and performs welding operations to weld the workpiece Wk1 based on a teaching program.

[0022] The processing unit P1 is connected to the MR device DV and the robot controller so that data communication is possible. The processing unit P1 receives a teaching operation from the teaching tool TL and performs interpolation of the taught operation path and movement speed as the teaching content indicated by the received teaching operation (i.e., the teaching information of the teaching tool TL). Based on the interpolation result of the teaching operation, the processing unit P1 generates a teaching program to cause the welding robot RB to perform a welding operation. The processing unit P1 transmits the generated teaching program to the robot controller.

[0023] Furthermore, the interpolation processing of the teaching content and the generation processing of the teaching program may be performed by the MR device DV. In such a case, the processing device P1 may be omitted from the configuration of the welding teaching system 100.

[0024] Next, an example of the internal configuration of the MR device DV and the processing unit P1 will be described with reference to Figure 2. Figure 2 is a diagram showing an example of the internal configuration of the MR device DV and the processing unit P1.

[0025] The MR device DV includes a communication unit 10, a processor 11, a memory 12, a display unit 13, a depth sensor 14, and a camera 15.

[0026] The communication unit 10 is connected to the teaching tool TL and the processing unit P1 via wireless or wired communication to perform data transmission and reception. The communication unit 10 outputs various data transmitted from the teaching tool TL and the processing unit P1 to the processor 11. The communication unit 10 transmits various data output from the processor 11 to the processing unit P1. Wireless communication here refers to communication via a wireless Local Area Network (LAN) such as Wi-Fi®. In the welding teaching system 100, if the processing unit P1 is omitted, the communication unit 10 is connected to the robot controller for data communication.

[0027] The processor 11 is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU") or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and works in cooperation with the memory 12 to perform various processes and controls. Specifically, the processor 11 refers to the programs and data held in the memory 12 and executes those programs to realize functions such as acquiring teaching information for the teaching tool TL, visualizing the mixed reality space, and generating and displaying interpolation result screens SC1 and SC2 (see Figures 12 and 13). Note that the processor 11 is configured to realize the same functions as the processor 21 of the processing unit P1 when the processing unit P1 is omitted in the welding teaching system 100.

[0028] The processor 11 calculates the relative positional relationship in three-dimensional space for each of the recognized or detected objects and production equipment, based on the objects detected by the depth sensor 14, the images captured by the camera 15, and the 3D model data of various production equipment stored in the memory 12.

[0029] Specifically, the processor 11 calculates the relative positional relationship between the workpiece Wk1 (virtual workpiece VWk1) and the teaching tool TL, which is modeled after a welding torch TC, in three-dimensional space, for each of the coordinate system and position of the workpiece Wk1 (virtual workpiece VWk1) and the coordinate system and position of the welding robot RB (virtual welding robot VRB). As a result, the processor 11 can generate an image in which the virtual production equipment is superimposed on the captured image of the real world and display it on the display unit 13.

[0030] Memory 12 includes, for example, Random Access Memory (hereinafter referred to as "RAM"), which serves as work memory used when executing each process of the processor 11, and Read Only Memory (hereinafter referred to as "ROM"), which stores programs and data that define the operation of each of the processors 11. Data or information generated or acquired by the processor 11 is temporarily stored in RAM. Programs that define the operation of the processor 11 are written in ROM.

[0031] The memory 12 stores at least a three-dimensional model of at least a part of the welding robot RB or the welding torch TC for welding the workpiece Wk1, three-dimensional models of various production facilities, information regarding the welding robot RB, taught teaching information, etc. Further, the memory 12 stores various data generated by the processor 11 and displayed on the display unit 13.

[0032] Here, the information regarding the welding robot RB is, for example, information regarding the coordinate system of the welding robot RB, the operable range (movable range) of each axis of the welding robot RB, etc.

[0033] The display unit 13 is configured using, for example, a Liquid Crystal Display (hereinafter referred to as "LCD") or an Organic Electroluminescence (hereinafter referred to as "EL"). The display unit 13 realizes Mixed Reality by displaying, for example, an image of a virtual space (e.g., a teaching image) in which a virtual production facility generated by the processor 11 is superimposed on a captured image of the real world captured by the camera 15. The display unit 13 displays an image generated by the processor 11, which is the real world itself, a mixed reality world in which a virtual production facility is superimposed on the real world, or interpolation result screens SC1, SC2 (see FIGS. 12 and 13) showing interpolation results.

[0034] The depth sensor 14 is a sensor that measures the distance between the MR device DV and an object in the real world and recognizes the three-dimensional shape of an object in the real world (e.g., the workpiece Wk1, the welding robot RB, or a jig). The depth sensor 14 outputs the recognition result to the processor 𝐻.

[0035] The camera 15 captures an area (real world) corresponding to the field of view of an operator wearing the MR device DV. The camera 15 outputs the captured image to the processor 11.

[0036] The processing device P1 includes a communication unit 20, a processor 21, a memory <00

[0037] ​The communication unit 20 is connected to the MR device DV and the robot controller via wireless or wired communication to perform data transmission and reception. The communication unit 20 outputs various data transmitted from the MR device DV to the processor 21. The communication unit 20 transmits various data output from the processor 21 to the MR device DV or the robot controller. Wireless communication here refers to communication via a wireless LAN such as Wi-Fi (registered trademark).

[0038] The processor 21 is configured, for example, using a CPU or FPGA, and works in cooperation with the memory 22 to perform various processes and controls. Specifically, the processor 21 refers to the programs and data held in the memory 22 and executes those programs to realize various functions for generating a welding teaching program.

[0039] Memory 22 includes, for example, RAM as work memory used when executing each process of the processor 21, and ROM which stores programs and data that define the operation of each of the processors 21. Data or information generated or acquired by the processor 21 is temporarily stored in RAM. Programs that define the operation of the processor 21 are written in ROM. Memory 22 includes a teaching information recording unit 221 and a work information recording unit 222. Note that the teaching information recording unit 221 and the work information recording unit 222 may be recorded in the memory 12 of the MR device DV. Memory 22 records a 3D model of the workpiece Wk1, a 3D model of the welding robot RB, information related to the welding robot RB, and information related to various production equipment, etc.

[0040] The display unit 23 is configured using, for example, an LCD or an organic EL display. The display unit 23 displays interpolation result screens SC1, SC2 (see Figures 12 and 13), etc., which show the interpolation results generated by the MR device DV.

[0041] The teaching information recording unit 221 records each of the multiple teaching information read using the teaching tool TL for each workpiece Wk1. The workpiece information recording unit 222 records the 3D model of the workpiece Wk1.

[0042] <Method for Teaching Welding Operations> The flow of the teaching operation will be explained with reference to Figure 3. Figure 3 is a diagram illustrating the flow of a series of teaching operations. The example shown in Figure 3 explains an example of teaching a welding operation to weld a straight weld line.

[0043] The operator operates the teaching tool TL to perform teaching operations to teach the starting point Pt10 of the welding robot RB, the ending point Pt1K of the welding robot RB, and at least one intermediate point Pt11, Pt12, Pt13, Pt14 that will move the welding robot RB from the starting point Pt10 to the ending point Pt1K. The operator performs teaching operations in the order of the starting point, intermediate point, and ending point in order to teach a series of movements of the welding robot RB that will operate with a single teaching program.

[0044] The MR device DV detects and acquires teaching information from the teaching tool TL operated by the operator using the depth sensor 14 and the camera 15. Specifically, the MR device DV acquires teaching information from the teaching tool TL at the start point, midpoint, and end point of the operation using the methods described later.

[0045] The teaching information detected by the MR device DV may consist only of three-dimensional information indicating the position of the teaching tool TL, or it may be six-dimensional information indicating the position and orientation of the teaching tool TL, and may be configurable by the operator. For example, when generating interpolation result screen SC1 (see Figure 12), the MR device DV may acquire the position (three dimensions) of the teaching tool TL, and when generating interpolation result screen SC2 (see Figure 13), it may acquire the position and orientation (six dimensions) of the teaching tool TL.

[0046] <Method for Calculating the Start Point> Next, the method for calculating the start point Pt10 will be explained with reference to Figure 4. Figure 4 is a diagram showing the method for calculating the start point Pt10.

[0047] The MR device DV calculates and acquires the position of the starting point Pt10 and the starting orientation of the welding torch TC at the starting point Pt10.

[0048] The MR device DV starts accepting teaching operations when it detects a long press operation by the operator, where a button (not shown) on the teaching tool TL is pressed for a predetermined time (e.g., several seconds) or longer. The MR device DV starts accepting teaching operations at the timing (long press start time T) when the long press operation of the button (not shown) on the teaching tool TL begins. ｐｒｅｓｓ＿ｓｔａｒｔ ), or long press start time T ｐｒｅｓｓ＿ｓｔａｒｔ This occurs after the start time (operation start time T) when the movement of the teaching tool TL begins. ａｃｔｉｏｎ＿ｓｔａｒｔ The teaching information (3D or 6D information) of the teaching tool TL (see Figure 6) is acquired as the starting position of the teaching tool TL.

[0049] The MR device DV starts operation at time T ａｃｔｉｏｎ＿ｓｔａｒｔ Subsequent steps include acquiring teaching information for the teaching tool TL and determining whether the position information included in the latest acquired teaching tool TL's teaching information is within the hand shake distance ΔL1 from the teaching tool TL's starting position. The MR device DV acquires teaching information for the teaching tool TL at discrete sample time Δt and repeatedly determines whether the acquired position of the teaching tool TL is within the hand shake distance ΔL1 until the position of the teaching tool TL is outside the hand shake distance ΔL1.

[0050] The hand shake distance ΔL1 referred to here is the distance at which the positional or postural tremors of the teaching tool TL caused by the operator's hand shake are absorbed. Furthermore, the hand shake distance ΔL1 may be not only a three-dimensional distance relative to the starting position of the operation, but also a six-dimensional (position and posture) distance relative to the starting posture of the operation. The hand shake distance ΔL1 can be set to any value by the operator.

[0051] If the MR device DV determines that the position of the teaching tool TL is located outside the hand shake distance ΔL1, it calculates the average or median value of each teaching position Pt21 to Pt2N of N (N: an integer greater than or equal to 1) of the teaching tool TL acquired within the hand shake distance ΔL1. The MR device DV acquires the calculated average or median value as the operation start point Pt10.

[0052] Here, the teaching position Pt21 is at time T２１ is the position of the teaching tool TL acquired at. The teaching position Pt2N is the position of the teaching tool TL acquired at time (T ２１ + (N × Δt)). Each of the teaching positions Pt21 to Pt2N may be associated with not only the position information of the teaching tool TL but also the posture information of the teaching tool TL.

[0053] Incidentally, the teaching information of the N teaching tools TL used for calculating the operation start point Pt10 may be acquired before the timing when the operation start point Pt10 of the teaching tool TL is acquired, or may be acquired after the timing when the operation start point Pt10 of the teaching tool TL is acquired. Also, the teaching information of the N teaching tools TL may be one or more pieces of teaching information that are considered to be acquired at a position close outside the hand tremor distance ΔL1 immediately before the operation start time T ａｃｔｉｏｎ＿ｓｔａｒｔ and are obtained by removing one or more pieces of teaching information.

[0054] <Method for calculating the operation end point> Next, referring to FIG. 5, the method for calculating the operation end point Pt1K will be described. FIG. 5 is a diagram showing the method for calculating the operation end point Pt1K.

[0055] The MR device DV calculates and acquires the position of the operation end point Pt1K and the operation end posture of the welding torch TC at the operation end point Pt1K, respectively.

[0056] When the MR device DV detects that the long-press operation of the button (not shown) of the teaching tool TL by the operator has been released, the MR device DV ends the reception of the teaching operation. The MR device DV calculates the operation end point Pt1K of the teaching tool TL based on the timing (long-press end time T ｐｒｅｓｓ＿ｅｎｄ , see FIG. 6) when the long-press operation of the button (not shown) of the teaching tool TL is released.

[0057] Incidentally, the operation end point Pt1K of the teaching tool TL is among the teaching information of the teaching tool TL acquired at each discrete sampling time Δt after the long-press end time T ｐｒｅｓｓ＿ｅｎｄ and is the long-press end time T ｐｒｅｓｓ＿ｅｎｄThe teaching information (3D information or 6D information) of a teaching tool TL may be obtained by calculating the average or median of the teaching information (3D information or 6D information) of multiple teaching tool TLs that are within the hand shake distance ΔL1.

[0058] The MR device DV has an operation end time T ａｃｔｉｏｎ＿ｅｎｄ Subsequently, the system acquires teaching information for the teaching tool TL and determines whether the position included in the acquired teaching information for the teaching tool TL is within the hand shake distance ΔL1 from the position of the teaching tool TL's operation end point Pt1K. The MR device DV acquires the position of the teaching tool TL at discrete sample time Δt and repeatedly determines whether the teaching information of the teaching tool TL is within the hand shake distance ΔL1 until the position of the teaching tool TL is outside the hand shake distance ΔL1.

[0059] If the MR device DV determines that the position of the teaching tool TL is located outside the hand shake distance ΔL1, it calculates the average or median value of the teaching positions Pt31 to Pt3M of the M (M: an integer greater than or equal to 1) teaching tools TL acquired within the hand shake distance ΔL1. The MR device DV acquires the calculated average or median value as the operation termination point Pt1K.

[0060] Here, the teaching position Pt31 is at time T ３１ This is the teaching information (3D or 6D information) of the teaching tool TL acquired at time (T). The teaching position Pt3N is at time (T ３１ This is the teaching information (3D or 6D information) of the teaching tool TL obtained in + (M × Δt).

[0061] Furthermore, the end point Pt1K of the teaching tool TL is the long-press end time T ｐｒｅｓｓ＿ｅｎｄ It may be recalculated based on the average or median of the position information (6-dimensional information) of M teaching tool TLs acquired around the time when the operation termination point Pt1K of the teaching tool TL was acquired (calculated), which is in a later time period.

[0062] <Method for calculating midpoints in straight line teaching> Next, we will explain how to calculate midpoints Pt11 to Pt14.

[0063] After acquiring the start point Pt10, the MR device DV calculates and acquires at least one intermediate point in order to determine the operating path of the welding torch TC up to the end point Pt1K.

[0064] After acquiring the starting point Pt10, the MR device DV acquires teaching information from the teaching tool TL at discrete sample time intervals Δt. The MR device DV stores the position or orientation indicated by each piece of teaching information from the teaching tool TL acquired at discrete sample time intervals Δt as position or orientation information for intermediate points Pt11 to Pt14 in a time series.

[0065] Furthermore, the MR device DV may acquire teaching information not only at discrete sample time Δt, but also each time the teaching tool TL moves a recognition distance ΔP (= constant distance). In such a case, the MR device DV stores the position or orientation indicated by each piece of teaching information of the teaching tool TL acquired each time it moves a recognition distance ΔP as position or orientation information of intermediate points Pt11 to Pt14 in a time series. Note that the recognition distance ΔP here may be the same distance as the hand shake distance ΔL1, or it may be a distance smaller or larger than the hand shake distance ΔL1. In addition, the MR device DV may acquire teaching information at indefinite time intervals based on the processing speed at which the processors 11 and 21 of the MR device DV or the processing device P1 process various information (data).

[0066] <Method for Interpolating the Motion Path> Next, with reference to Figure 6, the interpolation method for intermediate points Pt11 to Pt14 in linear motion will be explained. Figure 6 is a diagram showing the interpolation method for intermediate points Pt11 to Pt14 in linear motion. Note that, as an example, each intermediate point Pt11 to Pt14 shown in Figure 6 is calculated based on the position information of the teaching tool TL acquired at discrete sample time Δt.

[0067] <Method for Interpolating a Straight Path> The MR device DV interpolates the taught path of the welding robot RB based on the acquired start point Pt10, intermediate point, and end point Pt1K (i.e., teaching points).

[0068] First, the MR device DV interpolates the linear path RT1, which is the operating path, based on the operation start point Pt10 (3D or 6D information) and the operation end point Pt1K (3D or 6D information). Here, the linear path RT1 shown in Figure 6 is the path connecting the operation start point Pt10 and the operation end point Pt1K.

[0069] The MR device DV sets a camera shake distance ΔL2 that allows for positional deviations in a direction perpendicular to the straight-line path RT1 over the entire range of the straight-line path RT1. The MR device DV determines whether the position of each intermediate point acquired between the start point Pt10 and the end point Pt1K of the operation is within the camera shake distance ΔL2. Note that the camera shake distance ΔL2 here may be the same as or different from the camera shake distance ΔL1. Furthermore, the camera shake distance ΔL2 may be set to different values ​​depending on the type of operation path (e.g., straight line, arc, or weaving).

[0070] The MR device DV determines that the operation path of the welding robot RB taught by this teaching operation is the straight path RT1 if it determines that the number or percentage of intermediate points that are not within the hand shake distance ΔL2 corresponding to the straight path RT1 is less than or equal to a predetermined value (e.g., a few points, a few percent, etc.).

[0071] <Method for Interpolating Circular Paths> On the other hand, if the MR device DV determines that the number or proportion of intermediate points not within the hand shake distance ΔL2 is not less than a predetermined value (for example, a few points, a few percent, etc.), it estimates that the operation path of the welding robot RB taught by this teaching operation is a circular path and re-interpolates the operation path.

[0072] First, the MR device DV interpolates the arc path RT2, which is the movement path, based on the start point Pt10 (3D or 6D information), the end point Pt1K (3D or 6D information), and the midpoint between the start point Pt10 and the end point Pt1K. Here, the arc path RT2 shown in Figure 7 is the path connecting the start point Pt10 and the end point Pt1K.

[0073] The MR device DV sets a hand shake distance ΔL2 that allows for positional or attitudeal shake in a direction perpendicular to the arc path RT2 throughout the entire arc path RT2. The MR device DV determines whether the position of each intermediate point acquired between the start point Pt10 and the end point Pt1K of the operation is within the hand shake distance ΔL2.

[0074] The MR device DV determines that the operating path of the welding robot RB taught by this teaching operation is the arc path RT2 if it determines that the number or percentage of intermediate points that are not within the hand shake distance ΔL2 corresponding to the arc path RT2 is less than or equal to a predetermined value (e.g., a few points, a few percent, etc.).

[0075] <Method for Interpolating the Weaving Path> On the other hand, if the MR device DV determines that the number or proportion of intermediate points that are not within the hand shake distance ΔL2 corresponding to the arc path RT2 is not less than a predetermined value (for example, a few points, a few percent, etc.), it estimates that the operation path of the welding robot RB taught by this teaching operation is a weaving path and interpolates an auxiliary line LN1 for interpolating the weaving path RT3.

[0076] First, the MR device DV interpolates a linear path RT1 to interpolate the weaving path RT3, which is the movement path, based on the start point Pt10 (3D or 6D information) and the end point Pt1K (3D or 6D information). Here, the auxiliary line LN1 shown in Figure 8 is the path connecting the start point Pt10 and the end point Pt1K.

[0077] The MR device DV resets the camera shake distance ΔL2 over the entire length of the straight path RT1, allowing for positional deviations in a direction perpendicular to the straight path RT1. The MR device DV detects and counts the intermediate points acquired between the start point Pt10 and the end point Pt1K that fall outside the camera shake distance ΔL2.

[0078] Here, the MR device DV counts each detected midpoint that extends consecutively in the same direction relative to the auxiliary line LN1 as one extension. For example, in the example shown in Figure 8, each of the midpoints Pt51, Pt52, Pt53, Pt54, ..., Pt5W in group W (W: integer of 2 or more) is a midpoint that extends consecutively in the same direction relative to the auxiliary line LN1. In such a case, the MR device DV counts each of the midpoints Pt51 to Pt5W as one extension, and counts a total of W extensions.

[0079] The MR device DV calculates the average or median (= first distance WH1) between the auxiliary line LN1 and the maximum value of each intermediate point Pt51, Pt53, ... that extends in one direction relative to the auxiliary line LN1. The MR device DV also calculates the average or median (= second distance WH2) between the auxiliary line LN1 and the maximum value of each intermediate point Pt52, Pt54, ..., Pt5W that extends in the other direction relative to the auxiliary line LN1. Based on the calculated first distance WH1 and second distance WH2, the MR device DV calculates the weaving width WH. Note that the maximum value here refers to the maximum distance from the auxiliary line LN1.

[0080] The MR device DV calculates the weaving frequency based on the distance of the auxiliary line LN1 between the start point Pt10 and the end point Pt1K, and the counted number of overshoots W. The MR device DV interpolates the weaving path RT3 connecting the start point Pt10 and the end point Pt1K based on the calculated weaving width WH and the weaving frequency.

[0081] Furthermore, the MR device DV may interpolate a linear or circular path using algorithms such as the least squares method or Random Sample Consensus (RANSAC) for each acquired intermediate point. In such cases, the MR device DV determines whether short linear paths appear periodically as a result of the interpolation, and if it determines that short linear paths appear periodically, it interpolates the weaving path RT3. Note that the algorithm used for interpolating the motion path is not limited to the above example, but any known algorithm capable of interpolating the motion path based on multiple intermediate points is acceptable.

[0082] <Method for Interpolating Composite Paths> Furthermore, the MR device DV sequentially performs interpolation of the motion path based on the position or orientation of intermediate points acquired in real time during the teaching operation. This allows the MR device DV to interpolate motion paths that are composed of paths of different line types, or motion paths that are of the same line type but composed of different paths (for example, arc paths with different radii, straight paths in different directions, etc.).

[0083] For example, if the MR device DV determines, based on the interpolated motion path, that a straight path appears while an arc path is being taught, it determines that the taught motion path has switched from an arc path to a straight path, and can interpolate a motion path that is a combination of an arc path and a straight path. Also, if the MR device DV determines, based on the interpolated motion path, that an arc path appears while a straight path is being taught, it determines that the taught motion path has switched from a straight path to an arc path, and can interpolate a motion path that is a combination of a straight path and an arc path.

[0084] Furthermore, if the MR device DV determines that there is a change in radius in the arc path being taught, it interpolates the combined motion path of multiple arc paths, each with a different radius. Also, if the MR device DV determines that there is a change in the direction of the taught straight line in the straight line path being taught, it interpolates the combined motion path of multiple straight line paths, or if it determines that the change in the direction of the taught straight line is repeated at a predetermined period, it interpolates the combined motion path of a straight line path and a weaving path. The same applies to combinations of arc paths and weaving paths, and combinations of different weaving paths, although this will not be explained here.

[0085] <Method of interpolating movement speed> The MR device DV operates at the start time T ｐｒｅｓｓ＿ｓｔａｒｔ From the end time of operation T ｐｒｅｓｓ＿ｅｎｄ Operation time T ＿ａｃｔｉｏｎ The MR device DV calculates the travel distance L from the start point Pt10 to the end point Pt1K based on the interpolated motion path. ＿ａｃｔｉｏｎ The MR device DV calculates the calculated travel distance L. ＿ａｃｔｉｏｎ Operating time T ＿ａｃｔｉｏｎ The travel speed of the welding torch TC is calculated by dividing by L. ＿ａｃｔｉｏｎ This differs for straight paths, circular arc paths, and weaving paths, respectively.

[0086] <First Operation Procedure of the Welding Teaching System> Next, an example of the first operation procedure of the welding teaching system 100 will be described with reference to Figures 9 and 10, respectively. Figure 9 is a sequence diagram showing an example of the first operation procedure of the welding teaching system 100 in the embodiment. Figure 10 is a sequence diagram showing the interpolation procedure of the MR device DV or processing device P1 in the embodiment. Note that in the description of Figure 9, an example in which the processing device P1 performs interpolation processing will be described.

[0087] The first operation procedure is the procedure for performing interpolation of the movement path and movement speed after a series of teaching operations from the start point to the end point of the movement have been completed.

[0088] The operator performing the teaching aligns the teaching tool TL with the starting position of the welding robot RB relative to the workpiece Wk1. The operator then presses and holds a button (not shown) to start the teaching operation (St11).

[0089] When the MR device DV detects that a long press operation has been received by the teaching tool TL, it accepts the start of the teaching operation (St12). The MR device DV acquires the teaching information (3D information or 6D information) from the teaching tool TL and transmits it to the processing device P1 (St13). The MR device DV repeatedly executes the process in step St13 at discrete sample time Δt until it accepts the end of the teaching operation.

[0090] The processing unit P1 calculates the start point Pt10 and performs intermediate point acquisition (St14) based on the teaching information of the teaching tool TL transmitted from the MR device DV.

[0091] The operator aligns the teaching tool TL with the welding robot RB at the end of its movement relative to the workpiece Wk1. The operator then presses and holds a button (not shown) to end the teaching operation (St15).

[0092] When the MR device DV detects that a long press operation has been received by the teaching tool TL, it accepts that the teaching operation has ended. The MR device DV generates a notification indicating the end of the teaching operation and notifies the processing unit P1 (St16). The MR device DV also acquires the teaching information from the teaching tool TL and transmits it to the processing unit P1 (St16). The MR device DV may continue the process of acquiring and transmitting the teaching information from the teaching tool TL for a predetermined time (for example, several seconds) after the long press operation.

[0093] The processing unit P1 calculates and obtains the end point Pt1K of the operation. Based on the start point Pt10, the multiple intermediate points Pt11 to Pt14, and the end point Pt1K, the processing unit P1 interpolates the taught operation path and movement speed (St14).

[0094] Specifically, the processing unit P1 accepts the start of a teaching operation by the teaching tool TL based on a notification transmitted from the MR device DV (St141). The processing unit P1 acquires the teaching information of the teaching tool TL that is periodically transmitted from the MR device DV, and records and stores it in the memory 22.

[0095] The processing unit P1 repeatedly performs recognition processing of the position or orientation of the teaching tool TL at discrete sample times Δt or at fixed distance intervals based on the teaching information of the teaching tool TL (St142).

[0096] The processing unit P1 accepts the termination of the teaching operation by the teaching tool TL based on the notification sent from the MR device DV (St143).

[0097] When the processing unit P1 receives confirmation that the teaching operation has ended, it performs a recognition process for the position or orientation of the teaching tool TL acquired during the period up to the end of the teaching operation or during a predetermined time (e.g., a few seconds) after the end of the teaching operation. The processing unit P1 recognizes the long press start time T ｐｒｅｓｓ＿ｓｔａｒｔ or long press to end time T ｐｒｅｓｓ＿ｅｎｄ Based on the three-dimensional or six-dimensional orientation of the teaching tool TL up to this point, the type of motion path, the movement path, and the movement speed are interpolated (St144). The type of motion path here refers to whether the motion path is a straight line, a circular arc, or a weaving motion. If the motion path is a combination of multiple identical or different paths, the processing unit P1 interpolates the type of motion path for each section of the motion path.

[0098] The processing unit P1 transmits information about the start point Pt10 and the end point Pt1K of the operation, as well as information about the operation path and movement speed as interpolated results, to the MR device DV (St17).

[0099] The MR device DV superimposes the linear path RT1, which is the interpolation result, onto the workpiece Wk1 captured in the image taken by the camera 15, and superimposes interpolation information IF1, which includes information on the movement speed and information on the type of linear path RT1, at any position on the image to generate interpolation result screens SC1 and SC2 (see Figures 12 and 13), which are then displayed on the display unit 13 (St18).

[0100] The operator confirms, based on the interpolation result screen SC1 displayed on the display unit 13, whether the linear path RT1 obtained by interpolation matches the teaching content requested by the operator. The operator performs an input operation (St19) to indicate whether or not to use the interpolation result in generating the teaching program. This input operation may be performed by any method, such as pressing (selecting) a button (not shown) displayed on the interpolation result screen SC1 or operating the processing unit P1. The interpolation result screen SC1 may also be displayed on the display unit 23 of the processing unit P1. In such a case, the processing unit P1 accepts the operator's input operation to indicate whether or not to use the result in generating the teaching program.

[0101] The MR device DV transmits the input result regarding the usability of the interpolation result performed by the operator to the processing unit P1 (St20).

[0102] The processing unit P1 performs processing based on the input result (St21). Specifically, if the processing unit P1 obtains an input result indicating that the interpolation result can be used, it generates a teaching program to operate the welding robot RB based on the interpolation result (St211). On the other hand, if the processing unit P1 obtains an input result indicating that the interpolation result cannot be used, it omits the generation of the teaching program.

[0103] As a result, the welding teaching system 100 according to this embodiment can interpolate the operation path and movement speed necessary for generating a teaching program based on teaching operations using AR equipment. Therefore, the welding teaching system 100 eliminates the need for operators to input the operation path and movement speed, and can make the teaching work more efficient.

[0104] <Second Operation Procedure of the Welding Teaching System> Next, an example of the second operation procedure of the welding teaching system 100 will be described with reference to Figures 10 and 11, respectively. Figure 11 is a sequence diagram showing an example of the second operation procedure of the welding teaching system 100 in an embodiment.

[0105] In the explanation of Figure 11, we will describe an example in which the processing device P1 performs interpolation processing. Furthermore, the processing in steps St11 to St13, St15 to St16, and St17 to St21 shown in Figure 11 is the same as the processing shown in Figure 9, so we will omit the explanation.

[0106] The second operating procedure is the procedure for sequentially performing interpolation based on the attitude information of the teaching tool TL transmitted from the MR device DV.

[0107] The MR device DV and processing unit P1 sequentially repeat the processes of steps St13 to St18A to sequentially acquire teaching information for the teaching tool TL and interpolate the operation path based on the acquired teaching information for the teaching tool TL (RP).

[0108] The MR device DV acquires teaching information from the teaching tool TL and transmits it to the processing unit P1 (St13). The processing unit P1 repeatedly executes the process in step St13 at each discrete sample time Δt until it accepts the end of the teaching operation. The MR device DV calculates the start point Pt10 and acquires intermediate points (St14A). The processing unit P1 sequentially performs interpolation of the operation path based on the teaching information from each teaching tool TL acquired at each discrete sample time Δt (St14A). Step St14A is the same process as steps St141 to St142 shown in Figure 10.

[0109] The processing unit P1 transmits the latest interpolation information, which is the operation path information, to the MR device DV (St17A).

[0110] The MR device DV superimposes the linear path RT1, which is the interpolation result, onto the workpiece Wk1 captured in the image taken by the camera 15, and superimposes interpolation information IF1, which includes information on the movement speed and information on the type of linear path RT1, at any position on the image. This generates interpolation result screens SC1 and SC2 (see Figures 12 and 13) in real time and displays them on the display unit 13 (St18A).

[0111] When the MR device DV detects that the long-press operation by the teaching tool TL has ended, it accepts the end of the teaching operation. The MR device DV generates a notification indicating the end of the teaching operation and notifies the processing unit P1 (St16).

[0112] When the processing unit P1 receives a notification from the MR device DV, it interpolates the taught operating path and movement speed based on the operation start point Pt10, a plurality of intermediate points Pt11 to Pt14, and the operation end point Pt1K (St14B). Step St14B is the same process as steps St143 to St144 shown in Figure 10.

[0113] The processing unit P1 transmits information about the start point Pt10 and the end point Pt1K of the operation, as well as information about the operation path and movement speed as interpolated results, to the MR device DV (St17).

[0114] As a result, the welding teaching system 100 according to this embodiment can interpolate the movement path and movement speed necessary for generating a teaching program based on teaching operations using AR equipment. Furthermore, by sequentially performing interpolation processing, the welding teaching system 100 can interpolate the movement path even when the movement path is a composite path of multiple movement paths of the same or different types of welds. Therefore, the welding teaching system 100 eliminates the effort required for the operator to input the movement path and movement speed, making the teaching work more efficient.

[0115] Next, the interpolation result screen SC1 will be explained with reference to Figure 12. Figure 12 is a diagram showing an example of the interpolation result screen SC1.

[0116] The interpolation result screen SC1 is generated by the MR device DV that performs the interpolation process and is displayed on the display unit 13 of the MR device DV.

[0117] The MR device DV superimposes an interpolated linear path RT1 onto the real-world work Wk1 or the constructed virtual work VWk1 captured by the camera 15. The MR device DV generates interpolation information IF1, which includes the movement speed information "speed: 1.0 m / min" obtained by interpolation and the type of movement path information "interpolation: linear," and superimposes it at any position on the captured image captured by the camera 15. The MR device DV generates an interpolation result screen SC1, which includes the real-world work Wk1 or virtual work VWk1 with the movement path (linear path RT1) superimposed, and the interpolation information IF1, and displays it on the display unit 13.

[0118] As described above, the welding teaching system 100 can visualize the interpolation results to the operator by generating and displaying the interpolation result screen SC1. By visualizing the interpolation results to the operator, the welding teaching system 100 makes it possible for the operator to confirm whether the teaching content corresponding to the interpolation results is the teaching content requested by the operator. Furthermore, the welding teaching system 100 can determine whether or not it is possible to generate a teaching program using the interpolation results by receiving input from the operator regarding whether or not the interpolation results can be used.

[0119] Next, the interpolation result screen SC2 will be explained with reference to Figure 13. Figure 13 is a diagram showing an example of the interpolation result screen SC2.

[0120] The interpolation result screen SC2 is generated by the MR device DV and displayed on the display unit 13 of the MR device DV.

[0121] The MR device DV superimposes an interpolated linear path RT1 and an arrow indicating the direction of movement of the linear path RT1 onto the real-world workpiece Wk1 or the constructed virtual workpiece VWk1 captured by the camera 15.

[0122] Furthermore, the MR device DV generates multiple virtual welding torches VTC1, VTC2, VTC3, and VTC4 that move from the starting point Pt10 to the ending point Pt1K, based on the orientation information of the teaching tool TL acquired at each of the multiple intermediate points acquired between the starting point Pt10 and the ending point Pt1K. The MR device DV generates a video in which each of the multiple virtual welding torches VTC1 to VTC4 moves in chronological order along a linear path RT1 on the real-world workpiece Wk1 or the constructed virtual workpiece VWk1.

[0123] The MR device DV generates interpolation information IF2, which includes information on the movement speed obtained by interpolation ("speed: 1.0 m / min") and information on the type of movement path ("interpolation: straight line"), and superimposes it on any position on the captured image captured by the camera 15.

[0124] The MR device DV generates an interpolation result screen SC2 which includes an actual workpiece Wk1 or a virtual workpiece VWk1 in which the operation path (linear path RT1) and multiple virtual welding torches VTC1 to VTC4 are superimposed, as well as interpolation information IF2. The MR device DV displays the interpolation result screen SC2, which is a video of the multiple virtual welding torches VTC1 to VTC4 moving along the operation path (linear path RT1) superimposed on the actual workpiece Wk1 or virtual workpiece VWk1.

[0125] As described above, the welding teaching system 100 can visualize the interpolation results and the teaching content based on the movement and posture of the virtual welding torches VTC1 to VTC4 to the operator by generating and displaying the interpolation result screen SC2. By visualizing the interpolation results to the operator, the welding teaching system 100 makes it possible for the operator to confirm whether the teaching content corresponding to the interpolation results is the teaching content requested by the operator. In addition, the welding teaching system 100 can determine whether or not to generate a teaching program using the interpolation results by receiving input from the operator regarding the use of the interpolation results.

[0126] (Note) The following technologies are disclosed based on the descriptions of each embodiment above.

[0127] (Technology 1) A teaching point storage unit (memory 12) that stores teaching point data corresponding to teaching points (i.e., a start point Pt10, an end point Pt1K, and multiple intermediate points) used to display information (operation path and movement speed) related to the welding operation of a robot (e.g., a welding robot RB); a display device (display unit 13 of an MR device DV) that is configured to be wearable by an operator and displays an image superimposed on an image of the real environment (real world) or the real environment itself; a position relationship acquisition unit (processor 11) that acquires the relative position relationship of the real environment, teaching members used for teaching the robot (e.g., a teaching tool TL, a virtual teaching tool VTL, or the operator's fingers, etc.) or a 3D model corresponding to the teaching members, and the display device (display unit 13); A robot teaching system (welding teaching system 100) comprising: a detection unit (depth sensor 14 or camera 15) that detects an operation of a teaching member performed by the operator using the teaching member or an aerial operation performed by the operator in the air away from the display device (display unit 13) on the three-dimensional model displayed on the display device (display unit 13); an image generation unit (processor 11) that generates display images (interpolation result screens SC1, SC2) for displaying information related to the welding operation of the robot based on the relative positional relationship and the teaching member operation or the aerial operation detected by the detection unit (depth sensor 14 or camera 15); and an output unit (processor 11) that outputs the display images (interpolation result screens SC1, SC2) to the display device (display unit 13). As a result, the welding teaching system 100 according to the embodiment can generate and display interpolation result screens SC1, SC2 (see Figures 12 and 13) that visualize the movement path and movement speed as interpolated interpolation results based on teaching operations using XR equipment. Therefore, the welding teaching system 100 eliminates the need for the operator to input the movement path and travel speed.

[0128] (Technical 2) The robot teaching system (welding teaching system 100) according to (Technical 1), wherein the teaching member operation or the aerial operation is a continuous operation performed by the operator using the teaching member or a continuous operation performed by the operator in the air away from the display device (display unit 13). As a result, the welding teaching system 100 according to the embodiment can accept teaching operations using a teaching tool TL or teaching operations using the operator's fingers using an XR device.

[0129] (Technology 3) The detection unit (depth sensor 14 or camera 15) interpolates welding operation information (operation path and movement speed) for displaying information related to the welding operation of the robot based on the relative positional relationship acquired by the positional relationship acquisition unit (processor 11) while the operator is performing the continuous operation, and outputs the interpolated welding operation information to the image generation unit (processor 11), as described in (Technology 2) (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can interpolate welding operation information by accepting teaching operations using a teaching tool TL or teaching operations using the operator's fingers using an XR device.

[0130] (Technical 4) The robot teaching system (welding teaching system 100) according to (Technical 2), further comprising an interpolation unit (processor 11) that interpolates the shape of the motion trajectory (type of motion path) corresponding to the continuous operation based on the relative positional relationship acquired by the positional relationship acquisition unit (processor 11) while the continuous operation is being performed by the operator. As a result, the welding teaching system 100 according to the embodiment can estimate the type of motion path taught and interpolate the motion path based on teaching operations using XR equipment. Therefore, the welding teaching system 100 can eliminate the effort required for the operator to set the type of motion path.

[0131] (Technical 5) The interpolation unit (processor 11) interpolates the shape of the motion trajectory based on the start position (i.e., the position of the start point Pt10) and end position (i.e., the position of the end point Pt1K) of the continuous operation, and a plurality of intermediate positions (i.e., the positions of a plurality of intermediate points) between the start position and the end position obtained by the continuous operation, as described in (Technical 4) of the robot teaching system (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can estimate the type of motion path from the start point to the end point of the operation and interpolate the motion path based on teaching operations using XR equipment. Therefore, the welding teaching system 100 can eliminate the effort of the operator to input the type of motion path.

[0132] (Technical 6) The interpolation unit (processor 11) interpolates the shape of the motion trajectory based on the plurality of intermediate positions relative to the auxiliary line LN1 connecting the start position and the end position obtained by the continuous operation, as described in (Technical 5) of the robot teaching system (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can estimate whether or not the taught motion path is a weaving path based on teaching operations using XR equipment. Therefore, the welding teaching system 100 can eliminate the effort of the operator to input the type of motion path.

[0133] (Technical 7) The interpolation unit (processor 11) determines that the shape of the motion trajectory is a straight line if the plurality of intermediate positions are within a predetermined distance (hand shake distance ΔL2) in a direction perpendicular to the auxiliary line LN1, and determines that the shape of the motion trajectory is an arc if the plurality of intermediate positions are not within the predetermined distance (hand shake distance ΔL2), as described in (Technical 6) for the robot teaching system (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can estimate whether the type of taught motion path is a straight line or an arc based on the teaching operation using the XR device and the hand shake distance. Therefore, the welding teaching system 100 can eliminate the effort of the operator to input the type of motion path.

[0134] (Technical 8) The interpolation unit (processor 11) determines that the shape of the motion trajectory is weaving if the number of intermediate positions located outside the predetermined distance (hand shake distance ΔL2) is greater than or equal to a predetermined number, and determines that the shape of the motion trajectory is an arc if the number of intermediate positions located outside the predetermined distance is less than or equal to the predetermined number, as described in (Technical 7) of the robot teaching system (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can estimate whether the type of taught motion path is weaving or not based on the teaching operation using the XR device and the hand shake distance, and interpolate the motion path. Therefore, the welding teaching system 100 can eliminate the effort of the operator having to input the type of motion path.

[0135] (Technical 9) The robot teaching system (welding teaching system 100) described in (Technical 4), wherein the image generation unit (processor 11) generates the display images (interpolation result screens SC1, SC2) for displaying the shape of the motion trajectory interpolated by the interpolation unit (processor 11) on the display device (display unit 13). As a result, the welding teaching system 100 according to the embodiment can visualize the interpolated motion path to the operator based on teaching operations using XR equipment.

[0136] (Technical 10) The robot teaching system (welding teaching system 100) according to (Technical 2), further comprising an interpolation unit (processor 11) that interpolates the operating speed of the robot's welding operation (i.e., the movement speed of the welding torch TC) based on the relative positional relationship acquired by the positional relationship acquisition unit (processor 11) while the continuous operation is being performed by the operator, and the elapsed time (i.e., operation time) during which the continuous operation is being performed. As a result, the welding teaching system 100 according to the embodiment can interpolate the operation path and movement speed based on teaching operations using XR equipment. Therefore, the welding teaching system 100 can eliminate the need for the operator to set the type of operation path and the movement speed.

[0137] (Technical 11) The robot teaching system (welding teaching system 100) described in (Technical 10), wherein the image generation unit (processor 11) generates the display images (interpolation result screens SC1, SC2) for displaying the operation speed interpolated by the interpolation unit (processor 11) on the display device (display unit 13). As a result, the welding teaching system 100 according to the embodiment generates interpolation result screens SC1, SC2 (see Figures 12 and 13) that visualize the operation path and movement speed based on teaching operations using XR equipment, and makes them visible to the operator.

[0138] (Technical 12) The positional relationship acquisition unit (processor 11) acquires the relative positional relationship between the actual environment, the teaching member or the 3D model, the display device (display unit 13), and the robot present in the actual environment, as described in any one of (Technical 1) to (Technical 11) of the robot teaching system (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can accept teaching operations for the welding robot RB in the real world based on teaching operations using XR equipment.

[0139] (Technical 13) A robot teaching system (welding teaching system 100) according to any one of (Technical 1) to (Technical 11), further comprising a timing unit (processor 11) for timing elapsed time, wherein the position relationship acquisition unit (processor 11) acquires the relative position relationship each time the elapsed time timing by the timing unit exceeds a predetermined time (discrete sample time Δt). As a result, the welding teaching system 100 according to the embodiment can acquire teaching point data based on teaching operations at discrete sample time Δt intervals.

[0140] (Technical 14) The position relationship acquisition unit (processor 11) acquires the relative position relationship again within a predetermined processing time after acquiring the relative position relationship, as described in any one of (Technical 1) to (Technical 11) of the robot teaching system (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can acquire teaching point data at indefinite intervals based on the processing speed of the processor 11.

[0141] (Technical 15) A robot teaching system (welding teaching system 100) according to any one of (Technical 1) to (Technical 11), further comprising a measuring unit (depth sensor 14) that measures the distance between the display device (display unit 13) and the teaching member or the three-dimensional model based on the relative positional relationship acquired by the positional relationship acquisition unit (processor 11), wherein the positional relationship acquisition unit (processor 11) acquires the relative positional relationship each time the distance measured by the measuring unit exceeds a predetermined distance.

[0142] (Technical 16) The operation of the teaching member is a physical operation by the operator on an operation unit (physical button or virtual button) provided on the teaching member, as described in any one of (Technical 1) to (Technical 11) of the robot teaching system (welding teaching system 100). As a result, the welding teaching system 100 according to the embodiment can accept operator operations via a teaching tool TL or a virtual teaching tool VTL.

[0143] (Technical 17) The teaching member has an operating section (physical or virtual button) formed by the three-dimensional model that accepts operations from the teaching member or the operator, and the aerial operation is an operation performed by the operator in the air away from the display device (display unit 13) on the teaching member displayed via the teaching member, the robot teaching system (welding teaching system 100) according to any one of (Technical 1) to (Technical 11). As a result, the welding teaching system 100 according to the embodiment can accept operator operations by virtual buttons formed on the teaching tool TL or virtual teaching tool VTL.

[0144] (Technical 18) A robot teaching method performed by a system (welding teaching system 100) comprising: a display device (MR device DV) configured to be wearable by an operator and which displays an image superimposed on an image of the real environment (real world) or the real environment itself; and a teaching member (teaching tool TL) used for teaching a robot (e.g., a welding robot RB), wherein the system acquires the relative positional relationship of the real environment, the teaching member or a three-dimensional model corresponding to the teaching member, and the display device (display unit 13); detects a teaching member operation performed by the operator using the teaching member or an aerial operation performed by the operator in the air away from the display device (display unit 13) with respect to the three-dimensional model displayed on the display device (display unit 13); and acquires and stores teaching point data corresponding to teaching points (i.e., a start point Pt10, an end point Pt1K, and a plurality of intermediate points) used for displaying information related to the welding operation of the robot (operation path and movement speed), based on the relative positional relationship and the detected teaching member operation or aerial operation. A robot teaching method comprising: generating display images (interpolation result screens SC1, SC2) for displaying information related to the welding operation, and outputting the display images (interpolation result screens SC1, SC2) to the display device (display unit 13). As a result, the welding teaching system 100 according to the embodiment can generate and display interpolation result screens SC1, SC2 (see Figures 12 and 13) that visualize the interpolated movement path and movement speed as interpolated results based on teaching operations using XR equipment. Therefore, the welding teaching system 100 can eliminate the effort required for the operator to input the movement path and movement speed.

[0145] Although various embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It will be clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these will also be understood to fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention.

[0146] This application is based on Japanese Patent Application No. 2025-019621 filed on February 7, 2025, and its contents are incorporated herein by reference.

[0147] This disclosure is useful as a robot teaching system and robot teaching method for interpolating teaching content in teaching robot movements using XR equipment.

[0148] 10, 20 Communication unit 11, 21 Processor 12, 22 Memory 13, 23 Display unit 14 Depth sensor 15 Camera 100 Welding teaching system 221 Teaching information recording unit 222 Work information recording unit DV MR device LN1 Auxiliary line P1 Processing unit Pt10 Start point of operation Pt1K End point of operation Pt11, Pt12, Pt13, Pt14, Pt41, Pt42, Pt43, Pt44, Pt45, Pt46, Pt47, Pt48, Pt51, Pt52, Pt53, Pt54, Pt5W Midpoint RB Welding robot SC1, SC2 Interpolation result screen TC Welding torch TL Teaching tool VRB Virtual welding robot VTC, VTC1, VTC2, VTC3, VTC4: Virtual welding torches; Wk1, Wk2, Wk3: Workpieces; VWk1: Virtual workpieces

Claims

1. A robot teaching system comprising: a teaching point storage unit that stores teaching point data corresponding to teaching points used to display information about the welding operation of a robot; a display device configured to be wearable by an operator and that displays an image superimposed on an image of the real environment or the real environment itself; a position relationship acquisition unit that acquires the relative position relationship of the real environment, a teaching member used to teach the robot or a three-dimensional model corresponding to the teaching member, and the display device; a detection unit that detects a teaching member operation performed by the operator using the teaching member or an aerial operation performed by the operator in the air away from the display device on the three-dimensional model displayed on the display device; an image generation unit that generates a display image for displaying information about the welding operation of the robot based on the relative position relationship and the teaching member operation or the aerial operation detected by the detection unit; and an output unit that outputs the display image to the display device.

2. The robot teaching system according to claim 1, wherein the teaching member operation or the aerial operation is a continuous operation performed by the operator using the teaching member or a continuous operation performed by the operator in the air away from the display device.

3. The robot teaching system according to claim 2, wherein the detection unit interpolates welding operation information for displaying information regarding the welding operation of the robot based on the relative positional relationship acquired by the positional relationship acquisition unit while the operator is performing the continuous operation, and outputs the interpolated welding operation information to the image generation unit.

4. The robot teaching system according to claim 2, further comprising an interpolation unit that interpolates the shape of the motion trajectory corresponding to the continuous operation based on the relative positional relationship acquired by the positional relationship acquisition unit while the continuous operation is being performed by the operator.

5. The robot teaching system according to claim 4, wherein the interpolation unit interpolates the shape of the motion trajectory based on the start and end positions of the continuous operation and a plurality of intermediate positions between the start and end positions obtained by the continuous operation.

6. The robot teaching system according to claim 5, wherein the interpolation unit interpolates the shape of the motion trajectory based on the plurality of intermediate positions with respect to an auxiliary line connecting the start position and the end position obtained by the continuous operation.

7. The robot teaching system according to claim 6, wherein the interpolation unit determines that the shape of the motion trajectory is a straight line if the plurality of intermediate positions are within a predetermined distance in a direction perpendicular to the auxiliary line, and determines that the shape of the motion trajectory is an arc if the plurality of intermediate positions are not within the predetermined distance.

8. The robot teaching system according to claim 7, wherein the interpolation unit determines that the shape of the motion trajectory is weaving if the number of intermediate positions located outside the predetermined distance is greater than or equal to a predetermined number, and determines that the shape of the motion trajectory is an arc if the number of intermediate positions located outside the predetermined distance is not greater than or equal to the predetermined number.

9. The robot teaching system according to claim 4, wherein the image generation unit generates the display image for displaying the shape of the motion trajectory interpolated by the interpolation unit on the display device.

10. The robot teaching system according to claim 2, further comprising an interpolation unit that interpolates the operating speed of the welding operation of the robot based on the relative positional relationship acquired by the positional relationship acquisition unit while the continuous operation is being performed by the operator, and the elapsed time during which the continuous operation is being performed.

11. The robot teaching system according to claim 10, wherein the image generation unit generates the display image for displaying the operation speed interpolated by the interpolation unit on the display device.

12. The robot teaching system according to any one of claims 1 to 11, wherein the positional relationship acquisition unit acquires the relative positional relationship between the actual environment, the teaching member or the three-dimensional model, the display device, and the robot present in the actual environment.

13. The robot teaching system according to any one of claims 1 to 11, further comprising a timing unit for timing elapsed time, wherein the positional relationship acquisition unit acquires the relative positional relationship each time the elapsed time timing by the timing unit exceeds a predetermined time.

14. The robot teaching system according to any one of claims 1 to 11, wherein the position relationship acquisition unit acquires the relative position relationship again within a predetermined processing time after acquiring the relative position relationship.

15. A robot teaching system according to any one of claims 1 to 11, further comprising a measuring unit that measures the distance between the display device and the teaching member or the three-dimensional model based on the relative positional relationship acquired by the positional relationship acquisition unit, wherein the positional relationship acquisition unit acquires the relative positional relationship each time the distance measured by the measuring unit exceeds a predetermined distance.

16. The robot teaching system according to any one of claims 1 to 11, wherein the operation of the teaching member is a physical operation by the operator on an operating unit provided on the teaching member.

17. The robot teaching system according to any one of claims 1 to 11, wherein the teaching member has an operating section formed by the three-dimensional model that accepts operations from the teaching member or the operator, and the aerial operation is an operation performed by the operator in the air away from the display device on the teaching member displayed via the teaching member.

18. A robot teaching method performed by a system comprising: a display device configured to be wearable by an operator and which displays an image superimposed on an image of the real environment or the real environment itself; and a teaching member used for teaching a robot, the method comprising: acquiring the relative positional relationship between the real environment, the teaching member used for teaching the robot or a three-dimensional model corresponding to the teaching member, and the display device; detecting an operation of the teaching member performed by the operator using the teaching member or an aerial operation performed by the operator in the air away from the display device on the three-dimensional model displayed on the display device; acquiring and storing teaching point data corresponding to teaching points used for displaying information about the welding operation of the robot based on the relative positional relationship and the detected teaching member operation or aerial operation; generating a display image for displaying information about the welding operation; and outputting the display image to the display device.

Images