Teaching device and robot system
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- FANUC LTD
- Filing Date
- 2025-01-21
- Publication Date
- 2026-07-30
Smart Images
Figure JP2025001768_30072026_PF_FP_ABST
Abstract
Description
Teaching device and robot system
[0001] This disclosure relates to a teaching device and a robotic system.
[0002] A robot system is known that has the function of detecting an object based on image information acquired by a visual sensor. For example, Patent Documents 1 and 2 describe a robot system for inspecting an object by photographing it with a camera mounted on the robot and processing the captured image with an image processing device.
[0003] JP-A-2006-289531 JP-A-5-123989
[0004] In the robotic systems described above, it is necessary to adjust the position and orientation of the visual sensor or the position of the object so that the object or its features are near the center of the visual sensor's imaging range, in order to accurately detect the object. In such cases, adjustments are often made by gradually shifting the position of the visual sensor or the object while displaying the image captured by the visual sensor in real time on the display screen of the teaching control panel. However, this method has the problem that it is difficult for the user to understand which directions (up, down, left, and right) on the image correspond to in relation to the workspace or the robot. There is a need for technology that will allow the user to easily adjust the position of the object within the imaging range of the visual sensor.
[0005] One aspect of the present disclosure is a teaching device for teaching a robot, comprising: a coordinate system relationship calculation unit that calculates a relationship between the orientation of a visual sensor and the coordinate system based on setting information relating to a coordinate system and information relating to the position and orientation of a visual sensor; and a coordinate system display unit that displays information representing the first coordinate system on a display screen that displays an image captured by the visual sensor, based on the relationship calculated by the coordinate system relationship calculation unit with respect to a first coordinate system for operating the robot.
[0006] These and other objects, features, and advantages of the present invention will become even clearer from the detailed description of typical embodiments of the present invention shown in the accompanying drawings.
[0007] This is a diagram showing the overall configuration of a robot system according to one embodiment. This is a functional block diagram showing the functions of the robot control device. This is a diagram for explaining the coordinate system in the robot system. This is a diagram for explaining the coordinate system in the robot system. This is a diagram for explaining the coordinate system in the robot system. This is a flowchart showing the overall flow of the coordinate system display process. This is a diagram showing an example of a user interface screen for setting the display of the coordinate system. This is a diagram showing an example of a setting screen for executing a calibration program. This is a diagram showing the configuration of the robot system of the first embodiment regarding the adjustment of the position of an object in an captured image. This is a plan view from above of the vicinity of the placement position of the visual sensor in the workspace in the first embodiment. This is a diagram showing an example of the display of the coordinate system on the captured image in the first embodiment. This is a diagram showing the configuration of the robot system of the second embodiment regarding the adjustment of the position of an object in an captured image. This is a plan view from above of the vicinity of the placement position of the visual sensor in the workspace in the second embodiment. This is a diagram showing an example of the display of the coordinate system on the captured image in the second embodiment. This is a diagram showing the configuration of the robot system of the third embodiment regarding the adjustment of the position of an object in an captured image. This is a plan view from above of the vicinity of the placement position of the visual sensor in the workspace in the third embodiment. This figure shows an example of displaying a coordinate system on an captured image in the third embodiment.
[0008] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, similar components or functional parts are given the same reference numerals. For ease of understanding, the scale of these drawings has been appropriately changed. Furthermore, the embodiments shown in the drawings are just one example of how to carry out the present invention, and the present invention is not limited to the illustrated embodiments.
[0009] Figure 1 is a diagram showing the overall configuration of a robot system 100 according to one embodiment. The robot system 100 has a vision sensor 70 and has the function of detecting an object based on image information acquired by the vision sensor 70. A robot system 100 having such a vision detection function can be used as a system for performing various tasks such as inspection, detection, and handling of objects. In Figure 1, the vision sensor 70 is shown attached to the tip of the arm of the robot 30, but this is just an example, and the vision sensor 70 may be fixed in the workspace.
[0010] As shown in Figure 1, the robot system 100 includes a robot 30 with a hand 33 mounted on the tip of its arm, a robot control device 50 that controls the robot 30, a teaching control panel 10 connected to the robot control device 50, a vision sensor 70 attached to the tip of the arm of the robot 30, and an image processing device 20 that controls the vision sensor 70. The robot system 100 is configured to have the function of detecting an object 1 on the workbench 2 using the vision sensor 70. In this embodiment, the entire teaching function provided by the robot control device 50 and the teaching control panel 10 will also be referred to as the teaching device 40.
[0011] In such a robot system 100, in order for the visual sensor 70 to accurately detect the object 1, it is necessary to adjust the position and orientation of the visual sensor 70 or the position of the object 1 so that the object 1 or a feature of the object 1 is located near the center of the imaging range of the visual sensor 70. Such adjustments are made by displaying the captured image in real time on the display screen of the teaching operation panel 10, and adjusting the position and orientation of the visual sensor 70 or the position of the object 1 while confirming the position of the object 1 on the display screen. As will be described in detail below, the teaching device 40 and the robot system 100 according to this embodiment are configured to enable the user to make such adjustments to position the object 1 near the center of the imaging range with intuitive and easy operation.
[0012] In the configuration example shown in Figure 1, robot 30 is a vertical articulated robot, but various types of robots may be used depending on the task, such as a horizontal articulated robot, a parallel link robot, or a dual-arm robot. Robot 30 can perform desired tasks using an end effector attached to its wrist. The end effector is an external device that can be replaced depending on the application, such as a hand, welding gun, or tool. Figure 1 shows an example where a hand is used as the end effector.
[0013] The robot control device 50 controls the operation of the robot 30 according to an operation program or commands from the teaching control panel 10. The robot control device 50 may have a hardware configuration as a general computer, including a processor 51, memory (ROM, RAM, non-volatile memory, etc.), storage unit 52, operation unit, input / output interface, network interface, etc. (see Figure 2).
[0014] The teaching control panel 10 is connected to the robot control device 50. The teaching control panel 10 is used as an operating terminal for teaching (program creation) the robot 30's operation program and for making various other settings related to teaching. The teaching control panel 10 may be configured as a tablet terminal or the like. The teaching control panel 10 may have a hardware configuration as a general computer, including a processor, memory (ROM, RAM, non-volatile memory, etc.), storage unit, display unit 13, operation unit 14, input / output interface, network interface, etc. (see Figure 2). The display unit 13 may be configured as, for example, a liquid crystal display.
[0015] The image processing device 20 has the function of controlling the visual sensor 70 and performing image processing on the image captured by the visual sensor 70. The functions of the image processing device 20 include a function for detecting objects. The image processing device 20 may have, for example, a function for detecting objects using pattern matching or a learning model constructed to detect objects from an input image.
[0016] The image processing device 20 may be configured as a dedicated device, or it may be configured as a general-purpose information processing device such as a PC (personal computer), tablet terminal, or smartphone. The image processing device 20 may have a hardware configuration as a general computer, including a processor, memory (ROM, RAM, non-volatile memory, etc.), storage unit, input / output interface, network interface, display unit, operation unit, etc. Although Figure 1 shows an example configuration in which the image processing device 20 is provided as a separate device from the robot control device 50, the functions of the image processing device 20 may be integrated into the robot control device 50.
[0017] The visual sensor 70 may be a two-dimensional camera that acquires two-dimensional images, or it may be configured to have both the function of a two-dimensional camera and the function of a three-dimensional camera that can acquire three-dimensional position information (distance images, etc.). As the three-dimensional camera, a stereo camera or a TOF (Time of Flight) camera that acquires three-dimensional position information of an object using the optical time-of-flight method can be used.
[0018] Figure 2 is a block diagram showing the functions of the robot control device 50. As shown in Figure 2, the robot control device 50 includes an motion control unit 151, a coordinate system information setting unit 152, a vision sensor information setting unit 153, a coordinate system relationship calculation unit 154, a display setting unit 155, a coordinate system display unit 156, and an image rotation unit 157. These functional elements may also be realized by the robot control device's processor 51 executing software.
[0019] The storage unit 52 is a storage device consisting of, for example, a non-volatile memory or a hard disk drive. The storage unit 52 stores a robot program for controlling the robot 30, a program (vision program) for image processing such as workpiece detection based on images captured by the vision sensor 70, coordinate system information, calibration data, and various other setting information.
[0020] The motion control unit 151 controls the robot's movements according to the robot program or commands from the teaching control panel 10. The robot control device 50 includes a servo control unit (not shown) that performs servo control on the servo motors of each axis according to commands for each axis generated by the motion control unit 151.
[0021] The coordinate system information setting unit 152 provides a function for setting and saving information related to coordinate systems. The coordinate system information setting unit 152 may also have a function for accepting settings related to coordinate systems via a user interface. Here, with reference to Figures 3A to 3C, the coordinate systems set for the robot system 100 will be described. The coordinate systems set for the robot system 100 include the world coordinate system (sometimes called the robot coordinate system) fixed to the base of the robot 30, the tool coordinate system fixed to the tool of the robot 30, and the user coordinate system that can be freely set by the user in the workspace. Figure 3A shows the world coordinate system C fixed to the base of the robot 30. ０ This shows the world coordinate system C. ０ Information regarding this may be pre-configured and stored in the memory unit 52.
[0022] In Figure 3B, tool coordinate system C １０ An example is shown. Here, the tool coordinate system C １０ This shows an example where it is set at the tool tip. Tool coordinate system C １０ The position and orientation may be set by the functions of the coordinate system information setting unit 152. Note that the tool coordinate system C １０ Regarding the orientation, the orientation of the mechanical interface coordinate system, which is pre-set in the mechanical interface 31 at the tip of the arm, may be set as the default.
[0023] Figure 3C shows an example of a user coordinate system. Here, two user coordinate systems C are fixed within the workspace. ２１ , C ２２ An example of such a user coordinate system is shown. Such a user coordinate system may be fixed to a workbench, to a workpiece, or to the workspace.
[0024] In a robot system, it is common for the world coordinate system (robot coordinate system) to be set in advance. The user, for example, sets a user coordinate system and a tool coordinate system, and selects one coordinate system for operating the robot from these coordinate systems. The robot operates in accordance with the selected coordinate system for operation. Note that the user may select the world coordinate system for operating the robot. The coordinate system information setting unit 152 may have a function of accepting the selection of the coordinate system for operating the robot.
[0025] The visual sensor information setting unit 153 provides a function for setting information regarding the position (position and orientation) of the visual sensor 70. The visual sensor information setting unit 153 may have a function of performing calibration using a calibration jig such as a dot pattern. By calibration, for example, calibration data indicating the position (position and orientation) of the visual sensor 70 with respect to the world coordinate system C ０ can be obtained. Alternatively, the visual sensor information setting unit 153 may have a function of accepting the setting of numerical information representing the position and orientation of the visual sensor with respect to a reference coordinate system such as the world coordinate system. The visual sensor information setting unit 153 stores information regarding the position (position and orientation) of the visual sensor 70 set as described above, for example, in the storage unit 52.
[0026] The coordinate system relationship calculation unit 154 has a function of calculating the relationship between the coordinate system in the robot system 100 and the orientation of the visual sensor 70 (the orientation of the optical axis of the visual sensor 70) based on the setting information regarding the coordinate system in the robot system 100 and the information regarding the position (position and orientation) of the visual sensor 70 set via the function of the visual sensor information setting unit 153.
[0027] The display setting unit 155 provides a function for performing various settings regarding the display of the coordinate system on the display screen. The display setting unit 155 may be configured to accept user input for performing settings regarding the display of the coordinate system on the display screen.
[0028] The coordinate system display unit 156 has a function of displaying information representing the coordinate system on the display screen on which the image captured by the visual sensor 70 is displayed, based on the information representing the relationship between the coordinate system and the orientation of the visual sensor 70 calculated by the coordinate system relationship calculation unit 154 and the setting information set via the display setting unit 155. In the present embodiment, the image captured by the visual sensor 70 is displayed on the display unit 13 of the teaching operation panel 10, and the coordinate system display unit 156 displays information regarding the coordinate system on or around the captured image displayed on the display unit 13.
[0029] The image rotation unit 157 provides a function of rotating the captured image on the display unit 13. The image rotation unit 157 may have a function of automatically rotating the image so that a specific direction based on the coordinate system displayed on the display unit 13 always has a constant orientation with respect to the display screen of the display unit 13. Alternatively, the image rotation unit 157 may have a function of rotating the captured image on the display unit 13 according to a user operation.
[0030] FIG. 4 is a flowchart showing the overall flow of a process for displaying information regarding the coordinate system on the image captured by the visual sensor 70 (hereinafter also referred to as the coordinate system display process). The coordinate system display process in FIG. 4 is mainly executed under the control of the processor 51 of the robot control device 50.
[0031] When this process is started, an interface screen for receiving user input is displayed on the display unit 13, and various settings such as setting of the coordinate system by the user can be received (step S1). Note that this user interface screen is provided by the cooperation of the coordinate system information setting unit 152, the visual sensor information setting unit 153, the coordinate system relationship calculation unit 154, the display setting unit 155, and the image rotation unit 157. FIG. 5 shows an example of the user interface screen (hereinafter referred to as the setting screen 200) presented here.
[0032] As shown in Figure 5, the settings screen 200 includes a setting button 211 for setting the coordinate system and the position (position and orientation) of the visual sensor, and a button 212 for switching whether or not to enable the display of the coordinate system on the display screen. The settings screen 200 further includes a checkbox 213 for specifying that the image should be rotated so that a specific axis of the displayed coordinate system (in Figure 5, an example of the Y-axis is shown) is always facing upwards on the display screen, a checkbox 214 for specifying that the user coordinate system should be displayed, and a checkbox 215 for specifying that the tool coordinate system should be displayed. The settings screen 200 also includes a specification field 221 for specifying the number of the user coordinate system to be displayed, and a specification field 222 for specifying the number of the tool coordinate system to be displayed. Note that the configuration of the settings screen 200 shown here is just one example, and the settings screen 200 may be configured to allow the specification of other types of coordinate systems (for example, the world coordinate system) as the coordinate system to be displayed on the captured image.
[0033] When the setting button 211 on the setting screen 200 is pressed, a setting screen for setting the coordinate system and a setting screen for setting the position (position and orientation) of the vision sensor are displayed. The user can set the coordinate system via the setting screen for setting the coordinate system, and can also set the position (position and orientation) of the vision sensor via the setting screen for setting the position (position and orientation) of the vision sensor.
[0034] The user's coordinate system settings are accepted via the coordinate system settings screen, which is displayed by pressing the setting button 211 (step S2). In this case, the settings screen is provided by the functions of the coordinate system information setting unit 152. The user can set the position and orientation of the tool coordinate system, user coordinate system, etc., via the settings screen for setting the coordinate system. The information regarding the coordinate system set here is stored, for example, in the storage unit 52.
[0035] Next, in step S3, the position (position and orientation) of the visual sensor is accepted by the function of the visual sensor information setting unit 153. The user can set the position (position and orientation) of the visual sensor via a setting screen that is displayed by pressing the setting button 211. In this case, the setting screen may be a setting screen for executing a calibration program for the visual sensor, or it may be a setting screen for setting the position (position and orientation) of the visual sensor by numerical input.
[0036] Figure 6 shows an example of a settings screen 450 for executing a calibration program. The user places the calibration jig J in a predetermined position where it can be imaged by the visual sensor, makes the necessary settings on the settings screen 450, and executes the calibration. The calibration jig J is, for example, a jig with a dot pattern formed on it, including large dots M (only some of which are labeled) for defining a coordinate system (X axis, Y axis, origin O). The settings screen 450 includes an image G5 of the calibration jig J captured by the visual sensor 70 and an input field 451 for specifying the dot spacing. By executing the calibration, calibration data is obtained that includes external parameters corresponding to the transformation from the world coordinate system to the camera coordinate system (rotation and translation) and internal parameters corresponding to the transformation from the camera coordinate system to the image coordinate system.
[0037] If the settings screen for setting the position (position and orientation) of the vision sensor is a settings screen for setting the position and orientation of the vision sensor by numerical input, the user will, for example, set the position and orientation of the vision sensor relative to the world coordinate system (camera coordinate system) by numerical input.
[0038] Information regarding the position (position and orientation) of the visual sensor, as set above, is stored, for example, in the memory unit 52.
[0039] Next, in step S4, the image captured by the visual sensor 70 is displayed on the display unit 13. The function of step S4 may also be performed by the operation control unit 151 sending a command to the image processing device 20. Figure 5 shows the state in which the image G captured by the visual sensor 70 is displayed on the display unit 13.
[0040] Next, in step S5, the display setting unit 155 accepts settings related to the coordinate system display. Here, the display setting unit 155 accepts operations on the button 212, checkboxes 213, 214, and 215, and specification fields 221 and 222 on the setting screen 200. The user can switch the display of the coordinate system on the display screen by operating button 212. The user can also specify the type of coordinate system to be displayed on the captured image by operating checkboxes 214 and 215. The user can also display multiple types of coordinate systems (in this case, the tool coordinate system and the user coordinate system) on the captured image by checking both checkboxes 214 and 215. The user can specify the number of the user coordinate system to be displayed in specification field 221, and the number of the tool coordinate system to be displayed in specification field 222. The setting information related to the coordinate system display set here is used for display processing by the coordinate system display unit 156.
[0041] In this example, the display setting unit 155 accepts an operation in which the user sets the coordinate system to be displayed on the captured image via the setting screen 200. However, as will be explained in detail in the embodiments described later, the display setting unit 155 may also have a function to automatically set the coordinate system to be displayed on the captured image.
[0042] When checkbox 213 is checked, the image is rotated so that a specific axis of the coordinate system displayed on the screen always points upwards on the screen. The settings screen 200 may also have further settings that allow you to set the type of coordinate system to use as a reference, the type of axis to use as a reference, etc., when rotating the image.
[0043] In this way, by rotating the image so that a specific axis of a specific coordinate system points upward on the display screen, the user can easily grasp the relationship between the orientation of the captured image and the orientation of the workspace. The coordinate system display unit 156 may be configured so that the setting "rotate the image so that a specific axis of a specific coordinate system points upward on the display screen" is effective only if the orientation of the visual sensor (orientation of the optical axis) is perpendicular to the specific axis of the coordinate system.
[0044] The image rotation unit 157 may present a button 230 (see Figure 5) on the captured image G that allows the user to manually rotate the image. The button 230 may function to rotate the image according to the contents defined in the checkbox 213 when pressed, or it may function to rotate the image in fixed angular units (for example, 90 degrees).
[0045] Next, in step S6, the coordinate system relationship calculation unit 154 calculates the positional relationship between the orientation of the visual sensor 70 (i.e., the camera coordinate system) and other coordinate systems registered in the robot control device 50. Here, the coordinate system relationship calculation unit 154 may calculate the relationship between all coordinate systems registered in the robot control device 50 and the orientation of the visual sensor, or it may calculate the relationship between the coordinate system that is manually or automatically set to be displayed on the display screen and the orientation of the visual sensor. Here, the coordinate system relationship calculation unit 154 may calculate, for example, a transformation matrix representing the rotation and translation of other coordinate systems based on the camera coordinate system as the above relationship.
[0046] Next, in step S7, the coordinate system display unit 156 displays the coordinate system to be displayed, which was set based on the above relationship calculated by the coordinate system relationship calculation unit 154, on or around the image captured by the visual sensor 70. Figure 5 shows the "user coordinate system 0" (symbol C) specified in the designation field 221, on the captured image G. ２０ This shows an example where (indicated by) is displayed.
[0047] While observing the captured image G displayed in this manner, the user adjusts the position so that the object 1 is near the center of the imaging range of the visual sensor 70.
[0048] Furthermore, the user can configure the coordinate system to be displayed via the settings screen 200 at any time they wish. That is, as shown by the dashed arrows in the flowchart of Figure 4, the user can perform the processes from steps S5 to S7 at any time they wish as needed.
[0049] Note that the configuration of the coordinate system display process and the execution order of each step shown in Figure 4 are illustrative examples, and there are various possible variations in the coordinate system display process. For example, the setting of the coordinate system and the setting of the position and orientation of the visual sensor may be performed in advance before presenting the user interface screen (setting screen 200) as illustrated in Figure 5.
[0050] The following describes three embodiments relating to the display modes of the coordinate system set via the coordinate system display processing described above and displayed on the display screen. In the three embodiments described here, the configuration of the robot system differs depending on whether the visual sensor 70 is fixed in the workspace or mounted on the robot 30, and the objects that the user adjusts (position of the object, position or orientation of the visual sensor, position or orientation of the robot) differ depending on the difference in the configuration of the robot system.
[0051] The configuration of the robot system according to the first embodiment and the display of the coordinate system on the captured image will be described with reference to Figures 7A to 7C. As shown in Figure 7A, in the robot system 100 according to the first embodiment, the vision sensor 70 is a fixed camera fixed in the workspace. The vision sensor 70 is positioned to capture an image of the object 1 on the workbench 2.
[0052] Figure 7A shows the world coordinate system C set at the base of the robot 30. ０ Figure 7A shows the robot 30, vision sensor 70, workbench 2, object 1, etc. in world coordinate system C. ０This shows a side view along the Y-axis. Figure 7B shows a plan view of the area near the placement of the visual sensor 70 in the workspace, viewed from above (i.e., viewed from the direction of arrow A in Figure 7A). For the sake of explanation, Figure 7B shows only the object 1, the visual sensor 70, and the current imaging range R of the visual sensor 70. Note that Figure 7B is shown in world coordinate system C to show the positional relationship with Figure 7A. ０ This is filled in. World coordinate system C shown in Figures 7A and 7B. ０ As can be understood from this, the front side of Figure 7A corresponds to the bottom side of Figure 7B, and the back side of Figure 7A corresponds to the top side of Figure 7B. As shown in Figure 7B, the lower surface of the visual sensor 70 in Figure 7B is defined as the front surface 70a of the visual sensor, and the direction indicated by arrow 71 is defined as the forward direction of the visual sensor 70. Therefore, when the object 1 is imaged by the visual sensor 70 in the arrangement shown in the plan view of Figure 7B, the object 1 will appear at the bottom of the captured image.
[0053] Figure 7C shows image G1, which is captured by the visual sensor 70 in the plan view shown in Figure 7B and displayed on the display unit 13. As described above, in image G1, the object is shown at the bottom.
[0054] In the system configuration of the first embodiment shown in Figure 7A, the user adjusts the object to move it near the center of the captured image by either: - manually moving the object to adjust its position in the image, or - manually moving the visual sensor to adjust its position in the image, or both of the above. As shown in Figure 7C, one or more coordinate systems that are set to be displayed are shown in image G1. In Figure 7C, as an example, the world coordinate system C is shown in image G1. ０ This indicates the situation where the following is displayed.
[0055] By displaying the coordinate system within the captured image in this way, users can intuitively grasp the correspondence between directions in the workspace and directions in the image, and adjust the position of objects within the image intuitively and easily.
[0056] In the system configuration shown in Figure 7A, the user adjusts the position of the object 1 or the visual sensor 70 by moving their body within the workspace. Therefore, it is preferable that the coordinate system displayed on the image is a coordinate system fixed to the workspace. In the system configuration described in Figure 7A, the coordinate systems fixed to the workspace may include the user coordinate system, the camera coordinate system, and the world coordinate system. Among these coordinate systems, one preferred example is to select the coordinate system set closer to the object as the coordinate system to be displayed on the screen. On the other hand, the world coordinate system C ０ Since this can be said to represent the robot's orientation, it is a convenient coordinate system for users to understand their direction within the workspace. Therefore, there are advantages to selecting the world coordinate system as the coordinate system to display on the screen.
[0057] As shown in Figure 7C, one preferred method for displaying the coordinate system within the captured image is to use a fixed position such as the upper right corner or upper left corner of the screen.
[0058] The configuration of the robot system according to the second embodiment and the display of the coordinate system on the captured image will be described with reference to Figures 8A to 8C. As shown in Figure 8A, in the robot system 100 according to the second embodiment, the vision sensor 70 is a fixed camera fixed in the workspace. The vision sensor 70 is positioned to capture an image of the object 1 on the workbench 2. In the second embodiment, the user operates the robot 30 to grasp the object 1 with the hand 33 and adjust the position of the object 1 in the image captured by the vision sensor 70.
[0059] Figure 8A shows the world coordinate system C set at the base of the robot 30. ０ Figure 8A shows the robot 30, vision sensor 70, workbench 2, object 1, etc. in world coordinate system C. ０This shows a side view along the Y-axis. Figure 8B shows a plan view of the area near the placement of the visual sensor 70 in the workspace, viewed from above (i.e., viewed from the direction of arrow A in Figure 8A). For the sake of explanation, Figure 8B shows only the tip (hand 33) of the robot 30 that is gripping the object 1, the visual sensor 70, and the imaging range R of the visual sensor 70. Note that Figure 8B is set to the world coordinate system C to show the positional relationship with Figure 8A. ０ This is filled in. World coordinate system C shown in Figures 8A and 8B. ０ As can be seen, the right side of Figure 8A corresponds to the lower side of Figure 8B, and the left side of Figure 8A corresponds to the upper side of Figure 8B. As shown in Figure 8B, the lower surface of the visual sensor 70 in Figure 8B is defined as the front surface 70a of the visual sensor, and the direction indicated by arrow 71 is defined as the forward direction of the visual sensor 70. Therefore, when the object 1 is imaged by the visual sensor 70 in the arrangement shown in the plan view of Figure 8B, the tip of the robot 30 that grasps the object 1 (hand 33) will appear at the bottom of the captured image.
[0060] Figure 8C shows image G2, which is captured by the visual sensor 70 in the plan view shown in Figure 8B and displayed on the display unit 13. As described above, in image G2, the tip of the robot 30 that grasps the object 1 is shown on the lower side.
[0061] In the system configuration of the second embodiment shown in Figure 8A, the user operates the robot 30 to move the object 1 and adjust the position of the object in the image. As shown in Figure 8C, one or more coordinate systems that are set to be displayed are shown in image G2. In Figure 8C, as an example, tool coordinate system C is shown in image G2. １０ And, user coordinate system C ２１ This indicates the situation where the following is displayed.
[0062] By displaying the coordinate system within the captured image in this way, users can intuitively grasp the correspondence between directions in the workspace and directions in the image, and adjust the position of objects within the image intuitively and easily.
[0063] In a system configuration like the one shown in Figure 8A, the user will operate the robot 30 to move the object 1, so it is preferable to display the coordinate system selected for robot operation on image G2. For example, tool coordinate system C for robot operation. １０ If selected, the tool coordinate system C will be placed on image G1 as shown in Figure 8C. １０ By displaying this, the user can intuitively move the hand 33 (object 1) within image G2. The tool coordinate system C is used as the coordinate system for operating the robot. １０ If selected, the jog operation keys (direction keys) of the teaching control panel 10 and the tool coordinate system C １０ Since the directions of each coordinate axis are aligned, this also has the advantage of making adjustments easier for the user.
[0064] Furthermore, as shown in Figure 8C, it is a desirable practice to display both a coordinate system that changes according to the posture of the robot 30 (such as the tool coordinate system) and a coordinate system fixed in the workspace (such as the user coordinate system or world coordinate system) on the image G2. In Figure 8C, the tool coordinate system C is shown as the coordinate system that changes according to the posture of the robot 30. １０ The user coordinate system C is displayed as the coordinate system fixed in the workspace. ２１ This shows an example where it is displayed. Note that the user coordinate system C ２１ These are, for example, a coordinate system fixed to the workbench 2 and a coordinate system fixed within the workspace. By displaying both the coordinate system that changes in accordance with the change in the posture of the robot 30 and the coordinate system fixed in the workspace within the captured image, both intuitive understanding of the direction within the workspace and the direction within the image, and intuitive operation of the robot within the image are promoted, enabling efficient adjustment work. In addition, when configuring the display screen to show multiple types of coordinate systems as in this example, if the relationship with the orientation of the visual sensor has been calculated for at least one of the multiple types of coordinate systems, it is also possible to display multiple types of coordinate systems on the display screen using the known relationships between the multiple types of coordinate systems.
[0065] In the second embodiment, the display setting unit 155 may operate to automatically select the coordinate system selected for robot operation (e.g., the tool coordinate system) as the coordinate system to be displayed on the display screen. Alternatively, if the coordinate system selected for robot operation (e.g., the tool coordinate system) is set based on a certain coordinate system (e.g., a user coordinate system fixed in the workspace), the display setting unit 155 may operate to automatically select these two coordinate systems as the coordinate systems to be displayed on the display screen.
[0066] Therefore, in the second embodiment, the coordinate system to be displayed on the display screen can be automatically set in conjunction with the coordinate system selected for robot operation, or one or more coordinate systems that serve as the basis for robot operation, and the coordinate system can be displayed on the display screen. In other words, with the above configuration, the coordinate system can be displayed on the display screen in conjunction with the coordinate system for robot operation. This further improves the convenience for the user when adjusting the position of an object in the captured image.
[0067] (Third Embodiment) The configuration of the robot system according to the third embodiment and the display of the coordinate system on the captured image will be described with reference to Figures 9A to 9C. As shown in Figure 9A, in the robot system 100 according to the third embodiment, the vision sensor 70 is attached to the movable part (tip of the arm) of the robot 30.
[0068] Figure 9A shows the tool coordinate system C set in the tool section of the robot 30. １２ Figure 9A illustrates the robot 30, vision sensor 70, workbench 2, object 1, etc., in the tool coordinate system C in the posture of the robot 30 in Figure 9A. １２ This shows a side view along the Y-axis. Figure 9B shows a plan view of the area near the placement of the visual sensor 70 in the workspace, viewed from above (i.e., from the direction of arrow A in Figure 9A). For the sake of explanation, Figure 9B only shows the object 1, the visual sensor 70, and the imaging range R of the visual sensor 70. Note that in Figure 9B, the tool coordinate system C is shown to indicate the positional relationship with Figure 9A. １２ This is filled in. Tool coordinate system C shown in Figures 9A and 9B. １２As can be understood from this, the right side of Figure 9A corresponds to the right side of Figure 9B, and the left side of Figure 9A corresponds to the left side of Figure 9B. As shown in Figure 9B, the right side of the visual sensor 70 in Figure 9B is defined as the front surface 70a of the visual sensor, and the direction indicated by arrow 71 is defined as the forward direction of the visual sensor 70. Therefore, when the object 1 is imaged by the visual sensor 70 in the arrangement shown in the plan view of Figure 9B, the object 1 will appear on the lower left side of the captured image.
[0069] Figure 9C shows image G3, which is captured by the visual sensor 70 in the plan view shown in Figure 9B and displayed on the display unit 13. As described above, in image G3, the object 1 is captured on the lower left side.
[0070] In the system configuration of the third embodiment shown in Figure 9A, the user adjusts the position of the object in the image by either: moving the object by hand, or operating the robot to move the visual sensor and adjust the position of the object in the image. As shown in Figure 9C, one or more coordinate systems that are set to be displayed are shown in image G3. In Figure 9C, as an example, tool coordinate system C is shown in image G3. １２ This indicates the situation where the following is displayed.
[0071] By displaying the coordinate system within the captured image in this way, users can intuitively grasp the correspondence between directions in the workspace and directions in the image, and adjust the position of objects within the image intuitively and easily.
[0072] In the system configuration of the third embodiment, consider the case where the user moves object 1 by hand to adjust its position in image G3. In this case, the robot 30 is stationary, and therefore the tool coordinate system C in this case... １２ This represents a fixed coordinate system within the workspace. Therefore, in this case, the coordinate system fixed in the workspace is displayed within image G3, allowing the user to intuitively and easily adjust the position of object 1 within image G3.
[0073] Next, consider the case in the system configuration of the third embodiment in which the user adjusts the position of object 1 in the image by operating the robot 30 to move the vision sensor 70. In this case, within image G3, there is a tool coordinate system C, which is a coordinate system whose orientation changes along with the orientation of the robot 30 (tool unit). １２ As a result, the user can intuitively and easily adjust the position of object 1 within image G3. In this case as well, as described above for the second embodiment, multiple types of coordinate systems, such as the coordinate system selected for robot operation and the coordinate system fixed to the workspace, may be displayed on the display screen. Furthermore, the display of coordinate systems on the display screen may be linked to the coordinate system selected for robot operation, or to one or more coordinate systems that serve as the basis for robot operation. Therefore, according to the above configuration, coordinate systems can be displayed on the display screen in conjunction with the coordinate system for robot operation.
[0074] As described above, according to this embodiment, the user can easily adjust the position of the object within the imaging range of the visual sensor.
[0075] Furthermore, according to this embodiment, users can intuitively grasp the correspondence between directions in the workspace and directions in the image, thus preventing them from moving objects such as robots, vision sensors, and other objects in unintended directions, and thus improving safety.
[0076] Furthermore, displaying information on multiple coordinate systems on the screen when adjusting the position of an object within the captured image offers the following advantages. For example, the captured image can display both a coordinate system fixed to the workspace (world coordinate system or user coordinate system) and a coordinate system that changes according to the robot's posture (tool coordinate system). In this case, it is possible to verify whether the coordinate axes of the world coordinate system or user coordinate system and the coordinate axes of the tool coordinate system are in the intended relationship (e.g., perpendicular relationship). Alternatively, if the captured image displays both a user coordinate system fixed to the surface on which the object is placed and a camera coordinate system set for the vision sensor, it is possible to verify whether the vision sensor is pointing straight at the surface on which the object is placed. By displaying information on multiple coordinate systems in this way, it is possible to verify whether the coordinate system or the object is positioned as intended.
[0077] Although the above embodiments mainly describe configurations in which coordinate system information is superimposed on the captured image, the above-mentioned benefits to the user can be provided if the coordinate system information is displayed on the display screen of the display unit together with the captured image, such as around the captured image.
[0078] The coordinate system display process described above can also be broadly described as a process that calculates the relationship between the orientation of the visual sensor and the coordinate system based on setting information about the coordinate system and information about the position and orientation of the visual sensor, and then displays information representing the first coordinate system on the display screen where the image captured by the visual sensor is displayed, based on the relationship calculated for the first coordinate system for robot operation.
[0079] The functional arrangement in the functional block diagram (Figure 2) described above is illustrative, and there are various possible configurations for the functional arrangement to realize the functions of the teaching device 40. For example, there may be configurations in which some of the functions located in the robot control device 50 in the functional block diagram of Figure 2 are located in the teaching operation panel 10. For example, the display setting unit 155, the coordinate system display unit 156, and the image rotation unit 157 in the functional block diagram of Figure 2 may be located in the teaching operation panel 10.
[0080] In the functional block diagram of Figure 2, each functional block described as a function of the robot control device or teaching control panel may be realized by one or more processors of these devices executing various software stored in a memory device, or in this case, part of the function may be made up of hardware such as discrete circuits (i.e., the functional block may be realized by a combination of a processor and discrete circuits), or the functions shown in the functional block diagram may be realized by a hardware-based configuration such as an ASIC (Application Specific Integrated Circuit).
[0081] The program that performs various processes such as coordinate system display processing in the above-described embodiment, or the computer program for performing processes in each part of at least one processor of the robot control device or teaching control panel, may be provided in the form of a program product recorded on various computer-readable recording media (for example, semiconductor memory such as ROM, EEPROM, flash memory, magnetic recording media, or optical recording media such as CD-ROM, DVD-ROM).
[0082] While this disclosure has been described in detail, it is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the gist of this disclosure or from the spirit of this disclosure derived from the claims and their equivalents. Furthermore, these embodiments can be implemented in combination. For example, the order of operations and processes in the embodiments described above are given as examples only and are not limited thereto. The same applies when numerical values or mathematical formulas are used in the description of the embodiments described above.
[0083] The following further notes apply to the above embodiments and modifications. (Note 1) A teaching device (40) for teaching a robot, comprising: a coordinate system relationship calculation unit (154) that calculates a relationship between the orientation of a visual sensor and the coordinate system based on setting information relating to a coordinate system and information relating to the position and orientation of a visual sensor; and a coordinate system display unit (156) that displays information representing the first coordinate system on a display screen where an image captured by the visual sensor is displayed, based on the relationship calculated by the coordinate system relationship calculation unit with respect to a first coordinate system for operating the robot. (Note 2) The teaching device (40) according to Note 1, wherein the coordinate system display unit (156) further displays information representing the second coordinate system on the display screen based on the relationship calculated by the coordinate system relationship calculation unit with respect to a second coordinate system of a different type from the first coordinate system. (Note 3) The teaching device (40) according to Note 2, wherein one of the first coordinate system and the second coordinate system is a coordinate system whose orientation changes according to the posture of the robot, and the other of the first coordinate system and the second coordinate system is a coordinate system fixed in the workspace. (Note 4) The teaching device (40) according to any one of Notes 1 to 3, further comprising an image rotation unit (157) for rotating the image displayed on the display screen. (Note 5) The teaching device (40) according to Note 4, wherein the image rotation unit (157) rotates the image such that a specific direction based on the coordinate system displayed on the display screen is always a constant orientation relative to the display screen. (Note 6) The teaching device (40) according to Note 4, wherein the image rotation unit (157) rotates the image on the display screen in response to user operation. (Note 7) The teaching device (40) according to any one of Notes 1 to 6, wherein the vision sensor is fixed in the workspace. (Note 8) The teaching device (40) according to any one of Notes 1 to 6, wherein the visual sensor is attached to a movable part of the robot.(Note 9) A teaching device (40) according to any one of Notes 1 to 8, further comprising: a coordinate system information setting unit (152) that receives input of the setting information relating to the coordinate system; and a display setting unit (155) that automatically selects the first coordinate system to be displayed on the display screen from among the coordinate systems set via the coordinate system information setting unit. (Note 10) A teaching device (40) according to any one of Notes 1 to 9, wherein the first coordinate system is a coordinate system selected for the operation of the robot. (Note 11) A teaching device (40) for teaching a robot, comprising: a coordinate system relationship calculation unit (154) that calculates a relationship between the orientation of the visual sensor and the coordinate system based on setting information relating to the coordinate system and information relating to the position and orientation of the visual sensor; and a coordinate system display unit (156) that displays information representing the plurality of types of coordinate systems on a display screen on which an image captured by the visual sensor is displayed, based on the relationship calculated by the coordinate system relationship calculation unit for at least one of the plurality of types of coordinate systems. (Note 12) A robot system (100) comprising a vision sensor (70), a robot (30), and a teaching device (40) for teaching the robot, wherein the teaching device (40) comprises a coordinate system relationship calculation unit (154) that calculates the relationship between the orientation of the vision sensor and the coordinate system based on setting information relating to the coordinate system and information relating to the position and orientation of the vision sensor, and a coordinate system display unit (156) that displays information representing the first coordinate system on a display screen on which an image captured by the vision sensor is displayed, based on the relationship calculated by the coordinate system relationship calculation unit with respect to a first coordinate system for operating the robot.(Note 13) A robot system (100) comprising a visual sensor (70), a robot (30), and a teaching device (40) for teaching the robot, wherein the teaching device (40) comprises a coordinate system relationship calculation unit (154) that calculates a relationship between the orientation of the visual sensor and the coordinate system based on setting information relating to the coordinate system and information relating to the position and orientation of the visual sensor, and a coordinate system display unit (156) that displays information representing the plurality of coordinate systems on a display screen on which an image captured by the visual sensor is displayed, based on the relationship calculated by the coordinate system relationship calculation unit for at least one of the plurality of coordinate systems.
[0084] 1 Object 2 Workbench 10 Teaching control panel 13 Display unit 14 Operation unit 20 Image processing device 30 Robot 33 Hand 40 Teaching device 50 Robot control device 51 Processor 52 Memory unit 70 Visual sensor 100 Robot system 151 Motion control unit 152 Coordinate system information setting unit 153 Visual sensor information setting unit 154 Coordinate system relationship calculation unit 155 Display setting unit 156 Coordinate system display unit 157 Image rotation unit
Claims
1. A teaching device for teaching a robot, comprising: a coordinate system relationship calculation unit that calculates a relationship between the orientation of a visual sensor and the coordinate system based on setting information relating to a coordinate system and information relating to the position and orientation of a visual sensor; and a coordinate system display unit that displays information representing the first coordinate system on a display screen that displays an image captured by the visual sensor, based on the relationship calculated by the coordinate system relationship calculation unit with respect to a first coordinate system for operating the robot.
2. The teaching device according to claim 1, wherein the coordinate system display unit displays further information representing the second coordinate system on the display screen based on the relationship calculated by the coordinate system relationship calculation unit with respect to a second coordinate system of a different type from the first coordinate system.
3. The teaching device according to claim 2, wherein one of the first coordinate system and the second coordinate system is a coordinate system whose orientation changes according to the orientation of the robot, and the other of the first coordinate system and the second coordinate system is a coordinate system fixed in the workspace.
4. The teaching device according to any one of claims 1 to 3, further comprising an image rotation unit for rotating the image displayed on the display screen.
5. The teaching device according to claim 4, wherein the image rotation unit rotates the image such that a specific direction based on the coordinate system displayed on the display screen is always oriented in a constant direction relative to the display screen.
6. The teaching device according to claim 4, wherein the image rotation unit rotates the image on the display screen in response to user operation.
7. The teaching device according to any one of claims 1 to 6, wherein the visual sensor is fixed in the workspace.
8. The teaching device according to any one of claims 1 to 6, wherein the visual sensor is attached to a movable part of the robot.
9. The teaching device according to any one of claims 1 to 8, further comprising: a coordinate system information setting unit that receives input of setting information relating to a coordinate system; and a display setting unit that automatically selects the first coordinate system to be displayed on the display screen from among the coordinate systems set via the coordinate system information setting unit.
10. The teaching device according to any one of claims 1 to 9, wherein the first coordinate system is a coordinate system selected for the operation of the robot.
11. A teaching device for teaching a robot, comprising: a coordinate system relationship calculation unit that calculates a relationship between the orientation of a visual sensor and the coordinate system based on setting information relating to a coordinate system and information relating to the position and orientation of a visual sensor; and a coordinate system display unit that displays information representing the plurality of coordinate systems on a display screen that displays an image captured by the visual sensor, based on the relationship calculated by the coordinate system relationship calculation unit for at least one of a plurality of coordinate systems.
12. A robot system comprising: a vision sensor; a robot; and a teaching device for teaching the robot, wherein the teaching device includes: a coordinate system relationship calculation unit that calculates a relationship between the orientation of the vision sensor and the coordinate system based on setting information relating to a coordinate system and information relating to the position and orientation of the vision sensor; and a coordinate system display unit that displays information representing the first coordinate system on a display screen on which an image captured by the vision sensor is displayed, based on the relationship calculated by the coordinate system relationship calculation unit with respect to a first coordinate system for operating the robot.
13. A robot system comprising: a vision sensor; a robot; and a teaching device for teaching the robot, wherein the teaching device includes: a coordinate system relationship calculation unit that calculates a relationship between the orientation of the vision sensor and the coordinate system based on setting information relating to a coordinate system and information relating to the position and orientation of the vision sensor; and a coordinate system display unit that displays information representing the plurality of coordinate systems on a display screen on which an image captured by the vision sensor is displayed, based on the relationship calculated by the coordinate system relationship calculation unit for at least one of a plurality of coordinate systems.