Teaching device

Abstract

This teaching device comprises: an operation history recording unit that records an operation history of a user related to a setting parameter for image processing and a processing result of the image processing for a plurality of times of execution of the image processing; and an operation history display unit that, on the basis of the recorded operation history and processing result, performs display so that the relationship between adjustment of the setting parameter by the user and the processing result can be visually recognized.

Description

Teaching device

[0001] The present disclosure relates to a teaching device.

[0002] Vision systems are known for automating the handling of work by a robot or the inspection of articles on a production line by using a vision sensor. In this regard, Patent Document 1 describes a robot system that controls the operation of a robot arm based on an image captured by an imaging device. Patent Document 2 describes a robot system that automatically inspects the appearance of an inspection object having a three-dimensional shape.

[0003] Japanese Patent Application Laid-Open No. 2016-001768, Japanese Patent Application Laid-Open No. 07-098217

[0004] In order to execute image processing in the vision system as described above, adjustment of various parameters is required. Generally, adjustment of image processing parameters requires advanced knowledge and skills, and proficiency is required to be able to appropriately adjust the parameters. There is a need for a teaching device that enables a user to easily adjust the parameters for appropriately executing image processing.

[0005] One aspect of the present disclosure includes an operation history recording unit that records the operation history of a user regarding the setting parameters of image processing and the processing result of the image processing for a plurality of executions of the image processing, and an operation history display unit that performs display so that the relationship between the adjustment of the setting parameters by the user and the processing result can be visually recognized based on the recorded operation history and the processing result. The teaching device is provided.

[0006] From the detailed description of the typical embodiments of the present invention shown in the accompanying drawings, these objects, features, and advantages of the present invention, as well as other objects, features, and advantages, will become more apparent.

[0007] This is a diagram showing the equipment configuration of a robot system including an image processing device according to the first embodiment. This is a functional block diagram showing the functional configuration of the image processing device and robot control device according to the first embodiment. This is a diagram showing an example of a setting screen related to the visual detection function. This is a diagram showing an example of a display mode for graphing the operation history. This is a flowchart showing the operation history recording process in the first embodiment. This is a flowchart showing the operation history display process by the operation history display unit in the first embodiment. This is a flowchart showing the operation history display process when the function by the adjustment parameter determination unit is additionally applied. This is a functional block diagram showing the functional configuration of the image processing device and robot control device according to the second embodiment. This is a diagram showing an example of a display mode for suggesting parameters to be adjusted in the second embodiment. This is a flowchart showing the adjustment parameter determination process executed by the adjustment parameter determination unit in the second embodiment.

[0008] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, similar components or functional parts are given the same reference numerals. For ease of understanding, the scale of these drawings has been appropriately changed. Furthermore, the embodiments shown in the drawings are just one example of how to carry out the present invention, and the present invention is not limited to the illustrated embodiments.

[0009] This embodiment relates to a teaching device for providing image processing instruction in a vision system. In a robot system 100 as shown in Figure 1, the function of a teaching device for providing image processing instruction can be implemented, for example, on the image processing device 20 or on the robot control device 50. The embodiment described below will explain an example in which the function of a teaching device for providing image processing instruction is implemented on the image processing device 20.

[0010] Figure 1 of the first embodiment shows the equipment configuration of a robot system 100 including an image processing device 20 according to the first embodiment. The robot system 100 is configured to detect a workpiece using a visual sensor 70 and perform handling on the workpiece. Handling may include various tasks performed by the robot, such as pick and place, storage in containers or vessels, and palletizing. As shown in Figure 1, the robot system 100 comprises a robot 10, a robot control device 50 that controls the robot 10, a teaching control panel 40 connected to the robot control device 50, a visual sensor 70, and an image processing device 20 that controls the visual sensor 70. The image processing device 20 is connected to the robot control device 50. The robot system 100 also includes peripheral devices 80. Peripheral devices 80 include, for example, a workbench for placing workpieces, a container, and a transport device for transporting workpieces.

[0011] Here, robot 10 is described as a vertical articulated robot, but various types of robots may be used as robot 10 depending on the task, such as a horizontal articulated robot, a parallel link robot, or a dual-arm robot. Robot 10 can perform desired tasks using an end effector attached to its wrist. The end effector is an external device that can be replaced depending on the application, such as a hand or a tool. Figure 1 shows an example where a hand 15 is used as an example of an end effector.

[0012] The visual sensor 70 may be a two-dimensional camera that captures grayscale images or color images, or it may be a three-dimensional camera such as a stereo camera that can acquire depth images or three-dimensional point clouds. In Figure 1, an example is shown in which the visual sensor 70 is fixed in a position in the workspace where it can capture images of the workpiece, but the visual sensor 70 may also be attached to the tip of the arm of the robot 10.

[0013] The image processing device 20 has the function of performing various image processing operations such as detection and determination based on the image captured by the visual sensor 70. The image processing functions of the image processing device 20 include, for example, a function to detect a workpiece by pattern matching between the image of the workpiece in the captured image and a model pattern. In this embodiment, the visual sensor 70 is assumed to be calibrated, and the image processing device 20 is assumed to have calibration data (internal parameters and external parameters) that includes the definition of the relative positional relationship between the visual sensor 70 and the robot 10.

[0014] The image processing device 20 may have a configuration similar to a general computer, with memory (ROM, RAM, non-volatile memory, etc.), a display unit 23, an operation unit 24 including various input devices, and input / output interfaces connected to the processor 21 via a bus (see Figure 2). The input / output interfaces may include various input / output interfaces such as network interfaces, serial communication interfaces, and memory card interfaces.

[0015] In Figure 1, the image processing device 20 is shown as a separate device from the robot control device 50, but the functions of the image processing device 20 may also be incorporated into the robot control device 50.

[0016] The robot control device 50 controls the operation of the robot 10 according to an operation program or commands from the teaching control panel 40. The robot control device 50 may have a configuration similar to a general computer, with a processor connected via a bus to memory (ROM, RAM, non-volatile memory, etc.), input / output interfaces, and an operation unit including various operation switches. The input / output interfaces may include various input / output interfaces such as network interfaces, serial communication interfaces, and memory card interfaces.

[0017] The teaching control panel 40 is used as a device for teaching the robot 10 and for inputting and displaying information on the screen for various settings. The teaching control panel 40 may have a configuration similar to a general computer, with a processor connected via a bus to an operation unit consisting of memory (ROM, RAM, non-volatile memory, etc.), a display unit 43, an input device such as a keyboard (or software keys), and input / output interfaces (see Figure 2). The input / output interfaces may include various types of input / output interfaces such as a network interface, a serial communication interface, and a memory card interface.

[0018] Figure 2 is a functional block diagram showing the functional configuration of the image processing device 20 and the robot control device 50. As shown in Figure 2, the image processing device 20 includes a detection unit 121, a setting unit 122, an operation history recording unit 123, and an operation history display unit 124. As will be described later, the image processing device 20 may further include an adjustment parameter determination unit 125 for proposing setting parameters to be adjusted to the user. These functional blocks may be realized by the processor 21 executing software.

[0019] Figure 2 illustrates the storage unit 22 as a hardware component. The storage unit 22 is a storage device consisting of, for example, non-volatile memory or a hard disk drive. The storage unit 22 stores the operation history performed by the user via the settings screen for the visual detection function, detection results, and various other information related to image processing.

[0020] The detection unit 121 acquires an image of the workpiece captured by the visual sensor 70 and has the function of detecting the workpiece from the captured image using model data of the workpiece through pattern matching or the like.

[0021] The setting unit 122 provides a function for configuring the visual detection function performed by the detection unit 121. The setting unit 122 may be configured to display a setting screen on the display unit 23 for setting various setting parameters (hereinafter sometimes simply referred to as parameters) related to the visual detection function, and to accept various settings related to the visual detection function via the setting screen.

[0022] The operation history recording unit 123 has the function of recording the operation history performed by the user via the setting screen related to the visual detection function and the detection results obtained by executing the visual detection function. The operation history and detection results may be stored, for example, in the storage unit 22.

[0023] The operation history display unit 124 has a function to display the relationship between the user's parameter adjustments and the detection results, based on the user's operation history and detection results recorded by the operation history recording unit 123, so that the relationship can be visually confirmed.

[0024] As shown in Figure 2, the robot control device 50 has an motion control unit 151. The motion control unit 151 controls the movement of the robot 10 and the hand 15 according to the motion program or according to commands from the teaching control panel 40. The robot control device 50 also includes a servo control unit (not shown) that performs servo control to the servo motor of each axis according to the commands for each axis generated by the motion control unit 151.

[0025] Figure 3 shows an example of a setting screen 200 related to the visual detection function provided by the setting unit 122. As shown in Figure 3, the setting screen 200 includes a program display area 210 for displaying the detection program, a parameter setting screen 220 for setting various parameters related to the visual detection function, an image display area 230 for displaying the image of the detection result, and a detection result display area 240 for displaying numerical information of the detection result, etc.

[0026] The program display area 210 displays the detection program. Here, an example is shown in which the detection program (2D image detection) is composed of an imaging icon 201, a detection icon 202, and a correction data calculation icon 203. The imaging icon 201 corresponds to a command to capture an image of the target object using a visual sensor, the detection icon 202 corresponds to a command for detection processing based on the captured image, and the correction data calculation icon 203 corresponds to a command to calculate a correction position to correct the robot's position based on the position of the target object as a detection result. When the user selects the detection icon 202 on the settings screen 200 and selects the settings tab 251, the parameter setting screen 220 is displayed as a screen for setting various parameters related to the detection process. Figure 3 shows the state in which various parameter items related to the detection process are displayed on the parameter setting screen 220.

[0027] The user can input or adjust the values ​​of various parameters on the parameter setting screen 220. After inputting or adjusting the parameters on the parameter setting screen 220, the user can apply the adjusted parameters and run the detection program by, for example, operating the execute button 211. The image resulting from the execution of the detection process is displayed in the image display area 230, and the detection result information is displayed in the detection result display area 240. The user can check the detection results with the adjusted parameters via the image display area 230 and the detection result display area 240.

[0028] As an example, the parameters related to the detection process shall include the following: - Score threshold - Contrast threshold - Angle - Size - Aspect Ratio The score thresholds are thresholds related to the match score in detection processes such as pattern matching. The contrast threshold is a threshold related to the contrast in the captured image. The angle represents the rotation angle of the detected object relative to the reference orientation of the object. The size represents the size of the detected object relative to the reference size of the object. The aspect ratio represents the aspect ratio of the shape of the detected object relative to the reference shape of the object. Figure 3 shows the parameter setting screen 220 with the score threshold adjustment field 221, the contrast threshold adjustment field 222, the angle adjustment field 223, and the size adjustment field 224 displayed. All setting items can be displayed and operated by scrolling the display on the parameter setting screen 220 in Figure 3.

[0029] The following describes a function that allows users to easily set suitable parameters for the visual detection function via such a settings screen 200, primarily with reference to Figures 3 and 4 through 7.

[0030] Figure 5 is a flowchart showing the process for saving the operation history (operation history recording process) which is executed under the control of the processor 21. The operation history recording unit 123 starts the operation history recording process when, for example, the parameter setting screen 220 for setting the parameters of the detection process is activated. When the parameter setting screen 220 is activated, the setting unit 122 accepts parameter adjustments by the user via the parameter setting screen 220 (step S1). The setting unit 122 also executes the detection process according to instructions from the user (step S2).

[0031] When a detection result is obtained by executing the detection process, the operation history recording unit 123 associates the parameters used in the detection process (i.e., the parameter adjustments) with the detection result and stores them in the storage unit 22 (step S3). Typically, the user repeatedly adjusts the parameters and checks the detection result. Therefore, the processes from steps S1 to S4 are executed repeatedly (S4: YES → S1 loop processing), and the operation history recording unit 123 saves the results of multiple detection processes and the operation history of the parameters used in each detection process. For example, when the user performs an operation to indicate termination, the termination condition is met (S4: NO), and this process ends.

[0032] The operation history recording process shown in Figure 3 records the operation history and detection results when the user performs the detection process multiple times. The operation history display unit 124 has the function of displaying information about which parameters the user adjusted and how (user parameter adjustment history) based on the recorded operation history, so that the user can accurately grasp it visually. Figure 6 shows the operation history display process by the operation history display unit 124. For example, the operation history display unit 124 displays the operation history as a graph in response to a predetermined operation by the user that instructs the display of the operation history (step S11). Figure 4 shows an example of the display method of the graph of the operation history in step S11.

[0033] In step S11, the operation history display unit 124 displays the operation history display screen 300 shown in Figure 4 on the display unit 23. In the example in Figure 4, the changes in the parameter adjustment values ​​during the execution of the detection process five times are displayed in graph form. The operation history display screen 300 displays graphs 301 showing the changes in the adjustment of the score threshold, graph 302 showing the changes in the adjustment of the contrast threshold, graph 303 showing the changes in the adjustment of the angle (minimum value), graph 304 showing the changes in the adjustment of the angle (maximum value), graph 305 showing the changes in the adjustment of the size, and graph 306 showing the changes in the adjustment of the aspect ratio.

[0034] The operation history display unit 124 also has a function to display detection results associated with the operation history. This function may be implemented in a manner such that, as shown in Figure 4 as an example, when the user clicks on the number of executions, the detection result for that execution is displayed on a pop-up screen. In the example in Figure 4, when the user clicks on the execution count '3', an image 310 showing the detection result of the third detection process is displayed. Image 310 includes the captured image 311 as the detection result and numerical information 312 of the detection result (number of detections '1' and detection position). Also in the example in Figure 4, when the user clicks on the execution count '4', an image 320 showing the detection result of the fourth detection process is displayed. Image 320 includes the captured image 321 as the detection result and numerical information 322 of the detection result (number of detections '0').

[0035] This display method of the operation history allows the user to accurately and quickly grasp the adjustment status of each parameter. Simultaneously, the user can also understand the detection results for each parameter's adjustment state.

[0036] Therefore, users can quickly and easily understand the relationship between parameter adjustments and detection results. More specifically, users can enjoy the following benefits: - They can understand the state of the parameters when the detection result is no detection (number of detections '0') (which parameters were adjusted and which were not). - When the state changes from no detection to a state where detection results are obtained, they can understand which parameters were adjusted (the factors that improved the detection results). - When the state changes from a state where detection results are obtained to a state where no detection results are obtained, they can understand which parameters were adjusted (the factors that led to no detection). - They can understand the factors that cause false positives (e.g., over-adjusting parameters). - They can understand the factors that result in too many detections. - They can understand the factors that cause the processing time of the detection process to increase (for example, due to too many features to be processed).

[0037] As an example, from the operation history display screen 300 shown in Figure 4, the user can estimate that the reason for the failure to detect a factor in the fourth execution was due to the adjustment of the angle (graphs 303 and 304).

[0038] Therefore, according to the above-described display method of the operation history, even beginners unfamiliar with adjusting the parameters of the visual detection function can easily and quickly grasp the relationship between parameter adjustment and detection. Thus, it is possible to appropriately support the user in adjusting the parameters.

[0039] Next, we will describe the functions provided by the adjustment parameter determination unit 125 when the image processing device 20 is configured to have an adjustment parameter determination unit 125. The adjustment parameter determination unit 125 has a function to automatically estimate and suggest to the user the parameter that the user should adjust next, based on the recorded operation record and detection results. That is, the adjustment parameter determination unit 125 can function as an adjustment candidate suggestion unit that suggests candidate parameters that the user should adjust next. The adjustment parameter determination unit 125 may notify the user of the parameter that has been determined to be adjusted by display, or it may have a function to notify the user by voice.

[0040] Figure 7 is a flowchart showing the operation history display process when such a function by the adjustment parameter determination unit 125 is additionally applied. Similar to step S11 described above, after the process of graphing and displaying the operation history is performed (step S21), the adjustment parameter determination unit 125 performs a process of proposing parameters to be adjusted based on the recorded operation history and detection results (step S22).

[0041] In step S22, based on the relationship between the operation history and the detection result as shown in FIG. 4, the adjustment parameter determination unit 125 estimates the parameter whose value has changed between the third execution (detection count '1') and the fourth execution (detection count '0') as the parameter causing the non-detection, and (2) identifies the parameters whose values have not changed in the third execution (detection count '1') and the fourth execution (detection count '0'), and proposes the identified parameters as adjustment candidates. Such processing may be performed. The proposal of adjustment candidates may be performed, for example, by overlapping and displaying a message screen recommending that the identified adjustment candidates be adjusted on the operation history display screen 300.

[0042] Second Embodiment Hereinafter, an image processing apparatus according to the second embodiment will be described. Since the configuration of the robot system in which the image processing apparatus according to the second embodiment is used is the same as the configuration shown in FIG. 1, FIG. 1 will also be appropriately referred to in the description of this embodiment. As will be described in detail below, the image processing apparatus 20A according to the second embodiment provides a function of proposing to the user the parameters to be adjusted next based on the operation history performed by the user via the setting screen regarding the visual detection function.

[0043] FIG. 8 is a functional block diagram of the image processing apparatus 20A and the robot control apparatus 50 according to the second embodiment. In FIG. 8, functional blocks having the same functions as the functional blocks according to the first embodiment are denoted by the same reference numerals, and the description of those functions is omitted. As shown in FIG. 8, the image processing apparatus 20A includes a detection unit 121, a setting unit 122, an operation history recording unit 123A, an adjustment parameter determination unit 125A, and a learning unit 126. These functional blocks may be realized by the processor 21 executing software.

[0044] The storage unit 22 stores the operation history, detection results, adjustment candidate list, and various other information related to image processing performed by the user via the setting screen regarding the visual detection function.

[0045] ]Similar to the case of the first embodiment, the setting unit 122 presents the setting screen 200 (FIG. 3) of the visual detection function, and the user makes settings regarding the visual detection function via the setting screen 200.

[0046] The operation history recording unit 123A has a function of saving the operation history of parameters performed via the parameter setting screen 220. The operation history is stored in the storage unit 22.

[0047] The adjustment parameter determination unit 125A has a function of determining the parameter that the user should adjust next based on the recorded operation history and proposing it to the user. That is, the adjustment parameter determination unit 125A can function as an adjustment candidate proposal unit that proposes candidates for the parameter that the user should adjust next. The adjustment parameter determination unit 125A may notify the user of the parameter determined to be adjusted by display, or may have a function of notifying the user by voice.

[0048] The storage unit 22 may store an adjustment candidate list that defines the priority order for proposing regarding a plurality of parameters. The adjustment parameter determination unit 125 may have a function of determining the parameter that the user should adjust next using this adjustment candidate list. An example of the adjustment candidate list is shown in Table 1 below.

[0049]

[0050] The adjustment parameter determination unit 125 extracts the parameters that the user has not adjusted, which can be known by searching the operation history, and determines the parameter with the highest priority shown in the adjustment candidate list among the unadjusted parameters as the parameter to be adjusted next. For example, when the adjustment candidate list shown in Table 1 above is provided, when the user has already adjusted the contrast threshold in the operation history, the adjustment parameter determination unit 125A can determine the score threshold, which is the parameter with the highest priority among the unadjusted parameters (score threshold, angle, size, flatness ratio), as the parameter to be adjusted next.

[0051] The list of adjustment candidates may be generated by determining the priority of parameters that are considered effective in improving detection results based on past experience, experiments, etc.

[0052] Alternatively, the list of adjustment candidates may be determined based on a learning model constructed by the learning unit 126 by learning the relationship between a large number of training images and the manner in which parameters are adjusted by experts.

[0053] The learning unit 126 has the function of learning the relationship between images and the training data attached to those images using machine learning (supervised learning). For learning, a deep learning method using a multilayer neural network may be used. As the multilayer neural network, a convolutional neural network (CNN) suitable for the field of image recognition may be used. In the field of image recognition, CNNs are used for various tasks such as image classification and object detection. For example, in the task of image classification, the trained CNN (trained model) outputs a probability score for each class for the input image. By using this function of the learning unit 126, it is possible to learn the relationship between training images and the manner in which experts adjust parameters for training images (parameter adjustment order, adjustment amount, etc.). By inputting the image to be detected into the learning model constructed in this way, useful information such as the parameter adjustment order (priority) performed by experts can be extracted. The learning model constructed by the learning unit 126 may be stored in the memory unit 22.

[0054] The function of inputting data into the constructed learning model to obtain information on the priority of adjustments may be implemented as a function of the adjustment parameter determination unit 125A.

[0055] In this embodiment, an example configuration was described in which a learning model is constructed using the learning unit 126 provided in the image processing device 20A. However, the learning model may be generated by an external computer, and the image processing device 20A may acquire the learning model from the external computer and store it in the storage unit 22 for use. In such a configuration, the learning unit 126 in the image processing device 20A can be omitted.

[0056] The following explanation will primarily refer to Figures 3, 9, and 10 to describe the operation history saving function of the operation history recording unit 123A and the function of determining and suggesting the parameters that the user should adjust next using the adjustment parameter determination unit 125A.

[0057] Assume that the user has performed a predetermined operation to activate the parameter setting screen 220 shown in Figure 3. The operation history recording unit 123A starts recording the user's operations on the setting parameters in response to the activation of the parameter setting screen 220.

[0058] Figure 10 is a flowchart showing the adjustment parameter determination process executed by the adjustment parameter determination unit 125A when the parameter setting screen 220 is launched and the operation history recording unit 123A has started saving the operation history. The adjustment parameter determination unit 125A starts the adjustment parameter determination process in response to the detection of the following two events: (Event 1) The user instructs the execution of the detection process (Step S101) (Event 2) The user adjusts any parameter to its limit value (Step S103)

[0059] Assume that the user has adjusted the parameters and, for example, has instructed the detection process to be executed by operating the execute button 211 (step S101). In this case, the adjustment parameter determination unit 125A checks whether the workpiece was not detected as a result of the execution of the detection process (step S102). If the workpiece is not detected (S102: YES), it is considered that it is necessary to suggest parameters to be adjusted to the user, and the process from step S104 is performed. On the other hand, if the workpiece is detected (S102: NO), it is considered that it is not necessary to suggest parameters to be adjusted to the user, and the adjustment parameter determination process is terminated.

[0060] The loop processing from steps S104 to S106 corresponds to the process of determining the next parameter to be adjusted based on the adjustment candidate list and the operation history recorded by the operation history recording unit 123A. In step S104, the adjustment parameter determination unit 125A selects the parameter with the highest priority from the adjustment candidate list and sets it as an adjustment candidate. Next, the adjustment parameter determination unit 125A searches the operation history (step S105) and checks whether there is a record that the user has adjusted the determined adjustment candidate (step S106).

[0061] As a result, if there is a record of the user having adjusted a candidate for adjustment (S106: YES), that candidate for adjustment is a parameter that the user has already adjusted, and therefore, that candidate for adjustment is not included in the suggestions. In other words, in this case, the process returns to step S104, and the adjustment parameter determination unit 125A selects the next highest priority parameter from the list of adjustment candidates and sets it as an adjustment candidate. If it is determined in step S106 that there is no record of adjustment of the candidate for adjustment in the operation record (S106: NO), that candidate for adjustment is determined as the final adjustment candidate and the process proceeds to step S107. Through this loop process, the highest priority parameter among the parameters that the user has not yet adjusted can be determined as an adjustment candidate.

[0062] In step S107, the adjustment parameter determination unit 125A displays the finally determined adjustment candidates on the screen to suggest to the user as the next parameter to be adjusted. Figure 9 shows an example of the display for suggesting parameters in step S107. As shown in Figure 9, if the finally determined adjustment candidate is an angle, a message 410 suggesting that the angle parameter be adjusted may be displayed in the form of a pop-up screen superimposed on the "Angle" section of the parameter setting screen 220. Note that the display configuration in Figure 9 is an example, and other display configurations for suggesting parameters may be used, such as displaying the screen containing the suggestion message in a fixed area of ​​the setting screen 200.

[0063] Furthermore, if any parameter is adjusted to its limit value on the settings screen 200 (event 2 (step S103)), the processing from step S104 is also performed. In this case, by going through the loop processing from step S104 to S106, the final adjustment candidate is determined from among the parameters other than the one the user has adjusted to its limit value, and which the user has not yet adjusted, and which has the highest adjustment priority in the adjustment candidate list. For example, suppose the user has adjusted the contrast threshold (adjustment field 222) to its limit value, as shown in the parameter setting screen 220 in Figure 3. In this case, the final adjustment candidate can be determined from among the candidates other than the contrast threshold that the user has not yet adjusted, and which has the highest priority in the adjustment candidate list (for example, angle).

[0064] Event 1 represents a situation where the user has adjusted parameters and run the detection process, making it a suitable situation to suggest candidate parameters to adjust next. Event 2 represents a situation where the user has adjusted the parameters to their limit, suggesting that the user is not satisfied with the detection results, and is also a suitable situation to suggest candidate parameters to adjust next. Therefore, according to the parameter adjustment determination process described above, it is possible to suggest suitable parameters to adjust next to the user in situations that are suitable for suggesting parameters to adjust next.

[0065] In the above explanation, Event 2 was illustrated as the case where the user adjusts any parameter to its limit value. However, Event 2 could also be defined as the user adjusting the parameter to a value near the limit value (a value within a predetermined range relative to the limit value). In this case as well, the same effects as described above can be obtained.

[0066] As described above, each embodiment makes it possible for the user to easily adjust the parameters of image processing based on images captured by the visual sensor.

[0067] In the second embodiment described above, an example configuration was described in which the adjustment parameter determination process (Figure 10) is activated when either event 1 or event 2 occurs. In addition to these, other events that correspond to situations suitable for suggesting the next parameter to be adjusted may be adopted as events to activate the adjustment parameter determination process. For example, a situation in which there is insufficient information during the execution of the detection process is a situation suitable for suggesting the next parameter to be adjusted. Specifically, situations in which there is insufficient information during the execution of the detection process include situations in which the detection score in the detection process is lower than a certain value, and situations in which the number of features processed in the detection process (image processing) is lower than a certain value. By adding such situations as one of the events to activate the adjustment parameter determination process, it becomes possible to provide more appropriate support to users who are adjusting parameters.

[0068] In the above-described embodiment, we have explained the function of detecting an object as an image processing function, but the above-described embodiment can be applied to various types of vision systems, including object inspection and judgment.

[0069] Here, we will explain the flexibility of the functional arrangement shown in Figure 2 or Figure 8. Figure 2 or Figure 8 shows an example in which a functional block for teaching the visual detection function is placed within the image processing unit, and the image processing unit is configured as a teaching device for teaching the visual detection function. There is also a possible configuration in which at least some or all of the functional blocks placed within the image processing unit 20 or 20A are placed in the robot control unit 50, or in the robot control unit 50 and the teaching control panel 40. In this case, the robot control unit 50, or the robot control unit 50 and the teaching control panel 40, can function as a teaching device for teaching the visual detection function.

[0070] In the functional block diagrams of the image processing device and robot control device shown in Figure 2 or Figure 8, the functional blocks may be realized by one or more processors of these devices executing various software stored in a memory device, or in this case, some of the functions may be made up of hardware such as discrete circuits (i.e., the functional block may be realized by a combination of a processor and discrete circuits), or the functions shown in the functional block diagram may be realized by a hardware-based configuration such as an ASIC (Application Specific Integrated Circuit).

[0071] The various processes described in the above-described embodiment, such as operation history recording processing, operation history display processing, and adjustment parameter determination processing, or the programs that execute the processing of each part of the processor 21, may be provided in the form of a program product recorded on various recording media readable by a computer (for example, semiconductor memory such as ROM, EEPROM, flash memory, magnetic recording media, or optical recording media such as CD-ROM, DVD-ROM).

[0072] While this disclosure has been described in detail, it is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the gist of this disclosure or from the spirit of this disclosure derived from the claims and their equivalents. Furthermore, these embodiments can be implemented in combination. For example, the order of operations and processes in the embodiments described above are given as examples only and are not limited thereto. The same applies when numerical values ​​or mathematical formulas are used in the description of the embodiments described above.

[0073] The following additional notes are provided with respect to the above embodiments and modifications. (Note 1) A teaching device (20) comprising: an operation history recording unit (123) that records the user's operation history regarding the setting parameters of the image processing and the processing results of the image processing for multiple executions of the image processing; and an operation history display unit (124) that displays the relationship between the user's adjustment of the setting parameters and the processing results based on the recorded operation history and the processing results so that it can be visually confirmed. (Note 2) The teaching device (20) according to Note 1, wherein the operation history display unit (124) displays the history of adjustments of the setting parameters in a graph. (Note 3) The teaching device (20) according to Note 1 or 2, wherein the operation history display unit (124) displays the processing results for at least one of the multiple executions of the image processing in response to user operation. (Note 4) The teaching device (20) according to any one of Notes 1 to 3, further comprising an adjustment parameter determination unit (125) that determines setting parameters to be adjusted in order to improve the processing result of the image processing based on the recorded operation history and the processing result. (Note 5) The teaching device (20) according to Note 4, wherein the adjustment parameter determination unit (125) has a function to notify the user of the setting parameters to be adjusted that have been determined. (Note 6) The teaching device (20A) comprising: an operation history recording unit (123A) that records the user's operation history for one or more setting parameters of the image processing; and an adjustment parameter determination unit (125A) that determines the setting parameters to be adjusted next in order to improve the processing result of the image processing based on the recorded operation history. (Note 7) The image processing is a detection process for detecting an object, and the adjustment parameter determination unit (125A) determines the setting parameter to be adjusted next in response to any of the following events: (1) the detection result from the detection process is that no object has been detected, (2) one or more of the setting parameters has been adjusted to fall within a predetermined range based on a limit value, or (3) there is insufficient information from the execution of the detection process, as described in Note 6, teaching device (20A).(Note 8) The teaching device (20A) according to Note 6 or 7, further comprising a storage unit (22) that stores information representing the priority order of adjusting the one or more setting parameters, wherein the adjustment parameter determination unit (125A) determines the setting parameter to be adjusted next based on the information. (Note 9) The teaching device (20A) according to Note 8, wherein the adjustment parameter determination unit (125A) determines the setting parameter to be adjusted next, which is selected from the one or more setting parameters as the setting parameter that has the highest priority based on the information, and for which there is no record of adjustment in the operation history. (Note 10) The teaching device (20A) according to Note 8 or 9, wherein the information is generated by inputting the captured image that is the target of the image processing to a learning model constructed by learning based on a learning image and teacher data representing the order in which a skilled person adjusted the setting parameters on the learning image. (Note 11) The teaching device (20A) according to any one of Notes 6 to 10, wherein the adjustment parameter determination unit (125A) has a function of notifying the user of the setting parameter to be adjusted next that has been determined.

[0074] 10 Robot 15 Hand 20, 20A Image processing device 21 Processor 22 Memory unit 23 Display unit 24 Operation unit 40 Teaching control panel 43 Display unit 50 Robot control device 70 Vision sensor 80 Peripheral device 100 Robot system 121 Detection unit 122 Setting unit 123, 123A Operation history recording unit 124 Operation history display unit 125, 125A Adjustment parameter determination unit 151 Motion control unit 200 Setting screen 220 Parameter setting screen 300 Operation history display screen

Claims

1. A teaching device comprising: an operation history recording unit that records the user's operation history regarding the setting parameters of the image processing and the processing results of the image processing for multiple executions of the image processing; and an operation history display unit that displays the relationship between the user's adjustment of the setting parameters and the processing results based on the recorded operation history and the processing results so that the relationship can be visually confirmed.

2. The teaching device according to claim 1, wherein the operation history display unit displays the history of adjustments to the setting parameters in a graph.

3. The teaching device according to claim 1 or 2, wherein the operation history display unit displays the processing results for at least one of the multiple executions of the image processing in response to user operation.

4. The teaching device according to any one of claims 1 to 3, further comprising an adjustment parameter determination unit that determines setting parameters to be adjusted in order to improve the processing result of the image processing based on the recorded operation history and the processing result.

5. The teaching device according to claim 4, wherein the adjustment parameter determination unit has a function to notify the user of the determined setting parameters to be adjusted.

6. A teaching device comprising: an operation history recording unit that records the user's operation history for one or more setting parameters of image processing; and an adjustment parameter determination unit that determines the setting parameter to be adjusted next in order to improve the processing result of the image processing based on the recorded operation history.

7. The teaching device according to claim 6, wherein the image processing is a detection process for detecting an object, and the adjustment parameter determination unit determines the next setting parameter to be adjusted in response to any of the following events: (1) the detection result from the detection process is that no object has been detected, (2) one or more setting parameters have been adjusted to fall within a predetermined range based on a limit value, or (3) there is insufficient information from the execution of the detection process.

8. The teaching device according to claim 6 or 7, further comprising a storage unit for storing information representing the priority order of adjusting the one or more setting parameters, wherein the adjustment parameter determination unit further determines the setting parameter to be adjusted next based on the information.

9. The teaching device according to claim 8, wherein the adjustment parameter determination unit determines, among the one or more setting parameters, a setting parameter for which no adjustment record exists in the operation history, and which is selected from the information as having the highest priority, as the setting parameter to be adjusted next.

10. The teaching device according to claim 8 or 9, wherein the information is generated by inputting an image to be processed into a learning model constructed by learning based on a learning image and training data representing the sequence of adjustments of setting parameters made by an expert to the learning image.

11. The teaching device according to any one of claims 6 to 10, wherein the adjustment parameter determination unit has a function to notify the user of the setting parameter to be adjusted next, which has been determined.

Images