Device for detecting position of workpiece, control device, robot system, method, and computer program

Abstract

There has conventionally been a need to improve the accuracy of workpiece position detection.　A device 100 includes: an image acquiring unit 106 that acquires first image data reflecting a work region imaged in order to execute a first operation on a workpiece in the work region, and second image data reflecting a next work region imaged in order to execute a second operation, next after the first operation, on the workpiece in the next work region; a composite image generating unit 108 that generates composite image data obtained by combining the first image data and the second image data; and a position detecting unit 104 that detects the position of the workpiece that is the target of the second operation on the basis of the composite image data.

Description

Devices, control devices, robot systems, methods, and computer programs for detecting the position of a workpiece

[0001] This disclosure relates to devices, control devices, robotic systems, methods, and computer programs for detecting the position of a workpiece.

[0002] A device is known that detects the position of a workpiece based on image data captured by a visual sensor (for example, Patent Document 1).

[0003] Japanese Patent Publication No. 2019-162696

[0004] Traditionally, there has been a need to improve the accuracy of detecting the position of a workpiece.

[0005] In one embodiment of the present disclosure, a device for detecting the position of a workpiece based on image data captured by a vision sensor for a robot to perform work on the workpiece includes: an image acquisition unit that acquires first image data of a work area captured for performing a first operation on a workpiece within the work area, and second image data of a next work area captured for performing a second operation on a workpiece in a next work area adjacent to the work area, following the first operation; a composite image generation unit that generates composite image data by combining the first image data and the second image data; and a position detection unit that detects the position of the workpiece that is the target of the second operation based on the composite image data.

[0006] In another aspect of the present disclosure, a method for detecting the position of a workpiece based on image data captured by a vision sensor for a robot to perform an operation on the workpiece, the method comprising: a processor acquiring first image data of a work area captured for performing a first operation on the workpiece within the work area, and second image data of a subsequent work area captured for performing a second operation on a workpiece in a subsequent work area adjacent to the work area, after the first operation, generating composite image data by combining the first and second image data, and detecting the position of the workpiece to be operated on based on the composite image data.

[0007] This is a schematic diagram of a robot system according to one embodiment. This is a block diagram of a robot system according to one embodiment. This is a flowchart showing an example of the operation flow of the robot system shown in Figure 2. This shows an example of image data captured by a vision sensor. This is a diagram showing the work area and the next work area superimposed on the image data shown in Figure 4. This shows an example of a position database for the work area. This shows the state in the position database shown in Figure 6 where the status has been changed. This is a flowchart showing an example of the flow of step S5 in Figure 3. This shows other image data captured by a vision sensor. This is a diagram showing the work area and the next work area superimposed on the image data shown in Figure 9. This shows an example of first image data that captures the work area. This shows an example of second image data that captures the next work area. This shows an example of composite image data. This is a diagram showing the work area and the next work area superimposed on the composite image data shown in Figure 13. This shows an example of a position database for the next work area. This shows the work area and the next work area after resetting. This shows a position database for the work area. This shows other image data captured by a vision sensor. This is a diagram showing the work area and the next work area superimposed on the image data shown in Figure 18. This shows another example of second image data that captures the next work area. This shows another example of composite image data. Figure 21 is a diagram showing the work area and the next work area superimposed on the composite image data shown. This is a block diagram of a robot system according to another embodiment. This is a flowchart showing an example of the operation flow of the robot system shown in Figure 23. This is a diagram showing a robot in operation that has entered the field of view of the visual sensor. This shows an example of the robot's occupied area estimated in the sensor coordinate system. This shows the state in which the work area and the next work area are set in the image data. This shows yet another image data captured by the visual sensor. This shows another example of the first image data that captures the work area. This shows yet another example of the second image data that captures the next work area. This shows yet another example of composite image data. Figure 31 is a diagram showing the work area and the next work area superimposed on the composite image data shown. This is a diagram showing a robot in operation that has entered the field of view of the visual sensor. This shows another example of the robot's occupied area estimated in the sensor coordinate system. This shows the state in which the work area and the next work area are set in the image data. This shows yet another image data captured by the visual sensor. This shows yet another example of the second image data that captures the next work area.Here are yet another example of composite image data.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the various embodiments described below, the same elements will be denoted by the same reference numerals, and redundant explanations will be omitted. First, a robot system 10 according to one embodiment will be described with reference to Figures 1 and 2. The robot system 10 comprises a robot 12, a vision sensor 14, and a control device 16.

[0009] The robot 12 performs a predetermined operation WK on the workpiece W. In this embodiment, the robot 12 is a vertical articulated robot and has a robot base 18, a swivel body 20, a forearm 22, an upper arm 24, a wrist 26, and an end effector 28. The robot base 18 is fixed to the floor of the work cell or on an automated guided vehicle (AVG).

[0010] The rotating torso 20 is mounted on the robot base 18 so as to be able to rotate around a vertical axis. The lower arm 22 is mounted on the rotating torso 20 so as to be able to rotate around a horizontal axis at its base end, and the upper arm 24 is mounted on the tip of the lower arm 22 so as to be able to rotate around its base end. The wrist 26 is mounted on the tip of the upper arm 24 so as to be able to rotate around two mutually orthogonal axes.

[0011] The end effector 28 is detachably attached to the wrist portion 26. The end effector 28 is, for example, a robot hand capable of gripping a workpiece, a paint applicator for coating a workpiece, a welding torch for welding a workpiece, or a laser processing head for laser processing a workpiece, and performs a predetermined operation WK (workpiece handling, coating, welding, or laser processing) on ​​the workpiece.

[0012] Each component of the robot 12 (robot base 18, swivel torso 20, forearm 22, upper arm 24, wrist 26) is equipped with a servo motor 30 (Figure 2). These servo motors 30 rotate each movable element of the robot 12 (swivel torso 20, forearm 22, upper arm 24, wrist 26, end effector 28) in response to commands from the control device 16. As a result, the robot 12 moves the end effector 28 to any desired position.

[0013] The visual sensor 14 captures images of a subject, such as a workpiece W, for the robot 12 to perform a task WK. In this embodiment, the visual sensor 14 is a three-dimensional visual sensor having a stereo camera and an image processing processor. The visual sensor 14 may be fixed at a predetermined imaging position Pi. The visual sensor 14 is configured to capture images of a subject along its optical axis and to measure the distance to the subject.

[0014] As shown in Figure 1, the robot 12 is configured with a robot coordinate system C1 and a tool coordinate system C2. The robot coordinate system C1 is a coordinate system for automatically controlling the movement of each movable element of the robot 12. In this embodiment, the robot coordinate system C1 is configured with respect to the robot base 18 such that its origin is located at the center of the robot base 18 and its z-axis is parallel to (specifically coincides with) the rotation axis of the slewing body 20.

[0015] On the other hand, the tool coordinate system C2 is a coordinate system that defines the position of the end effector 28 in the robot coordinate system C1. In this embodiment, the tool coordinate system C2 is set relative to the end effector 28 such that its origin (so-called TCP) is located at the working position of the end effector 28 (for example, the workpiece gripping position, paint spray nozzle, welding position, or laser beam output port).

[0016] When moving the end effector 28, the control device 16 sets the tool coordinate system C2 in the robot coordinate system C1 and generates commands to each servo motor 30 of the robot 12 to position the end effector 28 at the position represented by the set tool coordinate system C2. In this way, the control device 16 can position the end effector 28 at any position in the robot coordinate system C1. In this paper, "position" may refer to both position and orientation.

[0017] On the other hand, the vision sensor 14 is configured with a sensor coordinate system C3. Sensor coordinate system C3 is a coordinate system that defines the position of the vision sensor 14 in the robot coordinate system C1 (specifically, the position and direction of the optical axis). In this embodiment, the sensor coordinate system C3 is configured with respect to the vision sensor 14 such that its origin is located at the center point of the stereo camera of the vision sensor 14 (specifically, an imaging sensor such as a CMOS or CCD), and its z-axis is parallel to (specifically coincides with) the optical axis.

[0018] Each pixel IE of the image data ID captured by the visual sensor 14 is represented as coordinates in the sensor coordinate system C3. For example, the visual sensor 14 captures either 3D point cloud image data, which represents the visual features (surfaces, edges, etc.) of the workpiece W as a 3D point cloud, or distance image data, which represents the visual features of the workpiece W as shades of color according to the distance from the visual sensor 14, as the image data ID. The point cloud of the 3D point cloud image data and each pixel constituting the distance image data constitute a pixel IE, which is represented as coordinates (x, y, z) in the sensor coordinate system C3.

[0019] The positional relationships between the robot coordinate system C1, the tool coordinate system C2, and the sensor coordinate system C3 are known through calibration. Therefore, the coordinates of the robot coordinate system C1, the tool coordinate system C2, and the sensor coordinate system C3 can be converted to each other via known transformation matrices (e.g., homogeneous transformation matrices or Jacobian matrices). These robot coordinate system C1, tool coordinate system C2, and sensor coordinate system C3 constitute a control coordinate system C for controlling the movement of the robot 12.

[0020] The control device 16 controls the operation of the robot 12 and the vision sensor 14. Specifically, as shown in Figure 2, the control device 16 is a computer having a processor 32, memory 34, and I / O interface 36. The processor 32 has a CPU or GPU, and is communicated with the memory 34 and I / O interface 36 via a bus 38. While communicating with these components, it performs calculation processing to realize the workpiece position detection function described later.

[0021] The memory 34 has RAM or ROM, etc., and stores various data temporarily or permanently. The memory 34 may consist of a computer-readable non-temporary storage medium such as volatile memory, non-volatile memory, magnetic storage medium, or optical storage medium. The I / O interface 36 has, for example, an Ethernet® port, a USB port, an optical fiber connector, or an HDMI® terminal, and communicates data with external devices by wired or wireless connection under commands from the processor 32. Each servo motor 30 and the vision sensor 14 of the robot 12 are communicated to the I / O interface 36.

[0022] In this embodiment, the processor 32 performs the operation WK of grasping and picking up the workpieces W, which are loosely stacked in container A (Figure 1), with the end effector 28. The operation of the robot system 10 will now be described with reference to Figure 3. When the processor 32 of the control device 16 receives a work start command from the operator, the higher-level controller, or the computer program PG, it starts the flow shown in Figure 3.

[0023] In step S1, the processor 32 operates the visual sensor 14 to image the workpiece W inside container A. Image data ID captured by the visual sensor 14 in step S1 １ An example is shown in Figure 4. As shown in Figure 4, the image data ID １ The image shows multiple workpieces W located within the field of view 40 of the visual sensor 14, and the image data ID １ In this system, each pixel IE (for example, a point in a 3D point cloud image data, or a pixel in a depth image) that captures the workpiece W is represented as a coordinate in the sensor coordinate system C3. The processor 32 receives the image data ID captured by the vision sensor 14. １ This is obtained. Note that in the example shown in Figure 4, container A is omitted for ease of understanding.

[0024] In step S2, the processor 32 detects the position P of the workpiece W. Specifically, the processor 32 uses a workpiece model WM (e.g., a 3D CAD model) that models the workpiece W to determine the image data ID. １Execute a model matching MT that matches the work W shown in the figure. By this model matching MT, the processor 32 uses the image data ID as the position P of the work W in the sensor coordinate system C3 １ Detect the coordinates Ps in the sensor coordinate system C3 of the feature point FP of the work W shown in the figure. Note that the feature point FP may be set at the center point, centroid, or vertex of the edge of the work W, etc.

[0025] Here, in the present embodiment, the processor 32 automatically sets the work area 42 and the next work area 44 adjacent to the work area 42 in the control coordinate system C for controlling the robot 12. An example of the work area 42 and the next work area 44 is shown in FIG. 5. In the example shown in FIG. 5, the processor 32 sets the work area 42 so as to include a predetermined area (specifically, the area on the plus side of the y-axis of the sensor coordinate system C3) in the sensor coordinate system C3.

[0026] On the other hand, the processor 32 sets the next work area 44 so as to be adjacent to the work area 42 so as to include another predetermined area (the area on the minus side of the y-axis of the sensor coordinate system C3) in the sensor coordinate system C3. The work area 42 in FIG. 5 is the area targeted for the first-stage work WK １ defines the area, while the next work area 44 is the second-stage work WK １ to be executed next after the first-stage work WK ２ defines the area to be the work target.

[0027] Further, in the present embodiment, the processor 32 sets the work area 42 and the next work area 44 to overlap each other in the overlapping area 46. Note that the work area 42 and the next work area 44 may be set in the sensor coordinate system C3 as a cubic three-dimensional area, or may be set in the sensor coordinate system C3 as a rectangular two-dimensional area parallel to the x-y plane of the sensor coordinate system C3.

[0028] Furthermore, in the example shown in Figure 5, the work area 42 and the next work area 44 are set to fall within the field of view 40 of the visual sensor 14. The coordinates of the sensor coordinate system C3 of each vertex defining the work area 42 and the next work area 44 may be predetermined by the operator. In this way, the processor 32 sets the work area 42 and the next work area 44 to predetermined coordinates in the control coordinate system C (in this embodiment, the sensor coordinate system C3). Therefore, the processor 32 functions as a region setting unit 102 (Figure 2) that automatically sets the work area 42 and the next work area 44 in the control coordinate system C. Note that the processor 32 may set the work area 42 and the next work area 44 in the sensor coordinate system C3 before or after executing step S1.

[0029] In step S2, the processor 32 detects the position P of the workpiece W located within the work area 42. Specifically, the processor 32 extracts the coordinates Ps of the workpiece W located within the work area 42 from among the coordinates Ps of the multiple workpiece W detected by model matching MT. For example, in the example shown in Figure 5, the processor 32 identifies the coordinates Ps of the workpiece W within the work area 42. １ Coordinates Ps １ And, Work W ２ Coordinates Ps ２ And, Work W ３ Coordinates Ps ３ Let's assume that it was detected.

[0030] In this case, the processor 32 determines the coordinates Ps of the detected sensor coordinate system C3. １ Ps ２ and Ps ３ Convert to robot coordinate system C1, workpiece W １ , W ２ and W ３ The coordinates of the robot coordinate system C1 are Pr １ , Pr ２ and Pr ３ The processor 32 then obtains the obtained coordinates Pr １ , Pr ２ and Pr ３ This is stored in the position database 48 for the work WK in the work area 42.

[0031] An example of the data structure of this position database 48 is shown in Figure 6. In the position database 48 in Figure 6, column 50 shows the rank i (i = 1 to 3) of the workpiece W on which the operation WK is performed, and column 52 shows the coordinates Pr (x, y, z, w, p, r) of the detected workpiece W. Of the coordinates Pr, the coordinates (x, y, z) indicate the position of the workpiece W in the robot coordinate system C1, and the coordinates (w, p, r) indicate the orientation of the workpiece W in the robot coordinate system C1 (so-called yaw, pitch, roll).

[0032] Meanwhile, column 54 shows the status of the work WK for work W. In Figure 6, "Waiting for work" indicates that work W with rank i is not yet worked on and that work for this work W is scheduled to be performed. Thus, the processor 32 detects work W ｊ (In the example in Figure 5, the coordinates Pr = 1, 2, 3) ｊ A location database 48 is created to store the data in association with rank i and status, and this database is stored in memory 34.

[0033] The processor 32 then processes the acquired multiple coordinates Pr ｊ A rank i may be assigned to it according to a predetermined rule RL1. This rule RL1 is, for example, the coordinate Pr of the robot coordinate system C1. ｊ Alternatively, the rule could be to assign rank i to each element in descending order of its z-coordinate (i.e., vertical height).

[0034] In this way, the processor 32 detects the position P of the workpiece W in the control coordinate system C (in this embodiment, the robot coordinate system C1). Therefore, the processor 32 functions as a position detection unit 104 (Figure 2) that detects the position P of the workpiece W. In the example in Figure 5, a total of three workpieces W are detected in step S2. １ ~W ３ We have described what to do if it is detected, but what number of workpieces W ｊ Please understand that it may be detected.

[0035] In step S3, the processor 32 refers to the location database 48 for the work area 42 that is stored at this point and determines whether the location database 48 contains coordinates Pr with the status "waiting for work". If the processor 32 determines YES, it proceeds to step S4; otherwise, it terminates the flow shown in Figure 3.

[0036] In step S4, the processor 32 performs the work WK for the work area 42. ｎ The process begins. Specifically, the processor 32 refers to the location database 48 (Figure 6) stored in memory 34 at this point, and selects the coordinate Pr where rank i is the highest and "status" is "waiting for work". １ The processor 32 reads out the coordinates Pr １ Based on this, the robot 12 is operated to move the end effector 28 (i.e., the tool coordinate system C2) to coordinate Pr １ Positioned, the end effector 28 controls the workpiece W １ Pick them out.

[0037] Work W １ When the task WK for is properly completed, as shown in Figure 7, the processor 32 moves the coordinates Pr for rank i=1. １ The "Status" of is changed to "Task Completed". Then, the processor 32 moves the coordinate Pr whose "Status" is "Waiting for Task". ｊ Read them in the order of rank i, and work W ２ and W ３ The processor 32 then picks up the work WK in order. The processor 32 changes the "status" to "work completed" each time a work WK is completed. In this way, in step S4 which is executed for the first time, the processor 32 picks up the work W in the work area 42. ｊ First stage of work WK for (j=1,2,3) １ Execute this.

[0038] In step S5, the processor 32 executes the position detection scheme DS. This position detection scheme DS will be explained with reference to Figure 8. In step S11, the processor 32 operates the visual sensor 14 to image the workpiece W inside container A, similar to step S11. Image data ID captured by the visual sensor 14 in the first execution of step S11 ２ An example is shown in Figure 9.

[0039] In this embodiment, in step S11, the processor 32 determines that the robot 12 is performing the task WK １ While this is being performed, the visual sensor 14, which has the work area 42 and the next work area 44 (Figure 5) within its field of view 40, receives the image data ID ２ The image is captured. As an example, in the previous step S4, the processor 32 captures the coordinates Pr of any rank i stored in the position database 48. ｉ Work W having ｊ Step S11 is executed when the task WK for is started (more specifically, when the robot 12 starts operating for the task WK) (for example, i = 1).

[0040] As another example, the processor 32 uses the coordinates Pr stored in the position database 48 that was referenced in the previous step S4. ｉ Depending on the number of coordinates, the timing for imaging the workpiece W in step S11 may be determined. For example, the position database 48 contains k (k≧2) coordinates Pr ｉ Assume that the following is stored. In this case, the processor 32 determines rank i = k - k ｔｈ (k ｔｈ Coordinates Pr with ≤k) ｉ Work W ｊ Step S11 may be performed when the work WK for is started.

[0041] For example, k = 10, and k ｔｈ Let's assume it's set to =4. In this case, the processor 32 will execute step S11 when it starts work WK for the workpiece W with coordinate Pr assigned rank i=6 in the position database 48. Threshold k for determining the imaging timing ｔｈThis can be predetermined by the operator, taking into consideration the time required for the calculation process in step S14 described later.

[0042] In the example shown in Figure 9, step S4 is the work WK １ Image data ID acquired after the start ２ The robot 12 (end effector 28, upper arm 24) in operation is visible in the image. Also, as shown in Figure 10, the image data ID ２ In the sensor coordinate system C3 that defines the coordinates, a work area 42 and a next work area 44 are set, similar to those in Figure 5.

[0043] In step S12, the processor 32 receives the first image data ID ｎ 'and the second image data ID ｎ＋１ ' and obtain the first image data ID. ｎ ' is the nth step of work WK for the workpiece W in the work area 42. ｎ This is image data of the work area 42, captured in order to perform the operation. On the other hand, the second image data ID ｎ＋１ ' is the nth stage of work WK ｎ Next, the (n+1)th step of work WK for the workpiece W in the next work area 44. ｎ＋１ This is image data of the next work area 44, captured in order to perform the operation.

[0044] Here, the variable "n" indicates the number of times the loop of steps S3 to S7 shown in Figure 3 has been executed. Therefore, when step S12 is executed for the first time, n = 1, so the processor 32 generates the first image data ID ｎ 'As the first stage of work WK １ Image data ID acquired to perform the first step S4 １ Among them, the image data ID that captures the work area 42 １ Get '.

[0045] Specifically, the processor 32 generates image data IDs obtained in step S1 from the visual sensor 14 capturing the subjects (container A, workpiece W) within the field of view 40. １ In (Figure 5), by disabling the pixel IE that captures the next work area 44, the image data ID that captures the work area 42 is disabled.１ Obtain the image data ID １ An example of the image data ID is shown in FIG. 11. As an example, when the image data ID １ is three-dimensional point cloud image data, the processor 32 deletes the point cloud within the next working area 44 from the image data ID １ shown in FIG. 5, and may generate the image data ID １ '.

[0046] As another example, when the image data ID １ is distance image data, the processor 32 sets the luminance α of the pixel that copies the next working area 44 to a predetermined value α １ (for example, α ０ = 0) in the image data ID ０ shown in FIG. 5, and may generate the image data ID １ '. Thus, in this embodiment, the processor 32 obtains the image data ID １ ' that copies the working area 42 based on the image data ID １ that copies the visual field 40 including the working area 42 and the next working area 44. １

[0047] On the other hand, the processor 32 obtains the image data ID ｎ＋１ ' as the second image data ID ２ ' that copies the next working area 44 from the image data ID ２ captured for executing the second-stage operation WK ２ (that is, the second step S4). Specifically, the processor 32 invalidates (removes the point cloud or sets the luminance α of the pixel to a predetermined value α ２ shown in FIG. 10) the pixel IE (the point cloud of the three-dimensional point cloud image data or the pixel of the distance image data) that copies the working area 42 in the image data ID ０ captured by the vision sensor 14 within the visual field 40 in the immediately previous step S11, and obtains the image data ID ２ ' that copies the next working area 44.

[0048] The image data ID ２An example of ' is shown in FIG. 12. Thus, the processor 32 images the image data ID ２ of the next working area 44 based on the ２ ' to generate the image data ID ２ ' and obtains the image data ID ｎ '. In this way, in the present embodiment, the processor 32 functions as an image acquisition unit 106 (FIG. 2) that acquires the first image data ID ｎ＋１ ' and the second image data ID

[0049] In step S13, the processor 32 generates composite image data CD ｎ ' by combining the first image data ID ｎ＋１ ' and the second image data ID ｎ '. For example, when executing the first step S13 (that is, n = 1), the processor 32 combines the image data ID １ ' shown in FIG. 11 and the image data ID ２ ' shown in FIG. 12 to generate composite image data CD １ '.

[0050] At this time, the processor 32 invalidates one of the first pixel IE １ ' that images the overlapping area 46 (FIG. 11) in the image data ID １ and the second pixel IE ２ ' that images the overlapping area 46 (FIG. 12) in the image data ID ２ according to a predetermined rule RL2. Specifically, the processor 32 overlaps the image data ID １ ' and the image data ID ２ ' with each other based on the sensor coordinate system C3. At this time, the first pixel IE １ ' of the image data ID １ and the second pixel IE ２ ' of the image data ID ２ [[ID=4..]]

[0051] As an example, the rule RL2 is the first pixel IE １ and the second pixel IE ２This includes a rule that invalidates the one with a larger distance δ from the coordinates in the sensor coordinate system C3 to the visual sensor 14 (i.e., the origin of the sensor coordinate system C3). For example, in the sensor coordinate system C3, the first pixel IE １ The z-coordinate of the second pixel IE ２ Assume it is greater than the z-coordinate of .

[0052] This is the first pixel IE １ The distance δ from the coordinates to the origin of the sensor coordinate system C3 (visual sensor 14) １ However, the second pixel IE ２ The distance δ from the coordinates to the origin of the sensor coordinate system C3. ２ Larger than (in other words, the first pixel IE) １ However, the second pixel IE ２ This means (further from the origin than). In this case, the processor 32 controls the first pixel IE １ Disable (i.e., remove the point cloud from the 3D point cloud image data, or set the brightness α of the pixels in the distance image data to a predetermined value α) ０ (Settings will be made.)

[0053] Another example is image data ID １ 'and ID ２ 'In this, priority is predetermined, and rule RL2 is the first pixel IE according to priority. １ and the second pixel IE ２ This includes a rule to invalidate one of the two. For example, image data IDs captured later in the time series. ２ A higher priority is assigned to the '. In this case, the processor 32 assigns a lower priority image data ID １ 'First pixel IE １ This would invalidate the image data ID captured earlier in the time series. １ A higher priority may be assigned to '.

[0054] By the method described above, the processor 32 generates the image data ID １ 'and ID ２ Synthesized image data CD created by combining ' １ Generates a composite image data CD. １An example of this is shown in Figure 13. In this embodiment, the processor 32 processes the first image data ID ｎ 'and the second image data ID ｎ＋１ ' and combined composite image data CD ｎ It functions as a composite image generation unit 108 (Figure 2) that generates the image.

[0055] In step S14, the processor 32 functions as a position detection unit 104 and generates the composite image data CD in the preceding step S13. ｎ Based on this, the (n+1)th stage of work WK ｎ＋１ The position P of the target workpiece W is detected. For example, when the first step S14 is executed (n=1), the processor 32 functions as a position detection unit 104 and generates the composite image data CD. １ Based on this, the second stage of work WK ２ This will detect the position P of the workpiece W that is the target.

[0056] As shown in Figure 14, the composite image data CD １ The sensor coordinate system C3 that defines the work area 42 and the next work area 44 are set. The processor 32, in the same manner as in step S2 described above, creates the synthesized image data CD. １ Model matching MT is performed on the workpiece W that is projected onto the screen. Here, in step S14, the processor 32 detects the position P of the workpiece W located in the next work area 44. Specifically, the processor 32 extracts the coordinates Ps of the multiple workpiece W detected by model matching MT that are located within the next work area 44.

[0057] For example, in the example shown in Figure 14, the processor 32 determines the coordinate Ps in the next work area 44 as the workpiece W ４ Coordinates Ps ４ And, Work W ５ Coordinates Ps ５ And, Work W ６ Coordinates Ps ６ Let's assume that it has detected the following. In this case, the processor 32 will determine the coordinates Ps of the detected sensor coordinate system C3. ４ Ps ５ and Ps ６ Convert to robot coordinate system C1, workpiece W４ , W ５ and W ６ The coordinates of the robot coordinate system C1 are Pr ４ , Pr ５ and Pr ６ The processor 32 then obtains the obtained coordinates Pr ４ , Pr ５ and Pr ６ This is the work WK for the next work area 44. ２ Stored in the location database 56.

[0058] An example of the data structure of this location database 56 is shown in Figure 15. In the location database 56 shown in Figure 15, column 58 is the work W ｊ The rank i (i = 1 to 3) is shown, and column 60 shows the detected work W. ｊ Coordinates Pr ｊ The values ​​(x, y, z, w, p, r) are shown (j = 4, 5, 6). Column 62 also shows the status mentioned above.

[0059] Thus, in this embodiment, the processor 32 creates a position database 56 for the next work area 44, separate from the position database 48 for the work area 42, and stores it in the memory 34. In this way, the processor 32 processes the work WK started in step S4. ｎ In parallel with this, step S5 is executed, followed by the (n+1)th stage of work WK. ｎ＋１ The position P of the target workpiece W is detected.

[0060] Referring again to Figure 3, in step S6, the processor 32 performs the nth step of work WK for the work area 42. ｎ It determines whether the process is complete or not. Specifically, the processor 32 refers to the position database 48 (Figures 6 and 7) at this point in time and checks the coordinates Pr whose "Status" is "Waiting for work". ｊ If none exists, determine YES and proceed to step S7, while coordinates Pr where "Status" is "Waiting for work" ｊ If it exists, determine NO and loop through step S6.

[0061] In step S7, the processor 32 functions as a region setting unit 102 to reset the work area 42 and the next work area 44. Specifically, the processor 32 resets the new work area 42 and the next work area 44 by swapping the work area 42 and the next work area 44 that are set in the sensor coordinate system C3 at this point. Figure 16 shows the work area 42 and the next work area 44 after being reset in step S7, which is executed for the first time.

[0062] As shown in Figure 16, the reset work area 42 has the same coordinates in the sensor coordinate system C3 as the previous work area 44 (Figure 5), while the reset next work area 44 has the same coordinates in the sensor coordinate system C3 as the previous work area 42. In this way, the processor 32 resets the work area 42 and the next work area 44.

[0063] Furthermore, the processor 32 uses the coordinates Pr stored in the position database 56 (Figure 15) for the next work area 44 at this point in time. ｊ The coordinates Pr obtained in step S14 are transferred to the position database 48 for the work area 42. As a result, as shown in Figure 17, the coordinates Pr obtained in step S14 are transferred. ｊ (j=4 to 6) will be transferred to the position database 48 for the work area 42.

[0064] In the example shown in Figure 17, workpiece W has a status of "Task Completed" (i.e., the task in step S4 has been completed). １ ~W ３ Coordinates Pr １ ~Pr ３ However, it remains in the location database 48. However, the processor 32 has set the status to "Task complete" for the coordinates Pr ｊ It may be deleted from the location database 48.

[0065] After step S7, the processor 32 returns to step S3. If the location database 48 shown in Figure 17 is stored at this point, the processor 32 determines YES in step S3 and executes the second step S4. In this step S4, the processor 32 refers to the location database 48 shown in Figure 17 and performs the second stage of work WK on the work area 42 (Figure 16) that was reset in the most recent step S7. ２ Start.

[0066] Next, the processor 32 executes step S5 for the second time. Specifically, in step S11, the processor 32 operates the visual sensor 14 to image the workpiece W inside container A. The image data ID captured by the visual sensor 14 in step S11 executed for the second time ３ An example is shown in Figure 18. In the example shown in Figure 18, the work WK was performed in the preceding step S4. ２ Image data ID acquired after the start ３ Work WK ２ The robot 12, which is in operation, is visible in the image. Also, as shown in Figure 19, the image data ID ３ The sensor coordinate system C3 that defines this has the reset work area 42 and the next work area 44 defined.

[0067] In the second step S12, the processor 32 functions as an image acquisition unit 106 and the first image data ID ２ 'and the second image data ID ３ Specifically, the processor 32 obtains the first image data ID. ２ 'Image data ID captured before resetting in step S7' ２ (Figure 12) is used. Therefore, the processor 32 generates the image data ID in the first step S12. ２ The value ' is read and obtained from memory 34.

[0068] Meanwhile, the processor 32 processes the second image data ID ３ As described above, the third stage of work WK ３ Image data ID acquired to perform the third step S4 ３(Figure 19) Image data ID of the next work area 44 ３ Obtain the image data ID. ３ An example of this is shown in Figure 20. In the second step S13, the processor 32 functions as a composite image generation unit 108 and generates the image data ID in the manner described above. ２ 'and ID ３ By combining ', the combined image data CD ２ Generates a composite image data CD. ２ An example of this is shown in Figure 21.

[0069] In the second step S14, the processor 32 functions as a position detection unit 104 and generates the synthesized image data CD. ２ Based on this, the third stage of work WK ３ The position P of the target workpiece W is detected. As shown in Figure 22, the composite image data CD ２ The sensor coordinate system C3 that defines this has the reset work area 42 and the next work area 44 defined.

[0070] The processor 32 detects the position P of the workpiece W located within the reset-up next work area 44. Specifically, the processor 32 extracts the coordinates Ps of the workpiece W located within the next work area 44 from among the multiple coordinates Ps of the workpiece W detected by model matching MT. As a result, the processor 32 identifies the coordinates Ps within the next work area 44 as, for example, the workpiece W in Figure 22. ７ Coordinates Ps ７ And, Work W ８ Coordinates Ps ８ It can detect this.

[0071] Then, the processor 32 calculates the coordinates Ps of the detected sensor coordinate system C3. ７ and Ps ８ Convert to robot coordinate system C1, and the coordinates Pr of robot coordinate system C1 ７ and Pr ８ The processor 32 then obtains the obtained coordinates Pr ７ and Pr ８This is then newly stored in the position database 56 (Figure 15) for the next work area 44. In this way, the processor 32 repeatedly executes the loop of steps S3 to S7 while determining YES in step S3.

[0072] As described above, in this embodiment, the processor 32 functions as a region setting unit 102, a position detection unit 104, an image acquisition unit 106, and a composite image generation unit 108, so that the robot 12 can perform the work WK ｎ To perform this operation, based on the image data ID captured by the visual sensor 14, the position P (coordinates Ps) of the workpiece W is determined. ｊ ,pr ｊ ) is detected. Therefore, the region setting unit 102, position detection unit 104, image acquisition unit 106, and composite image generation unit 108 constitute a device 100 (Figure 2) for detecting the position P of the workpiece W.

[0073] In this device 100, the image acquisition unit 106 performs the nth step of the work WK on the workpiece W within the work area 42. ｎ First image data ID capturing the work area 42 in order to perform (first operation) ｎ (For example, the image data ID in Figure 11) １ The image acquisition unit 106 obtains the nth step of the work WK. ｎ Next, the (n+1)th step of work WK for the workpiece W in the next work area 44. ｎ＋１ The ID of the second image data showing the next work area 44, which was captured in order to perform the (second operation) ｎ＋１ (For example, the image data ID in Figure 12) ２ Obtain ') (step S12).

[0074] The composite image generation unit 108 generates the first image data ID ｎ 'and the second image data ID ｎ＋１ ' and combined composite image data CD ｎ (For example, the composite image data CD of Figure 13) １ (Step S13) generates the composite image data CD. Then, the position detection unit 104 generates the composite image data CD. ｎ Based on this, the (n+1)th stage of work WK ｎ＋１ The position P of the target workpiece W is detected (step S14).

[0075] With this configuration, the (n+1)th stage of work WK ｎ＋１ The position P of the target workpiece W can be detected with high accuracy. This effect is explained below. For example, image data ID １ (Figure 11) and image data ID ２ The composite image data CD shown in Figure 14 is obtained by combining (Figure 12) and (Figure 14). １ In this case, image data ID １ 'and image data ID ２ At the boundary with (i.e., overlapping region 46), workpiece W in Figure 14 ４ It exists.

[0076] Also, image data ID ２ (Figure 12) and image data ID ３ The composite image data CD shown in Figure 22 is obtained by combining (Figure 20) and (Figure 22). ２ In this case, image data ID ２ 'and image data ID ３ At the boundary with (overlapping region 46), workpiece W in Figure 22 ８ It exists. According to this embodiment, the composite image data CD ｎ By detecting work W from, two image data IDs ｎ 'and ID ｎ＋１ Work W located at the boundary of ' ４ , W ８ Since the position P can be reliably detected, the detection accuracy of position P can be improved.

[0077] Furthermore, in the device 100, the image acquisition unit 106 is used when the robot 12 is performing the nth step of the work WK ｎ By causing the visual sensor 14 to image the next work area 44 while this is being performed, the second image data ID ｎ＋１ Obtain ' (step S11). According to this configuration, the nth stage of work WK ｎ Simultaneously, the next work area 44 is imaged, and the composite image data CD is created. ｎ This enables the generation of the next work area 44 and the detection of the workpiece W's position P within that area. This reduces the cycle time of the flow shown in Figure 3.

[0078] Furthermore, in the device 100, the area setting unit 102 automatically sets the work area 42 and the next work area 44 in the control coordinate system C (specifically, the sensor coordinate system C3) for controlling the robot 12. With this configuration, the setting of the work area 42 and the next work area 44 can be automated, thereby reducing the burden on the operator.

[0079] Furthermore, in the device 100, the area setting unit 102 sets the work area 42 to encompass one predetermined area in the control coordinate system C (sensor coordinate system C3), and sets the next work area 44 to encompass another predetermined area in the control coordinate system C. Then, when the position detection unit 104 detects position P (step S14), the area setting unit 102 resets the new work area 42 and next work area 44 (Figure 16) by swapping the set work area 42 and the next work area 44 with each other (step S7). With this configuration, the loop of steps S3 to S7 in Figure 3 can be executed quickly and efficiently with a relatively simple algorithm.

[0080] Furthermore, in the device 100, the image acquisition unit 106 captures a new second image data ID of the next work area 44 (Figure 16) after resetting. ｎ＋２ (For example, the image data ID in Figure 20) ３ The composite image generation unit obtains the ID of the second image data captured before the reset. ｎ＋１ (For example, the image data ID in Figure 12) ２ ') to a new first image data ID ｎ＋１ ' to be used as the new first image data ID ｎ＋１ 'and a new second image data ID ｎ＋２ By combining ' and , composite image data CD ｎ＋１ (For example, the composite image data CD of Figure 22) ２ This configuration generates a composite image data CD. ｎ＋１ It can be generated efficiently.

[0081] Furthermore, in the device 100, the work area 42 and the next work area 44 are located within the field of view 40 of the visual sensor 14, and the image acquisition unit 106 receives image data IDs of subjects within the field of view 40 captured by the visual sensor 14.ｎ In Figures 4, 9, and 18, the pixels IE that capture the work area 42 (for example, point clouds in 3D point cloud image data, or pixels in distance image data) are disabled (for example, by removing the point cloud, or by setting the brightness α of the pixels to a predetermined value α). ０ By setting the second image data ID to ( ), ｎ＋１ Obtain the second image data ID. According to this configuration, the second image data ID ｎ＋１ ' can be obtained efficiently.

[0082] Furthermore, in the device 100, the work area 42 and the next work area 44 are determined to overlap with each other in the overlapping area 46, and the composite image generation unit 108 generates the first image data ID ｎ 'The first pixel IE that depicts the overlapping region 46 １ And the second image data ID ｎ＋１ 'The second pixel IE that depicts the overlapping region 46 ２ After invalidating one of them in accordance with the predetermined rule RL2, the first image data ID ｎ 'and the second image data ID ｎ＋１ ' and are combined. According to this configuration, the combined image data CD ｎ Since the amount of computational processing required to generate the result can be reduced, the computational processing can be sped up.

[0083] Furthermore, as an example of the device 100, a predetermined rule RL2 is the first pixel IE １ and the second pixel IE ２ This includes a rule that disables the one with the larger distance δ from the coordinates in the control coordinate system C to the visual sensor 14. According to this configuration, the pixel IE closest to the visual sensor 14 １ or IE ２ Composite image data CD to have only ｎ It can generate.

[0084] Another example of the device 100 is the first image data ID ｎ ' and second image data ID ｎ＋１ ' ' includes a rule that a priority order is set, and the predetermined rule RL2 invalidates one of the others according to the priority order. According to this configuration, the operator can create a composite image data CD ｎ Pixel IE that constitutes the１ or IE ２ These can be arbitrarily selected by setting priorities.

[0085] In the above embodiment, the processor 32 determines when the robot 12 performs the task WK ｎ The case described above involves having the visual sensor 14 capture an image of the next work area 44 in step S11 while the process is being executed. However, the processor 32 is not limited to this case and may perform the imaging in step S11 in Figure 8 at any timing. For example, the processor 32 may perform the imaging in step S11 in step S4 of the work WK ｎ Step S11 may be executed when the process is completed.

[0086] Note that the work area 42 and the next work area 44 are not limited to the example shown in Figure 5, and may be set in any shape (circular, polygonal, elliptical, etc.) and at any position. Also, the work area 42 and the next work area 44 may be set in advance by the operator. In this case, the area setting unit 102 can be omitted from the device 100.

[0087] In the above embodiment, the work area 42 and the next work area 44 are set to fit within the field of view 40 of one visual sensor 14, and in the image data ID captured by the visual sensor 14 within the field of view 40, the pixel IE that captures the work area 42 is disabled, thereby creating the second image data ID ｎ＋１ This section described the case where ' is obtained'.

[0088] However, the system is not limited to this. For example, a first visual sensor 14A that takes the work area 42 in Figure 5 into its field of view and a second visual sensor 14B that takes the next work area 44 into its field of view may be provided. In this case, the first visual sensor 14A receives the first image data ID １ The second visual sensor 14B captures the second image data ID ｎ＋１ You may image '.

[0089] In the above embodiment, the processor 32 displays the first pixel IE that represents the overlapping region 46. １ and the second pixel IE ２The case where one of the two is disabled was described. However, the processor 32 is not limited to this, and the pixel IE １ and IE ２ Composite image data CD having both of the above ｎ Alternatively, the processor 32 may generate a composite image data CD. ｎ The pixels are quantized (i.e., voxelized), and the number of pixels per voxel is made uniform, so that the pixel IE is uniform. １ or IE ２ You may disable it.

[0090] Furthermore, the above-described embodiment described a case in which the work area 42 and the next work area 44 are defined to overlap with each other in the overlapping area 46. However, the processor 32 is not limited to this, and may set the work area 42 and the next work area 44 so that the boundary of the work area 42 coincides with the boundary of the next work area 44. In this case, the overlapping area 46 does not occur.

[0091] Furthermore, the above-described embodiment described a case where the work area 42 and the next work area 44 are set to fit within the field of view 40 of the visual sensor 14 (in other words, as areas smaller than the field of view 40). However, the invention is not limited to this, and the work area 42 and the next work area 44 may be set to be the same size as the field of view 40, or they may be set to be larger than the field of view 40.

[0092] In step S2 described above, the processor 32 outputs the image data ID １ (Figure 5) Of the workpieces W shown, model matching MT is performed on the workpieces W in the work area 42, but it is not necessary to perform model matching MT on the workpieces W in the next work area 44. In this case, the processor 32, in step S2, the image data ID １ The next work area 44 is disabled, and the image data ID of work area 42 is disabled. １ You may generate (Figure 11).

[0093] Similarly, in step S14 described above, the processor 32 creates the synthesized image data CD. ｎ(Figure 14) Of the workpieces W shown, model matching MT is performed on the workpieces W in the next work area 44, while model matching MT does not need to be performed on the workpieces W in the work area 42. In this case, in step S14, the processor 32 generates the composite image data CD. ｎ Alternatively, the pixel IE that captures the work area 42 may be disabled, and image data that captures the next work area 44 may be generated.

[0094] Furthermore, in steps S2 and S14 described above, the processor 32 controls a plurality of workpieces W ｊ When detecting one workpiece W ｊ After detecting the one workpiece W, ｊ Other workpieces W located at a predetermined distance Δ or more away from ｊ＋１ It may also be detected. That is, in this case, the two workpieces W that were detected ｊ and W ｊ＋１ They are separated from each other by a distance of Δ or more.

[0095] Here, in step S4, one workpiece W ｊ When the workpieces are picked up, the arrangement of the workpieces W inside container A may change due to load shifting or other reasons. ｊ and W ｊ＋１ By detecting other workpieces W ｊ＋１ Since the impact of changes in arrangement on the picking up process is reduced, the other workpiece W ｊ＋１ It can pick them up with high precision.

[0096] Furthermore, in the above-described embodiment, in steps S2 and S14, a plurality of workpieces W ｊ Coordinates Ps ｊ , Pr ｊ The case in which a workpiece W is detected has been described. However, it is not limited to this, and in steps S2 and S14, the processor 32 detects a workpiece W ｊ Coordinates Ps ｊ , Pr ｊ Only the workpiece W detected in step S14 may be detected. In this case, the processor 32 detects only the workpiece W ｊ Coordinates Pr ｊThis may be stored in the position database 48 for the work area 42. In other words, in this case, the position database 56 for the next work area 44 can be omitted.

[0097] Next, other functions of the robot system 10 will be described with reference to Figure 23. In this embodiment, the device 100 further includes an occupied area estimation unit 110 in addition to the area setting unit 102, position detection unit 104, image acquisition unit 106, and composite image generation unit 108. Hereinafter, other examples of the operation of the robot system 10 will be described with reference to Figure 24. In the flow chart of Figure 24, the same step numbers are used for processes similar to those in the flow chart of Figure 3, and redundant explanations are omitted.

[0098] After the start of the flow in Figure 24, the processor 32, similar to the embodiment described above, captures the workpiece W in step S1 and generates the image data ID shown in Figures 4 and 5. １ This is obtained. Then, in step S2, the processor 32 functions as a position detection unit 104 and detects the position P of the workpiece W (coordinates Ps in the sensor coordinate system C3).

[0099] In this embodiment, the processor 32 detects the coordinates Ps of one workpiece W in step S2. Hereafter, in step S2, the workpiece W in Figure 5 １ Coordinates Ps １ The following describes the case where the detected work W is detected. １ Coordinates Ps １ Convert to robot coordinate system C1, and the coordinates Pr of robot coordinate system C1 １ The processor 32 then obtains the coordinate Pr １ This is stored in the location database 48 (Figure 6).

[0100] In step S21, the processor 32 performs the first stage of work WK １ Movement path MP of robot 12 when executing １ Based on the work WK １ The processor estimates the area 64 (Figure 26) occupied by the robot 12 during execution. Specifically, the processor estimates the coordinates Pr stored in the position database 48 at this time. １Based on, Work W １ Movement path MP of robot 12 when picking up １ Obtain it.

[0101] As shown in Figure 25, the processor 32 sends the workpiece W to the robot 12. １ When picking up (Figure 5), the end effector 28 moves along the MP path. １ Move along the coordinates Pr １ The processor 32 positions the end effector 28 at coordinate Pr １ Operation command CM for positioning １ The operation command CM is calculated and the operation command CM １ From the travel route MP １ We seek.

[0102] Meanwhile, the processor 32 acquires a robot model 12M (for example, a 3D CAD model) that models the robot 12. Then, the processor 32 combines the robot model 12M with the movement path MP. １ Based on this, the robot model 12M moves along the path MP in the robot coordinate system C1. １ The area 64 occupied by the robot model 12M during simulated movement along the line is estimated.

[0103] The processor 32 converts the calculated occupied area 64 into the sensor coordinate system C3 and defines the occupied area 64 in the sensor coordinate system C3. An example of the occupied area 64 estimated in this way is shown in Figure 26. Thus, in this embodiment, the processor 32 functions as an occupied area estimation unit 110 (Figure 2) that estimates the occupied area 64 of the robot 12 based on the movement path MP1 of the robot 12.

[0104] In step S22, the processor 32 functions as a region setting unit 102 to set a work area 66 and a next work area 68 adjacent to the work area 66. Specifically, the processor 32 sets the occupied area 64 estimated in step S21 as the work area 66, while setting the area other than the occupied area 64 as the next work area 68. The work area 66 and the next work area 68 set in this way are shown in Figure 27. The coordinates of the sensor coordinate system C3 of each vertex defining the next work area 68 may be predetermined by the operator.

[0105] After step S22, the processor 32 executes steps S3 and S4. In step S4, the processor 32 uses the coordinates Ps stored in the position database 48 at this point in time. １ Based on this, the first stage of work WK １ For example, the workpiece W in the work area 66 １ The process of picking up the items begins. Next, as step S5, the processor 32 executes the flow shown in Figure 8.

[0106] Specifically, in step S11, the processor 32 operates the visual sensor 14 to image the workpiece W inside container A. At this time, the processor 32, as in the embodiment described above, performs the workpiece WK in the preceding step S4. １ Step S11 may be performed when the system starts. Image data ID acquired in step S11 ２ An example is shown in Figure 28.

[0107] In step S12, the processor 32 functions as an image acquisition unit 106 and acquires the first image data ID １ 'and the second image data ID ２ Specifically, the processor 32 obtains the first image data ID. １ 'As the first stage of work WK １ Image data ID captured to perform the operation １ (Figure 27) Image data ID of the work area 66 １ ' is obtained. At this time, the processor 32, similar to the embodiment described above, obtains the image data ID. １By disabling the pixel IE that captures the next work area 68, the image data ID that captures the work area 66 １ Obtain the image data ID. １ An example of this is shown in Figure 29.

[0108] Meanwhile, processor 32 performs the second stage of work WK ２ Image data ID captured to perform the operation ２ (Figure 28) Image data ID of the next work area 68 ２ ' is obtained. At this time, the processor 32, similar to the embodiment described above, obtains the image data ID. ２ In this process, by disabling the pixel IE that captures the work area 66, the image data ID that captures the next work area 68 is generated. ２ Obtain the image data ID. ２ An example of this is shown in Figure 30.

[0109] In step S13, the processor 32 functions as a composite image generation unit 108 and generates the first image data ID １ 'and the second image data ID ２ ' and combined composite image data CD １ Generates a composite image data CD. １ An example is shown in Figure 31. In step S14, the processor 32 functions as a position detection unit 104 and generates the composite image data CD in the preceding step S13. １ Based on this, the second stage of work WK ２ The position P of the target workpiece W is detected. Specifically, as shown in Figure 32, the composite image data CD １ The sensor coordinate system C3 that defines this includes a work area 66 and a next work area 68.

[0110] The processor 32 synthesizes image data CD using model matching MT. １ From the coordinates Ps of the sensor coordinate system C3 of multiple workpieces W detected, the coordinates Ps that are within the next work area 68 are extracted. For example, in the example shown in Figure 32, the processor 32 identifies the coordinates Ps within the next work area 68 as workpiece W ２ Coordinates Ps ２Assume that it has detected the coordinates Ps of the detected sensor coordinate system C3. ２ Convert to robot coordinate system C1, workpiece W ２ The coordinates of the robot coordinate system C1 are Pr ２ The processor 32 then obtains the obtained coordinates Pr ２ This is the work WK for the next work area 68. ２ It is stored in the location database 56 (Figure 15).

[0111] Referring again to Figure 24, when YES is determined in step S6, in step S23, the processor 32 functions as the occupied area estimation unit 110 and performs the second stage of work WK ２ Movement path MP of robot 12 when executing ２ Based on the work WK ２ The robot 12's occupied area 70 (Figure 34) during execution is estimated. Specifically, the processor 32 refers to the position database 56 for the next work area 68 at this point and the coordinates Pr detected in the previous step S14. ２ The processor 32 then obtains the coordinate Pr ２ Based on, Work W ２ (Figure 32) Movement path MP of robot 12 when picking up ２ Obtain it.

[0112] As shown in Figure 33, the processor 32 sends the workpiece W to the robot 12. ２ When picking up (Figure 32), the end effector 28 moves along the MP path. ２ Move along the coordinates Pr ２ The processor 32 positions the end effector 28 at coordinate Pr ２ Operation command CM for positioning ２ The operation command CM is calculated and the operation command CM ２ From, travel route MP ２ The processor 32 then calculates the robot model 12M and the movement path MP. ２ Based on this, the occupied area 70 in the sensor coordinate system C3 is estimated. An example of the occupied area 70 estimated in this way is shown in Figure 34.

[0113] In step S24, the processor 32 functions as a region setting unit 102 to reset the work area 66 and the next work area 68. Specifically, the processor 32 resets the occupied area 70 estimated in the most recent step S23 as the work area 66, while resetting the area other than the occupied area 70 as the next work area 68. The reset work area 66 and the next work area 68 are shown in Figure 35. The processor 32 also uses the coordinates Pr stored in the position database 56 for the next work area 68 at this point. ２ This is then transferred to the location database 48 for the work area 66.

[0114] After step S24, the processor 32 returns to step S3 and then executes a second step S4. In this step S4, the processor 32 processes the work W ２ In contrast, the second stage of work WK ２ The process begins. Then, the processor 32 executes the second step S5. Specifically, in step S11, the processor 32 operates the visual sensor 14 to image the workpiece W. The image data ID captured by the visual sensor 14 in the second execution of step S11. ３ An example is shown in Figure 36. In the example shown in Figure 36, the work WK was performed in the preceding step S4. ２ Image data ID acquired after the start ３ Work WK ２ The robot 12, which is performing the operation, is visible in the image.

[0115] In the second step S12, the processor 32 functions as an image acquisition unit 106 and the first image data ID ２ 'and the second image data ID ３ Specifically, the processor 32 obtains the first image data ID. ２ 'Image data ID captured before resetting in step S24' ２ (Figure 30) is used. Therefore, the processor 32 generates the image data ID in the first step S12. ２ The second image data ID is read from memory 34 and obtained. Meanwhile, the processor 32 reads the second image data ID ３ 'The third stage of work WK ３Image data ID acquired to perform the operation ３ (Figure 36) Image data ID of the next work area 68 (Figure 35) ３ Obtain the image data ID. ３ An example of this is shown in Figure 37.

[0116] In the second step S13, the processor 32 functions as a composite image generation unit 108 and generates image data ID in the manner described above. ２ 'and ID ３ By combining ', the combined image data CD ２ Generates a composite image data CD. ２ An example of this is shown in Figure 38. Then, in the second step S14, the processor 32 functions as a position detection unit 104 and the synthesized image data CD ２ Based on this, the third stage of work WK ３ The position P of the target workpiece W is detected. Synthetic image data CD ２ The sensor coordinate system C3 that defines this has defined the reset work area 66 and the next work area 68 (Figure 35).

[0117] The processor 32 detects the position P of the workpiece W in the next work area 68 after resetting. As a result, the processor 32 determines the coordinates Ps within the next work area 68 to be the workpiece W located at the boundary between the work area 66 and the next work area 68 as shown in Figure 35. ３ Coordinates Ps ３ The processor 32 can detect the coordinates Ps of the detected sensor coordinate system C3. ３ Convert to robot coordinate system C1, and the coordinates Pr of robot coordinate system C1 ３ This is then newly stored in the position database 48 for the work area 66. In this way, while the processor 32 determines YES in step S3, it repeatedly executes the loop of steps S3 to S6, S23 and S24.

[0118] As described above, in the apparatus 100 according to this embodiment, the occupied area estimation unit 110 performs the nth step of the work WK ｎ Movement path MP of robot 12 when executing ｊ Based on the work WK ｎThe robot 12's occupied areas 64 and 70 during execution are estimated (steps S21 and S23). The area setting unit 102 then sets the occupied areas 64 and 70 as the work area 66, while setting the areas other than the occupied areas 64 and 70 as the next work area 68 (steps S22 and S24). With this configuration, the operating range of the robot 12 during operation can be set as the work area 66, so the size of the work area 66 can be optimized.

[0119] The processor 32 may execute the flow shown in Figure 3 or Figure 24 according to the computer program PG stored in memory 34. Furthermore, the functions of the device 100 executed by the processor 32 may be functional modules realized by the computer program PG. The computer program PG may also be provided as a program product recorded in memory.

[0120] In the above-described embodiment, the processor 32 performs one workpiece W in step S2 in Figure 24 and step S14 performed in step S5 in Figure 24. １ , W ２ , W ３ The case in which the above is detected has been described. However, the processor 32 may, in steps S2 and S14, detect a plurality of workpieces W ｊ Coordinates Pr ｊ It may be detected.

[0121] In this case, the processor 32, in the subsequent steps S22 and S24, each coordinate Pr ｊ Operation command CM for positioning the end effector 28 ｊ The operation command CM is calculated and the operation command CM ｊ Multiple travel routes MP ｊ The processor 32 then calculates the robot model 12M and multiple movement paths MP. ｊ Based on this, the robot model 12M moves along each movement path MP in the robot coordinate system C1. １ The area occupied by the robot model 12M during simulated movement along the specified path is estimated.

[0122] For example, in step S2 in Figure 24, the processor 32 generates an image data ID １ In Figure 5, workpiece W １ And, workpiece W in Figure 32 ２ Assume that the above is detected. In this case, in step S21, the processor 32 determines the travel path MP １ Based on (Figure 25), the occupied area 64 (Figure 26) is estimated, and the movement path MP is also estimated. ２ The occupied area 70 is estimated based on (Figure 33). Then, in step S22, the processor 32 sets the occupied areas 64 and 70 as the work area 66, while setting the area other than the occupied areas 64 and 70 as the next work area 68.

[0123] The processor 32 may also provide an overlapping area in steps S22 and S24. The work area 66 and the next work area 68 may be set to have an overlapping area. The disabling of pixels IE in the overlapping area may be the same as in the embodiment described with reference to Figure 3.

[0124] In the above embodiment, the case in which the processor 32 creates the position databases 48 and 56 separately was described. However, the processor 32 is not limited to this case, and the coordinates Pr detected in step S14 ｊ The coordinates Pr detected in step S14 may be stored in the location database 48. In this case, the processor 32 will store the coordinates Pr ｊ In contrast, the workpiece W in the next work area 44 ｊ Identifying information indicating that it is such may be added.

[0125] In the above-described embodiment, the case was described in which the processor 32 sets the work areas 42 and 66 and the next work areas 44 and 68 to the sensor coordinate system C3. However, the processor 32 is not limited to this, and may set the work area 42 or 66 and the next work area 44 or 68 to the robot coordinate system C1 or the tool coordinate system C2. Also, in the above-described embodiment, the processor 32 sets the coordinates Ps of the sensor coordinate system C3 to the position database 48 or 56. ｊ You may store it.

[0126] Furthermore, the above-described embodiment described the case where the functions of the device 100 are implemented in the control device 16. However, the functions of the device 100 are not limited to this, and may be implemented in any computer, such as a higher-level controller of the control device 16, or a teaching device for teaching the robot 12 how to operate.

[0127] Furthermore, the robot 12 is not limited to a vertical articulated robot, but may be any type of robot, such as a horizontal articulated robot or a parallel link robot. Also, the operation WK performed by the robot 12 is not limited to work handling, such as picking up a workpiece W, but may be any operation WK, such as coating, welding, or laser processing.

[0128] Although the present disclosure has been described in detail above, it is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the gist of the present disclosure or from the spirit of the present disclosure derived from the claims and their equivalents. Furthermore, these embodiments can be implemented in combination. For example, the order of operations and processes in the embodiments described above are shown as examples only and are not limited thereto. The same applies when numerical values ​​or mathematical formulas are used in the description of the embodiments described above.

[0129] As described above, this disclosure describes the following embodiments: (Embodiment 1) A device 100 for detecting the position P of a workpiece W based on image data ID captured by a visual sensor 14 in order for a robot 12 to perform a work WK on the workpiece W, wherein the device detects the position P of a workpiece W within the work areas 42, 66. ｎ First image data ID capturing the work area 42, 66, which was imaged in order to perform the operation. ｎ 'and the first work WK ｎ Next, a second operation WK is performed on the workpiece W in the adjacent work areas 44 and 68, which are adjacent to the work areas 42 and 66. ｎ＋１ A second image data ID that captures the next work area 44, 68, which was imaged in order to perform the next operation. ｎ＋１Image acquisition unit 106 that acquires the first image data ID ｎ 'and the second image data ID ｎ＋１ ' and combined composite image data CD ｎ A composite image generation unit 108 that generates composite image data CD ｎ Based on this, the second work WK ｎ＋１ The apparatus 100 includes a position detection unit 104 that detects the position P of the workpiece W to be performed. (Aspect 2) The image acquisition unit 106 is used when the robot 12 performs the first work WK ｎ By causing the visual sensor 14 to image the next work area 44, 68 while this is being performed, the second image data ID ｎ＋１ Apparatus 100 according to Embodiment 1, which acquires '. (Embodiment 3) Apparatus 100 according to Embodiment 1 or 2, further comprising an area setting unit 102 that automatically sets work areas 42, 66 and next work areas 44, 68 in a control coordinate system C for controlling the robot 12. (Embodiment 4) Apparatus 100 according to Embodiment 3, wherein the area setting unit 102 sets work area 42 to include one predetermined area in the control coordinate system C, sets next work area 44 to include another predetermined area in the control coordinate system C, and when the position detection unit 104 detects a position P, it resets the set work area 42 and next work area 44 by swapping them with each other. (Embodiment 5) Image acquisition unit 106 acquires a new second image data ID of the next work area 44 after resetting. ｎ＋２ The composite image generation unit 108 obtains the second image data ID captured before resetting. ｎ＋１ ' to a new first image data ID ｎ＋１ ' to be used as the new first image data ID ｎ＋１ 'and a new second image data ID ｎ＋２ By combining ' and , composite image data CD ｎ＋１ Apparatus 100 according to Embodiment 4, which generates (Embodiment 6) First operation WK ｎ Movement path MP of robot 12 when executing ｊ Based on the first work WK ｎThe apparatus 100 according to Embodiment 3, further comprising an occupied area estimation unit 110 that estimates the occupied areas 64 and 70 of the robot 12 during execution, and an area setting unit 102 that sets the occupied areas 64 and 70 as a work area 66, while setting the area other than the occupied areas 64 and 70 as the next work area 68. (Embodiment 7) The work area and the next work area are contained within the field of view of a visual sensor, and the image acquisition unit acquires a second image data by disabling pixels that capture the work area in the image data captured by the visual sensor of a subject within the field of view. (Embodiment 8) The work areas 42 and 66 and the next work areas 44 and 68 are determined to overlap with each other in the overlapping area 46, and the composite image generation unit 108 sets the first image data ID ｎ 'The first pixel IE that depicts the overlapping region 46 １ And the second image data ID ｎ＋１ 'The second pixel IE that depicts the overlapping region 46 ２ After invalidating one of them in accordance with the predetermined rule RL2, the first image data ID ｎ 'and the second image data ID ｎ＋１ Apparatus 100 according to any one of embodiments 1 to 7, which synthesizes ' and . (Embodiment 9) Pixel IE is represented as coordinates of a control coordinate system C for controlling the robot 12, and a predetermined rule RL2 is the first pixel IE １ and the second pixel IE ２ The apparatus 100 according to embodiment 8, which includes a rule to disable the one with the larger distance δ from the coordinates in the control coordinate system C to the visual sensor 14. (Embodiment 10) First image data ID ｎ ' and second image data ID ｎ＋１Apparatus 100 according to Embodiment 8, wherein a priority order is set, and a predetermined rule RL2 includes a rule that invalidates one of the priorities according to the priority order. (Embodiment 11) A control device 16 that controls a robot 12, comprising the apparatus 100 according to any one of Embodiments 1 to 10. (Embodiment 12) A robot system 10 comprising a robot 12 that performs a task WK on a workpiece W, a vision sensor 14 that images the workpiece W for the task WK, and the control device 16 according to Embodiment 11. (Embodiment 13) A method for detecting the position P of a workpiece W based on image data IDs captured by the vision sensor 14 for the robot 12 to perform a task WK on the workpiece W, wherein a processor 32 performs a first task WK on the workpiece W in a work area 42, 66 ｎ First image data ID capturing the work area 42, 66, which was imaged in order to perform the operation. ｎ 'and the first work WK ｎ Next, a second operation WK is performed on the workpiece W in the adjacent work areas 44 and 68, which are adjacent to the work areas 42 and 66. ｎ＋１ A second image data ID that captures the next work area 44, 68, which was imaged in order to perform the next operation. ｎ＋１ ' and obtain the first image data ID ｎ 'and the second image data ID ｎ＋１ ' and combined composite image data CD ｎ Generates a composite image data CD. ｎ Based on this, the second work WK ｎ＋１ A method for detecting the position P of the target workpiece W. (Aspect 14) A computer program PG that causes a processor 32 to execute the method described in Aspect 13.

[0130] 10 Robot system 12 Robot 14 Vision sensor 16 Control device 40 Field of view 42, 66 Work area 44, 68 Next work area 46 Overlapping area 100 Device 102 Area setting unit 104 Position detection unit 106 Image acquisition unit 108 Composite image generation unit 110 Occupied area estimation unit

Claims

1. A device for detecting the position of a workpiece based on image data captured by a vision sensor for a robot to perform work on the workpiece, comprising: an image acquisition unit that acquires first image data capturing the work area for performing a first operation on the workpiece within the work area, and second image data capturing the next work area for performing a second operation on the workpiece in the next work area adjacent to the work area, following the first operation; a composite image generation unit that generates composite image data by combining the first image data and the second image data; and a position detection unit that detects the position of the workpiece that is the target of the second operation based on the composite image data.

2. The apparatus according to claim 1, wherein the image acquisition unit acquires the second image data by causing the vision sensor to image the next work area while the robot is performing the first work.

3. The apparatus according to claim 1, further comprising a region setting unit for automatically setting the work area and the next work area in a control coordinate system for controlling the robot.

4. The apparatus according to claim 3, wherein the area setting unit sets the work area to include one predetermined area in the control coordinate system, sets the next work area to include another predetermined area in the control coordinate system, and when the position detection unit detects the position, it resets the new work area and the next work area by swapping the set work area and the next work area with each other.

5. The apparatus according to claim 4, wherein the image acquisition unit acquires new second image data obtained by imaging the next work area after the reset, and the composite image generation unit generates the composite image data by using the second image data captured before the reset as new first image data and compositing the new first image data with the new second image data.

6. The apparatus according to claim 3, further comprising an occupied area estimation unit that estimates the area occupied by the robot during the execution of the first operation based on the movement path of the robot when the first operation is performed, wherein the area setting unit sets the occupied area as the work area and sets an area other than the occupied area as the next work area.

7. The apparatus according to claim 1, wherein the work area and the next work area are located within the field of view of the visual sensor, and the image acquisition unit acquires the second image data by disabling the pixels that capture the work area in the image data captured by the visual sensor of a subject within the field of view.

8. The apparatus according to claim 1, wherein the work area and the next work area are determined to overlap with each other in an overlapping area, and the composite image generation unit invalidates one of the first pixels that represent the overlapping area in the first image data and the second pixels that represent the overlapping area in the second image data according to a predetermined rule, and then combines the first image data and the second image data.

9. The apparatus according to claim 8, wherein the pixels are represented as coordinates in a control coordinate system for controlling the robot, and the predetermined rule includes a rule that invalidates one of the first and second pixels, which has a larger distance from its coordinates in the control coordinate system to the visual sensor.

10. The apparatus according to claim 8, wherein the first image data and the second image data are assigned a priority order, and the predetermined rule includes a rule that invalidates one of them according to the priority order.

11. A control device comprising the apparatus described in claim 1, for controlling the robot.

12. A robot system comprising: a robot that performs work on a workpiece; a visual sensor that images the workpiece for the purpose of performing the workpiece; and the control device according to claim 11.

13. A method for detecting the position of a workpiece based on image data captured by a vision sensor for a robot to perform an operation on the workpiece, the method comprising: a processor acquiring first image data of a work area captured for performing a first operation on the workpiece within the work area, and second image data of a next work area captured for performing a second operation on the workpiece in a next work area adjacent to the work area, following the first operation; generating composite image data by combining the first image data and the second image data; and detecting the position of the workpiece that is the target of the second operation based on the composite image data.

14. A computer program that causes the processor to execute the method described in claim 13.

Images