Robot control device and robot system

Abstract

Provided is a robot control device that controls a robot that extracts a workpiece on the basis of three-dimensional point cloud data which is generated by a 3D vision sensor and which indicates, through a plurality of three-dimensional points, a surface shape of a subject region in which the workpiece may exist, the robot control device comprising: a grouping unit that classifies the plurality of three-dimensional points into a plurality of groups existing in proximity on the same virtual plane; an obstacle model generation unit that generates an obstacle model composed of a hexahedron in which a square on the virtual plane that envelopes the three-dimensional points belonging to the same group is defined as the upper surface and a square obtained by vertically projecting the upper surface onto a surface on which the workpiece is placed is defined as the bottom surface; and an operation configuration unit that configures operation of the robot for extracting the workpiece so as not to interfere with the obstacle model.

Description

Robot control device and robot system

[0001] The present disclosure relates to a robot control device and a robot system.

[0002] A system that controls a work picking operation by a robot based on three-dimensional point cloud data acquired by a vision sensor is known. In such a system, the operation of the robot is set so that the robot and the work to be picked up do not interfere with the three-dimensional points of the three-dimensional point cloud data. The number of three-dimensional points in the three-dimensional point cloud data is generally 100,000 or more, and some exceed 1 million, and it is expected to further increase with the increase in the resolution of the vision sensor.

[0003] In the conventional system, in order to check the presence or absence of interference for each of all the three-dimensional points one by one, when the number of three-dimensional points increases, the calculation load increases and the processing time becomes long. Therefore, it has also been proposed to thin out the three-dimensional points and check the presence or absence of interference (see, for example, Patent Document 1).

[0004] Japanese Patent No. 7433501

[0005] When thinning out the three-dimensional points as in Patent Document 1, the accuracy of the interference determination decreases, so the three-dimensional points cannot be significantly thinned out. Therefore, with the method of Patent Document 1, it is difficult to significantly reduce the calculation load related to the interference determination. A technique that can reduce the load of the interference determination in the work extraction operation is desired.

[0006] A robot control device according to one aspect of the present disclosure is a robot control device that controls a robot to retrieve a workpiece based on three-dimensional point cloud data generated by a 3D vision sensor, which indicates the surface shape of a subject area where a workpiece may exist using a plurality of three-dimensional points, and comprises: a grouping unit that classifies the plurality of three-dimensional points into a plurality of groups that are located close together on the same virtual plane; an obstacle model generation unit that generates an obstacle model consisting of a hexahedron whose upper surface is a rectangle on the virtual plane that encloses the three-dimensional points belonging to the same group, and whose bottom surface is a rectangle obtained by projecting the upper surface perpendicularly onto the surface on which the workpiece is placed; and an operation setting unit that sets the operation of the robot to retrieve the workpiece so as not to interfere with the obstacle model.

[0007] This is a schematic diagram showing the configuration of a robot system according to the first embodiment of this disclosure. This is a flowchart of the workpiece removal procedure in the robot system of Figure 1. This is a flowchart of a workpiece removal procedure in the robot system of Figure 1 that is different from that in Figure 2.

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Figure 1 is a schematic diagram showing the configuration of a robot system 1 according to the first embodiment of the present disclosure. The robot system 1 takes out workpieces W, which are randomly stored in a container C, one by one.

[0009] The robot system 1 includes a 3D vision sensor 10 that generates 3D point cloud data indicating the surface shape of a subject area where a workpiece W may exist using a plurality of 3D points, a robot 20 that retrieves the workpiece, and a robot control device 30 that controls the robot 20 based on the 3D point cloud data provided by the 3D vision sensor 10. The robot control device 30 itself is one embodiment of the robot control device according to this disclosure.

[0010] The 3D vision sensor 10 measures the distance to the subject for each 2D position and generates 3D point cloud data that represents the combination of 2D position and distance as 3D points. The 3D vision sensor 10 may be fixed above the container C as in this embodiment, or it may be attached to the robot 20.

[0011] The robot 20 has a hand 21 at its end for holding a workpiece W, and is configured to determine the position and orientation of the hand 21. The robot 20 can be a vertical articulated robot as illustrated in Figure 1, but is not limited to this, and may be a Cartesian coordinate robot, a SCARA robot, a parallel link robot, etc. The hand 21 may be configured to have, for example, gripping fingers for grasping the workpiece W, a suction pad for adsorbing the workpiece W, etc.

[0012] The robot control device 30 can be implemented by one or more computer devices that have, for example, memory, a CPU, an input / output interface, etc., and execute an appropriate program. The robot control device 30 includes a data acquisition unit 31, a workpiece recognition unit 32, a target determination unit 33, a workpiece model placement unit 34, a grouping unit 35, an obstacle model generation unit 36, an operation setting unit 37, and an operation execution unit 38. Note that these components of the robot control device 30 are categorized by their functions and do not necessarily need to be clearly distinguishable in terms of physical structure and program structure.

[0013] The data acquisition unit 31 acquires three-dimensional point cloud data of the subject area, including the container C containing the workpiece W, from the 3D vision sensor 10. For this purpose, the data acquisition unit 31 may be configured to command the 3D vision sensor 10 to generate three-dimensional point cloud data.

[0014] The workpiece recognition unit 32 recognizes the workpiece W from the 3D point cloud data. The workpiece recognition unit 32 can be configured to perform well-known pattern recognition. Even if only workpiece W is present in the container C, depending on the shape and arrangement of the workpiece, the 3D point cloud data may contain 3D points corresponding to parts of workpiece W that cannot be recognized as workpiece W.

[0015] The target determination unit 33 determines the next target work W to be extracted from the work W recognized by the work recognition unit 32. Based on well-known technology, the target work W may be selected, for example, from among multiple recognized work W, one that is not overlapped by other work W.

[0016] The work model placement unit 34 may be configured to place work models, each modeled after a work recognized by the work recognition unit 32, into a virtual space. Alternatively, the work model placement unit 34 may be configured to place only the work model of the target work Wo determined by the target determination unit into the virtual space. The work model placement unit 34 may be configured to place work models in a virtual space where 3D points of 3D point cloud data are placed, and to delete the 3D points corresponding to the placed work models, that is, to replace the 3D points with work models.

[0017] The grouping unit 35 classifies multiple three-dimensional points into multiple groups that are located close together on the same virtual plane. Note that "located on the same virtual plane" allows for some error and can be determined by a predetermined criterion, such as using a correlation coefficient. The grouping unit 35 may be configured to group only three-dimensional points in the three-dimensional point cloud data other than the three-dimensional point corresponding to the target workpiece Wo, or three-dimensional points in the three-dimensional point cloud data other than the three-dimensional point corresponding to the workpiece W recognized by the workpiece recognition unit 32. In other words, the grouping unit 35 may group three-dimensional points that the workpiece model placement unit 34 did not replace with a workpiece model.

[0018] The obstacle model generation unit 36 ​​generates an obstacle model consisting of a hexahedron, with a rectangle on a virtual plane that encloses three-dimensional points belonging to the same group as its top surface, and a rectangle obtained by projecting the top surface perpendicularly onto the surface on which the workpiece W is placed as its bottom surface. The "rectangle on a virtual plane that encloses three-dimensional points belonging to the same group" is defined as the smallest rectangle that includes all three-dimensional points belonging to the group, includes at least one three-dimensional point on each edge, and is determined according to a predetermined algorithm. The surface on which the workpiece W is placed may be a plane at a predetermined distance, or it may be a plane defined by three-dimensional points that are determined to correspond to the bottom surface of the container C exposed in an area where the workpiece W does not exist.

[0019] The operation setting unit 37 sets the operation of the robot 20 for picking up the target workpiece Wo so as not to interfere with the obstacle model generated by the obstacle model generation unit 36. Specifically, the operation setting unit 37 determines the operation of the robot 20 so as to move the hand model so that the hand model, which is a model of the hand 21, does not interfere with the obstacle model adjacent to the workpiece model of the target workpiece Wo, and the workpiece model of the target workpiece Wo can be grasped and picked up by the hand model.

[0020] The motion execution unit 38 controls the robot 20 to execute the actions set by the motion setting unit 37. The motion execution unit 38 is a common configuration in conventional robot control devices and is well-known technology.

[0021] Figure 2 shows the procedure for removing a workpiece W in the robot system 1. This workpiece removal procedure includes a 3D point cloud data acquisition step (step S01), a workpiece recognition step (step S02), a 3D point exclusion step (step S03), a grouping step (step S04), a loop parameter setting step (step S05), an obstacle model generation step (step S06), a loop termination determination step (step S07), a loop parameter addition step (step S08), a target workpiece determination step (step S09), a workpiece model placement step (step S10), an interference determination step (step S11), a removal operation setting step (step S12), and a workpiece removal step (step S13).

[0022] In step S01, the 3D point cloud data acquisition process, the data acquisition unit 31 acquires 3D point cloud data of the subject area from the 3D vision sensor 10.

[0023] In the workpiece recognition step S02, the workpiece recognition unit 32 recognizes the workpiece W by pattern matching of the 3D point cloud data acquired by the data acquisition unit 31.

[0024] In step S03, the 3D point exclusion step, the 3D points corresponding to the workpiece W recognized by the workpiece recognition unit 32 are excluded from the 3D point cloud data.

[0025] In the grouping step S04, the 3D point cloud data, which is obtained by excluding the 3D points corresponding to the workpiece W in the 3D point exclusion step, i.e., the 3D points other than the 3D points corresponding to the workpiece W, is grouped by the grouping unit 35. The grouping unit 35 further identifies the number N of groups into which the 3D points have been classified.

[0026] In step S05, the loop parameter setting step, the loop parameter I for processing each group of 3D points classified by the grouping unit 35 is set to an initial value of 1.

[0027] In the obstacle model generation step S06, the obstacle model generation unit 36 ​​generates an obstacle model consisting of a hexahedron, with a rectangle on a virtual plane that encloses the three-dimensional points belonging to the Ith group as its top surface, and a rectangle obtained by projecting the top surface perpendicularly onto the surface on which the workpiece W is placed as its bottom surface.

[0028] In the loop termination determination step S07, it is checked whether the loop parameter I has reached the number of groups N of three-dimensional points. If I < N in the loop termination determination step, the process proceeds to step S07; if I = N, the process proceeds to step S08.

[0029] In the loop parameter addition step S08, 1 is added to I in order to perform the obstacle model generation step for the next group. In other words, after performing the loop parameter addition step, the process returns to the obstacle model generation step S06.

[0030] In step S09, the target work determination step, the target determination unit 33 selects the next target work Wo to be retrieved from among the workpieces W recognized in the workpiece recognition step.

[0031] In step S10, the work model placement process, the work model placement unit 34 places the work models of all recognized workpieces W into a virtual space where obstacle models are placed. At this time, the work model of the target workpiece Wo is set as an object to be retrieved, and the work models of workpieces W other than the target workpiece Wo are set as obstacles.

[0032] In the interference determination step S11, the operation setting unit 37 determines whether it is possible to set an operation to remove the work model of the target work Wo without the hand model and the work model of the target work Wo interfering with the obstacle model and other work models. If it is determined in this interference determination step that no operation without interference can be found, the process returns to the target work determination step S09, a target work W is selected from among the other work W, and steps S10 to S11 are executed again. If an operation without interference can be found in the interference determination step, the process proceeds to step S12.

[0033] In step S12, the extraction operation setting step, the operation that was determined not to cause interference in the interference determination step is set as the extraction operation.

[0034] In step S13, the workpiece removal process, the operation execution unit 38 executes the operation set in the operation setting process. In other words, the operation execution unit 38 actually removes the target workpiece Wo using the robot 20.

[0035] As described above, by replacing 3D points that could not be recognized as workpiece W with obstacle models, the number of objects that need to be checked for interference with obstacles indicated by 3D points that could not be recognized as workpiece Wo during the workpiece Wo retrieval process is significantly reduced. This greatly reduces the burden of interference detection during the workpiece W retrieval process.

[0036] Furthermore, Figure 3 shows the procedure for removing the workpiece W in the robot system 1, which can be used instead of Figure 2. This workpiece removal procedure includes a 3D point cloud data acquisition step (step S21), a workpiece recognition step (step S22), a target workpiece determination step (step S23), a 3D point exclusion step (step S24), a grouping step (step S25), a loop parameter setting step (step S26), an obstacle model generation step (step S27), a loop end determination step (step S28), a loop parameter addition step (step S29), a workpiece model placement step (step S30), an interference determination step (step S31), a removal operation setting step (step S32), and a workpiece removal step (step S33).

[0037] In the processing procedure shown in Figure 3, although the order of execution differs in some parts, the processing content of the data acquisition process, work recognition process, target work determination process, grouping process, loop parameter setting process, obstacle model generation process, loop termination determination process, loop parameter addition process, extraction operation setting process, and work extraction process is the same as the processing content of the process with the same name in the processing procedure shown in Figure 2.

[0038] In the processing procedure shown in Figure 3, the work recognition step in step S22 is followed by the target work determination step in step S23. If the loop processing is terminated in the loop termination determination step in step S28, the work model placement step in step S30 is executed.

[0039] In the 3D point exclusion step S24, only the 3D points corresponding to the target workpiece Wo determined in the target workpiece determination step are excluded from the 3D point cloud data. In other words, 3D points corresponding to workpieces W other than the target workpiece W are not excluded from the 3D point cloud data.

[0040] In the work model placement step S30, only the work model of the target work Wo is placed in a virtual space where obstacle models corresponding to groups of 3D points other than the 3D points corresponding to the target work Wo are placed.

[0041] In the interference determination step (step S31), it is confirmed whether an extraction operation is possible without interference between the work model and hand model of the target workpiece Wo and the obstacle model. In the procedure shown in Figure 3, there is no work model that would act as an obstacle, and interference is confirmed with a hexahedral obstacle model corresponding to a workpiece that would act as an obstacle. Also, in the procedure shown in Figure 3, if an operation that does not cause interference cannot be found in the interference determination step, the process returns to the target workpiece determination step (step S21), a target workpiece Wo is selected from among the other workpieces W, and steps S22 to S24 are executed again.

[0042] The procedure shown in Figure 3 also replaces a large number of three-dimensional points with a relatively small number of obstacle models to perform interference detection during the workpiece removal process, thereby significantly reducing the load of interference detection.

[0043] Regarding the above embodiments and modifications, the following additional remarks are further disclosed. (Supplementary Note 1) A robot control device (30) is a robot control device (30) that controls a robot (20) to extract a workpiece (W) based on three-dimensional point cloud data composed of a plurality of three-dimensional points in a subject area where the workpiece (W) generated by a 3D vision sensor (20) may exist. The robot control device (30) includes a grouping unit (36) that classifies a plurality of three-dimensional points into a plurality of groups that are close to each other on the same virtual plane, and an obstacle model generation unit (35) that generates an obstacle model composed of a hexahedron having a rectangle on a virtual plane that envelopes the three-dimensional points belonging to the same group as the upper surface and a rectangle obtained by projecting the upper surface perpendicularly to the surface on which the workpiece (W) is placed as the bottom surface. The robot control device (30) further includes an operation setting unit (37) that sets the operation of the robot (20) so as not to interfere with the obstacle model.

[0044] (Supplementary Note 2) The robot control device (30) according to Supplementary Note 1 further includes a workpiece recognition unit (32) that recognizes the workpiece (W) from the three-dimensional point cloud data, and a target determination unit (33) that determines a target workpiece (Wo) to be extracted next from the workpieces (W) recognized by the workpiece recognition unit (32). The grouping unit (36) may classify only the three-dimensional points other than the three-dimensional points corresponding to the target workpiece (Wo) in the three-dimensional point cloud data into groups.

[0045] (Supplementary Note 3) The robot control device (30) according to Supplementary Note 1 further includes a workpiece recognition unit (32) that recognizes the workpiece (W) from the three-dimensional point cloud data, a workpiece model placement unit (34) that places workpiece models obtained by modeling the workpieces (W) recognized by the workpiece recognition unit (32), and a target determination unit (33) that determines a target workpiece (Wo) to be extracted next from the workpieces (W) recognized by the workpiece recognition unit (32). The grouping unit (36) classifies only the three-dimensional points other than the three-dimensional points corresponding to the workpieces (W) recognized by the workpiece recognition unit (32) in the three-dimensional point cloud data into groups. The operation setting unit (37) may set the operation of the robot (20) so that the workpiece model of the target workpiece (Wo) can be extracted without interfering with the obstacle model and other workpiece models.

[0046] (Note 4) The robot system (1) comprises a 3D vision sensor (20) that generates 3D point cloud data consisting of multiple 3D points in a subject area where a workpiece (W) may exist, a robot (20) that takes out the workpiece (W), and a robot control device (30) according to any of Notes (1) to (3) that controls the robot (20) based on the 3D point cloud data provided by the 3D vision sensor (20).

[0047] Although the present disclosure has been described in detail above, it is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the gist of the present disclosure or from the spirit of the present disclosure derived from the claims and their equivalents. Furthermore, these embodiments can be implemented in combination. For example, the order of operations and processes in the embodiments described above are shown as examples only and are not limited thereto. The same applies when numerical values ​​or mathematical formulas are used in the description of the embodiments described above.

[0048] 1 Robot system 10 3D vision sensor 20 Robot 30 Robot control device 31 Data acquisition unit 32 Work recognition unit 33 Target determination unit 34 Work model placement unit 35 Grouping unit 36 ​​Obstacle model generation unit 37 Operation setting unit 38 Operation execution unit C Container W Work

Claims

1. A robot control device for controlling a robot that retrieves a workpiece based on three-dimensional point cloud data generated by a 3D vision sensor, which shows the surface shape of a subject area where a workpiece may exist using a plurality of three-dimensional points, comprising: a grouping unit that classifies the plurality of three-dimensional points into a plurality of groups that are located close together on the same virtual plane; an obstacle model generation unit that generates an obstacle model consisting of a hexahedron whose top surface is a rectangle on the virtual plane that encloses the three-dimensional points belonging to the same group, and whose bottom surface is a rectangle obtained by projecting the top surface perpendicularly onto the surface on which the workpiece is placed; and an operation setting unit that sets the operation of the robot for retrieving the workpiece so as not to interfere with the obstacle model.

2. The robot control device according to claim 1, further comprising: a workpiece recognition unit that recognizes the workpiece from the three-dimensional point cloud data; and a target determination unit that determines the next target workpiece to be extracted from the workpieces recognized by the workpiece recognition unit, wherein the grouping unit classifies only the three-dimensional points in the three-dimensional point cloud data other than the three-dimensional points corresponding to the target workpiece into a group.

3. The robot control device according to claim 1, further comprising: a work recognition unit that recognizes the work from the three-dimensional point cloud data; a work model placement unit that arranges work models that model the work recognized by the work recognition unit; and a target determination unit that determines the next target work to be extracted from the work recognized by the work recognition unit, wherein the grouping unit classifies only the three-dimensional points in the three-dimensional point cloud data other than the three-dimensional points corresponding to the work recognized by the work recognition unit into groups, and the operation setting unit sets the operation of the robot so that the work model of the target work can be extracted without interference with the obstacle model and other work models.

4. A robot system comprising: a 3D vision sensor that generates three-dimensional point cloud data consisting of multiple three-dimensional points in a subject area where a workpiece may exist; a robot that retrieves the workpiece; and a robot control device according to any one of claims 1 to 3 that controls the robot based on the three-dimensional point cloud data provided by the 3D vision sensor.

Images