Processing device and program

Abstract

This processing device comprises an acquisition unit and a determination unit. The acquisition unit acquires object information pertaining to the shape of an object held by the holding unit, holding unit information pertaining to the shape of the holding unit, and disposition destination information pertaining to a disposition destination of the object. The determination unit determines a disposition-enabled area in which the holding unit can dispose the object on the basis of the object information, the holding unit information, and the disposition destination information which are acquired by the acquisition unit.

Description

Processing Device and Program

[0001] The present disclosure relates to a technique for arranging an object.

[0002] Patent Document 1 describes a technique for arranging an object.

[0003] Japanese Patent Application Laid-Open No. 2019-934954

[0004] A processing device and a program are disclosed. In one embodiment, the processing device includes an acquisition unit and a determination unit. The acquisition unit acquires object information regarding the shape of an object held in a holding unit, holding unit information regarding the shape of the holding unit, and arrangement destination information regarding the arrangement destination of the object. The determination unit determines an arrangeable area in which the holding unit can arrange the object based on the object information, the holding unit information, and the arrangement destination information acquired by the acquisition unit.

[0005] Also, in one embodiment, the program is a program for causing a computer device to function as the above-described processing device.

[0006] Figure 1 is a schematic diagram showing an example of a robot system. Figure 2 is a schematic diagram showing an example of a placement jig. Figure 3 is a schematic diagram showing an example of a robot system. Figure 4 is a schematic diagram showing an example of the configuration of a processing unit. Figure 5 is a flowchart showing an example of the operation of a processing unit. Figure 6 is a schematic diagram showing an example of the configuration of a holding unit. Figure 7 is a schematic diagram showing an example of opening / closing range information. Figure 8 is a schematic diagram showing an example of open state shape information. Figure 9 is a schematic diagram showing an example of suction opening shape information. Figure 10 is a schematic diagram showing an example of object information. Figure 11 is a schematic diagram showing an example of placement destination information. Figure 12 is a schematic diagram showing an example of a first convolutional binary image. Figure 13 is a schematic diagram showing an example of a second convolutional binary image. Figure 14 is a schematic diagram showing an example of a logical AND image. Figure 15 is a schematic diagram showing an example of placement information. Figure 16 is a schematic diagram showing an example of a method for acquiring placement information. Figure 17 is a schematic diagram showing an example of placement information. Figure 18 is a schematic diagram showing an example of placement information. Figure 19 is a schematic diagram showing an example of an inverted fourth convolutional image. Figure 20 is a schematic diagram showing an example of how an object is placed on a placement jig. Figure 21 is a schematic diagram showing an example of how an object is placed on a placement jig. Figure 22 is a schematic diagram showing an example of how an object is placed on a placement jig. Figure 23 is a schematic diagram showing an example of a binary image. Figure 24 is a schematic diagram showing an example of a fifth convolution binary image. Figure 25 is a schematic diagram showing an example of an edge binary image. Figure 26 is a schematic diagram to explain an example of the direction of acquiring the shortest distance. Figure 27 is a schematic diagram showing an example of a histogram of the shortest distance. Figure 28 is a schematic diagram showing an example of a histogram of the shortest distance. Figure 29 is a schematic diagram showing an example of a histogram of the shortest distance. Figure 30 is a schematic diagram showing an example of a histogram of the shortest distance. Figure 31 is a schematic diagram to explain an example of the direction of acquiring the shortest distance. Figure 32 is a schematic diagram to explain an example of the direction of acquiring the shortest distance. Figure 33 is a schematic diagram to explain an example of the direction of acquiring the shortest distance. Figure 34 is a schematic diagram to explain an example of the direction of acquiring the shortest distance. Figure 35 is a schematic diagram showing an example of an object image. Figure 36 is a schematic diagram showing an example of a placement destination image. Figure 37 is a schematic diagram showing an example of a placement position determined by the determination unit. Figure 38 is a schematic diagram showing an example of a histogram of the shortest distance.Figure 39 is a schematic diagram showing an example of a histogram of the shortest distance. Figure 40 is a schematic diagram showing an example of an object image. Figure 41 is a schematic diagram showing an example of a placement position determined by the determination unit. Figure 42 is a schematic diagram showing an example of a placement position determined by the determination unit.

[0007] Figure 1 is a schematic diagram showing an example of a robot system 50. As shown in Figure 1, the robot system 50 includes, for example, a robot 10, a processing unit 1, and a camera 20.

[0008] The robot 10 can, for example, hold an object 30 located in one place, move the held object 30 to another location, and perform a transfer operation to place it in that other location. The object 30 can also be called an object to be transferred or an object to be worked on. However, the operations that the robot 10 can perform are not limited to these.

[0009] The robot 10 is, for example, an arm-type robot, and comprises an arm 11 and an end effector 12 connected to the arm 11. The end effector 12 is capable of holding, for example, an object 30. The end effector 12 has a holding part 13 capable of holding the object 30.

[0010] In the example shown in Figure 1, the holding part 13 has a plurality of finger portions 13a capable of gripping the object 30. The plurality of finger portions 13a can hold the object 30 by opening and closing. It can also be said that the holding part 13 can hold the object 30 by opening and closing. Each finger portion 13a is, for example, a long, slender member extending in a straight line. The end effector 12 can hold the object 30 by sandwiching it between the plurality of finger portions 13a.

[0011] Multiple finger portions 13a can grip an object 30 by closing them while the object 30 is located between the multiple finger portions 13a. In other words, multiple finger portions 13a can grip an object 30 by moving toward each other while the object 30 is located between the multiple finger portions 13a. On the other hand, the multiple finger portions 13a that grip the object 30 can release the grip (in other words, hold) of the object 30 by opening them. In other words, the multiple finger portions 13a that grip the object 30 can release the grip of the object 30 by moving toward each other. In the example in Figure 1, the holding portion 13 has two finger portions 13a, but the number of finger portions 13a that the holding portion 13 has may be three or more.

[0012] The arm 11, for example, has multiple joints. The posture of the arm 11 changes as the amount of rotation of at least one of the multiple joints changes. As a result of the change in the posture of the arm 11, the position and posture of the end effector 12 change. In addition, as a result of the change in the posture of the arm 11, the position and posture of the object 30 held by the end effector 12 change.

[0013] In the example shown in Figure 1, the object 30 is placed on a workbench 40. The robot 10 holds the object 30 with multiple fingers 13a and moves the held object 30 to, for example, a workbench 45 and places it on the workbench 45. On the workbench 45, for example, is a placement jig 35 that defines the placement location of the object 30 on the workbench 45. The robot 10 places the object 30 to be held onto the placement jig 35 from above. The placement jig 35 can also be called a positioning jig. In Figure 1, the placement of the object 30 on the placement jig 35 is shown by a dashed line.

[0014] Figure 2 is a schematic diagram showing an example of a placement jig 35. Figure 2 shows an example of the placement jig 35 on a workbench 45 as viewed from above the workbench 45. In the example in Figure 2, the placement jig 35 comprises a plate-shaped base portion 35a and a rectangular frame-shaped portion 35b provided on the base portion 35a. In Figure 2, for convenience, the frame-shaped portion 35b is shown with diagonal lines to make its shape easier to understand. In the drawings described later, the portion provided on the base portion 35a is also shown with diagonal lines for the same reason.

[0015] The object 30 is placed in the area surrounded by the frame-shaped portion 35b on the upper surface of the base portion 35a. The frame-shaped portion 35b functions as a placement area defining portion that defines the area in which the object 30 is to be placed. The placement area defining portion can also be said to be a position restricting portion that restricts the position of the object 30 placed on the placement jig 35.

[0016] The holding part 13, which holds the object 30, approaches the placement jig 35 from above and places the object 30 inside the frame-shaped part 35b. At this time, if the object 30 held by the holding part 13 or the holding part 13 itself comes into contact with the frame-shaped part 35b, the holding part 13 may not be able to place the object 30 inside the frame-shaped part 35b. The frame-shaped part 35b, which functions as a placement target area defining part (in other words, a position restricting part), can be said to be an obstacle that hinders the placement of the object 30.

[0017] The shape of the placement jig 35 is not limited to the above example. For example, the placement jig 35 may include a plurality of pins (also called positioning pins) erected on the base portion 35a that function as a defining area for placement. Alternatively, the placement jig 35 may include a plurality of straight plate-like portions (for example, a plurality of rectangular parallelepipeds) erected on the base portion 35a that function as a defining area for placement. In the placement jig 35, the portion that functions as a defining area for placement becomes an obstacle to the placement of the object 30.

[0018] Furthermore, the robot 10 may place multiple objects 30 of the same shape into the placement jig 35, or it may place multiple objects 30 of different shapes into the placement jig 35. In this case, the robot 10 may, for example, hold each of the multiple objects 30 on the workbench 40 and place them into the placement jig 35.

[0019] When multiple objects 30 of different shapes are placed in the placement jig 35, the placement jig 35 may be provided with a placement area regulating section for each of the multiple shapes of the objects 30, corresponding to that shape. For example, the placement jig 35 may include a square or annular frame-shaped section that defines the placement area of ​​an object 30 having a first shape, a plurality of pins that define the placement area of ​​an object 30 having a second shape, and a plurality of straight plate-shaped sections that define the placement area of ​​an object 30 having a third shape.

[0020] In the example above, the holding unit 13 places the object 30 on the placement jig 35, but the placement location of the object 30 is not limited to the placement jig 35. For example, the holding unit 13 may place the object 30 in a container (such as a tray) on the workbench 45, or on another component. When the object 30 is placed inside a container, the peripheral wall of the container (in other words, the side wall) becomes an obstacle to the placement of the object 30. Hereafter, when we simply refer to an obstacle, we mean an obstacle that hinders the placement of the object 30, such as the frame-shaped part 35b in Figure 2.

[0021] The holding portion 13 of the end effector 12 may include a suction portion 13b for adsorbing and holding the object 30, as shown in Figure 3. The suction portion 13b can, for example, vacuum adsorb the object 30. The suction portion 13b is also called, for example, a suction pad or a vacuum pad.

[0022] The suction part 13b is hollow and has a suction opening at its tip. When the suction part 13b picks up an object 30, the suction opening is blocked by the object 30. The negative pressure generating unit of the robot 10 generates negative pressure inside the suction part 13b, reducing the pressure inside the suction part 13b, which causes the suction part 13b to pick up the object 30. On the other hand, when the negative pressure inside the suction part 13b is released while the suction part 13b is picking up the object 30, the suction (in other words, holding) of the object 30 by the suction part 13b is released.

[0023] Hereafter, the holding part 13 having the suction portion 13b may be referred to as the suction holding part 13. Also, the holding part 13 having multiple finger portions 13a may be referred to as the opening / closing holding part 13.

[0024] The camera 20 is fixed to the end effector 12, for example. Therefore, the shooting range of the camera 20 changes according to the position and orientation of the end effector 12. It can also be said that the shooting range of the camera 20 changes according to the orientation of the arm 11.

[0025] Camera 20 is, for example, a three-dimensional camera. The camera image 21 (see Figure 4) generated by camera 20 includes, for example, a color image and a depth image. Camera 20 outputs the generated camera image 21 to the processing unit 1. For generating the depth image, a stereo method may be used, a projector method may be used, a combination of the stereo method and the projector method may be used, or other methods may be used. The depth image is, for example, a grayscale image. The color image may be, for example, an RGB image. The color image can also be said to be a captured image showing the shooting range of camera 20. Both the depth image and the color image can also be called camera images.

[0026] The processing unit 1 can, for example, control the robot 10 (specifically, the arm 11 and the end effector 12) based on the camera image 21 acquired by the camera 20. The processing unit 1 can, for example, determine the holding position (hereinafter sometimes simply referred to as the holding position) in which the holding unit 13 holds the object 30 based on the camera image 21. The holding position can also be said to be the position of the holding unit 13 when it holds the object 30. The processing unit 1 controls the arm 11 and the end effector 12 so that the holding unit 13 holds the object 30 at the determined holding position.

[0027] Furthermore, the processing unit 1 can determine, for example, the placement position in which the holding unit 13 will position the object 30 based on the camera image 21. When the object 30 is placed on the placement jig 35, the processing unit 1 determines the placement position in the placement jig 35 (more specifically, on the base portion 35a of the placement jig 35) in which the holding unit 13 will position the object 30. For example, the processing unit 1 determines the placement position of the object 30 such that the holding unit 13 that holds the object 30 and the object 30 do not interfere with any obstacles (for example, the frame-shaped portion 35b in Figure 2) at the placement location of the object 30. The processing unit 1 controls the arm 11 and the end effector 12 so that the holding unit 13 that holds the object 30 positions the object 30 at the determined placement position. Hereafter, when simply referred to as the placement position, it means the placement position in which the holding unit 13 positions the object 30 (in other words, the position in which the object 30 is placed).

[0028] <Example of Processing Unit Configuration> Figure 4 is a schematic diagram showing an example of the configuration of the processing unit 1. As shown in Figure 4, the processing unit 1 includes, for example, a control unit 2, a storage unit 3, an interface 4, and an interface 5. The processing unit 1 can also be called, for example, a computer device. Alternatively, the processing unit 1 can also be called, for example, a processing circuit.

[0029] Interface 4 can communicate with the camera 20. The control unit 2 can acquire camera images 21 generated by the camera 20 through interface 4. Interface 4 can also be called, for example, an interface circuit, a communication unit, or a communication circuit. Interface 4 may communicate with the camera 20 via wired communication or wireless communication.

[0030] Interface 5 can communicate with the robot 10. The control unit 2 can control the robot 10 through interface 5. Interface 5 can also be called, for example, an interface circuit, a communication unit, or a communication circuit. Interface 5 may communicate with the robot 10 via wired communication or wireless communication.

[0031] The control unit 2 can comprehensively manage the operation of the processing unit 1 by controlling other components of the processing unit 1. The control unit 2 can also be called, for example, a control circuit. The control unit 2 includes at least one processor to provide control and processing capabilities for performing various functions, as will be described in more detail below.

[0032] According to various embodiments, at least one processor may be implemented as a single integrated circuit (IC) or as a plurality of communicably connected integrated circuits IC and / or discrete circuits. At least one processor can be implemented according to various known techniques.

[0033] In one embodiment, the processor includes one or more circuits or units configured to perform one or more data computation procedures or processes by, for example, executing instructions stored in associated memory. In other embodiments, the processor may be firmware (e.g., discrete logic components) configured to perform one or more data computation procedures or processes.

[0034] According to various embodiments, the processor may include one or more processors, controllers, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), digital signal processing devices, programmable logic devices, field-programmable gate arrays, or any combination of these devices or configurations, or other known combinations of devices and configurations, and may perform the functions described below.

[0035] The control unit 2 may include, for example, a CPU (Central Processing Unit) as a processor. The storage unit 3 may include non-temporary recording media that can be read by the CPU of the control unit 2, such as ROM (Read Only Memory) and RAM (Random Access Memory). The storage unit 3 stores, for example, a program 3a for controlling the processing unit 1. Various functions of the control unit 2 are realized, for example, by the CPU of the control unit 2 executing the program 3a in the storage unit 3.

[0036] The configuration of the processing unit 1 is not limited to the above example. For example, the control unit 2 may include multiple CPUs. The control unit 2 may also include at least one DSP (Digital Signal Processor). Furthermore, all or some of the functions of the control unit 2 may be implemented by hardware circuits that do not require software to realize those functions. In addition, the storage unit 3 may include a computer-readable, non-temporary recording medium other than ROM and RAM. The storage unit 3 may include, for example, a small hard disk drive and an SSD (Solid State Drive).

[0037] Furthermore, the processing unit 1 may include a display unit controlled by the control unit 2. The display unit may be, for example, a liquid crystal display, an organic electroluminescent (EL) display, or a plasma display. The display unit of the processing unit 1 may, for example, display at least one of the color image and the depth image generated by the camera 20.

[0038] Furthermore, the processing unit 1 may include an input unit for receiving input from the user. The input unit may include, for example, a mouse and a keyboard. The input unit may also include a touch sensor for receiving user touch operations. The input unit may also include a microphone for receiving user voice input.

[0039] Further, the processing device 1 may be composed of a plurality of computer devices. Further, the processing device 1 may be a cloud server. In this case, the interface 5 of the processing device 1 may communicate with the robot 10 through a network including the Internet or the like.

[0040] <Example of the configuration of the control unit of the processing device> As shown in FIG. 4, the control unit 2 includes, for example, a robot control unit 2a, an acquisition unit 2b, and a determination unit 2c. Each of the robot control unit 2a, the acquisition unit 2b, and the determination unit 2c is a functional block generated by the control unit 2 executing the program 3a in the storage unit 3.

[0041] The robot control unit 2a can control the robot 10 through the interface 5. The robot control unit 2a can control the posture and position of the holding unit 13, for example, by controlling the posture of the arm 11. The robot control unit 2a controls the position of the holding unit 13, for example, by controlling the reference position (also referred to as the representative position) of the holding unit 13. The reference position of the holding unit 13 is also referred to as, for example, TCP (Tool Center Point). The reference position of the opening / closing holding unit 13 is set, for example, at the midpoint between the tips of the two finger parts 13a. Also, the reference position of the suction holding unit 13 is set, for example, at the center of the suction opening of the suction part 13b. Hereinafter, the reference position of the holding unit 13 (in other words, TCP) may be referred to as the holding unit reference position or simply the reference position.

[0042] The robot control unit 2a can control the opening and closing of the opening / closing holding unit 13 of the end effector 12 (specifically, the opening and closing of the plurality of finger parts 13a). By controlling the opening and closing of the opening / closing holding unit 13, the robot control unit 2a can cause the opening / closing holding unit 13 to hold the object 30 or release the holding of the object 30 by the opening / closing holding unit 13. When the holding unit 13 has a suction part 13b, the robot control unit 2a can control the negative pressure generation part provided in the robot 10 to cause the suction holding unit 13 (specifically, the suction part 13b) to suck the object 30 or release the suction of the object 30 by the suction holding unit 13.

[0043] The acquisition unit 2b acquires object information regarding the shape of the object 30 based on, for example, a camera image 21 obtained by photographing the object 30 with the camera 20. Further, the acquisition unit 2b acquires placement destination information regarding the placement destination (in this example, the placement jig 35) based on, for example, a camera image 21 obtained by photographing the placement destination of the object 30 with the camera 20. Further, the acquisition unit 2b acquires holding unit information regarding the shape of the holding unit 13.

[0044] The determination unit 2c determines, for example, a holding position where the holding unit 13 holds the object 30 based on the object information and the holding unit information acquired by the acquisition unit 2b. Further, the determination unit 2c determines a placement possible region where the holding unit 13 can place the object 30 based on the object information, the holding unit information, and the placement destination information acquired by the acquisition unit 2b. Then, the determination unit 2c determines a placement position where the holding unit 13 places the object 30 from the determined placement possible region.

[0045] The robot control unit 2a controls the arm 11 and the holding unit 13 so that the holding unit 13 holds the object 30 at the holding position determined by the determination unit 2c. Further, the robot control unit 2a controls the arm 11 and the holding unit 13 so that the holding unit 13 holding the object 30 places the object 30 at the placement position determined by the determination unit 2c.

[0046] Note that all functions or some functions of the robot control unit 2a may be realized by a hardware circuit that does not require software for realizing the functions. The same applies to the acquisition unit 2b and the determination unit 2c.

[0047] In the above example, the processing device 1 controls the robot 10, but a device different from the processing device 1 may control the robot 10. In this case, the processing device 1 notifies the control device that controls the robot 10 of the determined holding position and placement position. Then, the control device controls the arm 11 and the holding unit 13 so that the holding unit 13 holds the object 30 at the holding position notified from the processing device 1. Further, the control device controls the arm 11 and the holding unit 13 so that the holding unit 13 holding the object 30 places the object 30 at the placement position notified from the processing device 1.

[0048] <Example of Processing Unit Operation> Figure 5 is a flowchart showing an example of the operation of the processing unit 1. Below, we will mainly describe an example of the operation of the processing unit 1 when the end effector 12 has an opening / closing holding unit 13.

[0049] In step s1, the acquisition unit 2b of the control unit 2 of the processing device 1 acquires information about the shape of the holding unit 13. In this example, the holding unit information is stored in the storage unit 3. The acquisition unit 2b acquires the holding unit information from the storage unit 3. The holding unit information can also be said to be information based on the shape of the holding unit 13.

[0050] The holding part information includes, for example, holding range information that represents the range in which the holding part 13 can hold the object 30. The holding range information is information based on the shape of the holding part 13. The holding part information (also called the opening / closing holding part information) relating to the shape of the opening / closing holding part 13 includes, for example, opening / closing range information 100 that represents the opening / closing range of the opening / closing holding part 13 (in other words, the opening / closing range of the multiple finger parts 13a). If the object 30 is positioned between two open finger parts 13a, the two finger parts 13a can hold the object 30, so it can be said that the opening / closing range information 100 represents the range in which the opening / closing holding part 13 can hold the object 30. It can also be said that the opening / closing range information 100 represents the range in which the opening / closing holding part 13 can sandwich the object 30. Hereafter, the opening / closing range of the opening / closing holding part 13 may be referred to as the holding part opening / closing range.

[0051] Furthermore, the holding part information includes interference range information that represents the range in which the holding part 13 interferes with other objects other than the object 30 when the holding part 13 holds the object 30. The interference range information can also be said to be collision range information that represents the range in which the holding part 13 collides with other objects other than the object 30 when the holding part 13 holds the object 30. The interference range information is information based on the shape of the holding part 13.

[0052] The opening / closing holding part information includes open state shape information 110, which represents the shape of the opening / closing holding part 13 (also called the open holding part 13) in the open state, as interference range information. Since the opening / closing holding part 13 is in the open state when it holds the object 30, the open state shape information 110 can be said to represent the range in which the opening / closing holding part 13 interferes with other objects other than the object 30 when the opening / closing holding part 13 holds the object 30.

[0053] Figure 6 is a schematic diagram showing an example of how the two finger portions 13a of the opening / closing holding part 13 are viewed from the tip side. Figure 6 shows the shape of the tips of the two finger portions 13a in the open state. As shown in Figure 6, the shape of the tips of the finger portions 13a (in other words, the shape of the end face on the tip side of the finger portions 13a) is, for example, rectangular. In this example, as shown in Figure 6, the distance between the two finger portions 13a in the open state is represented by a1. The distance a1 can also be said to be the size of the gap between the two finger portions 13a in the open state in the opening / closing direction. The size of the finger portions 13a in the opening / closing direction is called the width b1. The size of the finger portions 13a in the longitudinal direction and in the direction perpendicular to the opening / closing direction is called the depth c1.

[0054] Figure 7 is a schematic diagram showing an example of opening / closing range information 100. The opening / closing range information 100 is, for example, a binary image. Hereafter, the opening / closing range information 100 may be referred to as the opening / closing range image 100.

[0055] The outline of the opening / closing range image 100 is, for example, a square. In other words, the number of pixels in the vertical direction (in other words, the column direction) and the horizontal direction (in other words, the row direction) of the opening / closing range image 100 are the same.

[0056] The opening / closing range image 100 includes an opening / closing range equivalent region 101 that corresponds to the opening / closing range of the holding part. Each pixel value in the opening / closing range equivalent region 101 is, for example, "1". In the opening / closing range image 100, each pixel value in areas other than the opening / closing range equivalent region 101 is "0". The opening / closing range equivalent region 101 can also be said to be, for example, a mask of the holding part opening / closing range. The opening / closing range equivalent region 101 represents the opening / closing range of the holding part.

[0057] The opening / closing range equivalent area 101 has a shape corresponding to the opening / closing range of the opening / closing holding part 13. For example, the opening / closing range equivalent area 101 is a rectangle that is long horizontally. The horizontal size of the opening / closing range equivalent area 101 (in other words, the number of pixels in the horizontal direction) corresponds to the distance a1 between the two finger parts 13a in the open state. The vertical size of the opening / closing range equivalent area 101 (in other words, the number of pixels in the vertical direction) corresponds to the depth c1 of the finger parts 13a. The opening / closing range equivalent area 101 has a shape corresponding to the distance a1 between the two finger parts 13a in the open state and the shape of the finger parts 13a (specifically, the depth c1 of the finger parts 13a). The opening / closing range equivalent area 101 can be said to represent the range in which the opening / closing holding part 13 can hold the object 30. Furthermore, the opening / closing range equivalent area 101 can be said to represent the range in which the holding part 13 contacts the object 30 and generates a holding force on the object 30. Furthermore, the opening / closing range equivalent region 101 can also be said to represent the gap between the two finger portions 13a in the open state.

[0058] The center of the opening / closing range equivalent region 101 coincides with the center of the opening / closing range image 100. The center of the opening / closing range image 100, in other words, the center of the opening / closing range equivalent region 101, corresponds to the reference position of the holding part (in other words, TCP).

[0059] Figure 8 is a schematic diagram showing an example of open state shape information 110. Open state shape information 110 is, for example, a binary image. Hereafter, open state shape information 110 may be referred to as open state shape image 110.

[0060] The outline of the open state shape image 110 is, for example, a square. The image size of the open state shape image 110 matches, for example, the image size of the open / closed range image 100. In other words, the vertical and horizontal dimensions of the open state shape image 110 match the vertical and horizontal dimensions of the open / closed range image 100, respectively.

[0061] The open state shape image 110 includes an open-holding portion equivalent region 111, which corresponds to the open-holding portion 13. The open-holding portion equivalent region 111 corresponds to the open-holding portion 13 when viewed from the tip side, for example. The open-holding portion equivalent region 111 can also be said to correspond to the tip of the open-holding portion 13. The open-holding portion equivalent region 111 can also be said to represent the shape of the open-holding portion 13, the shape of the tip of the open-holding portion 13, or the shape of the open-holding portion 13 when viewed from the tip side. The open-holding portion equivalent region 111 can also be said to be a mask of the open-holding portion 13.

[0062] The open-holding region 111 includes two finger-corresponding regions 112, each corresponding to one of the two finger portions 13a. The finger-corresponding regions 112 can be said to represent the shape of the finger portion 13a, the shape of the tip of the finger portion 13a, or the shape of the finger portion 13a when viewed from the tip side. The finger-corresponding regions 112 can also be said to be masks for the finger portion 13a. Each pixel value in the finger-corresponding region 112 is, for example, "1". In the open-state shape image 110, each pixel value in the regions other than the two finger-corresponding regions 112 is "0".

[0063] In the open state shape image 110, the two finger-equivalent regions 112 are arranged horizontally with a gap between them. The finger-equivalent regions 112 form, for example, a rectangle that is long vertically. The horizontal and vertical dimensions of the finger-equivalent regions 112 correspond to the width b1 and depth c1 of the finger portion 13a, respectively. The vertical dimensions of the finger-equivalent regions 112 (in other words, the number of pixels in the vertical direction) match the vertical dimensions of the open / close range equivalent region 101 (in other words, the number of pixels in the vertical direction). The gap between the two finger-equivalent regions 112 corresponds to the gap a1 between the two finger portions 13a in the open state. The gap between the two finger-equivalent regions 112 can also be said to be the horizontal dimensions of the gap between the two finger-equivalent regions 112 (in other words, the number of pixels in the horizontal direction of the gap). The gap between the two finger-equivalent regions 112 matches the horizontal dimensions of the open / close range equivalent region 101.

[0064] The open-holding portion equivalent region 111 has a shape corresponding to the distance a1 between the two finger portions 13a in the open state and the shape of the finger portions 13a (specifically, the width b1 and depth c1 of the finger portions 13a). The open-holding portion equivalent region 111 can also be said to represent the region in which the open / closed holding portion 13 interferes with objects other than the object 30 when the open / closed holding portion 13 is in the open state in an attempt to hold the object 30.

[0065] The center of the center line connecting the centers of the two finger-equivalent regions 112 coincides with the center of the open-state shape image 110. In other words, the center of the gap between the two finger-equivalent regions 112 coincides with the center of the open-state shape image 110. The center of the open-state shape image 110 corresponds to the reference position of the holding part (in other words, TCP).

[0066] The holding information (also called the holding information) for the adsorption holding part 13 includes adsorption opening shape information 120, which represents the shape of the adsorption opening of the adsorption part 13b, as holding range information. Figure 9 is a schematic diagram showing an example of the adsorption opening shape information 120.

[0067] The adsorption aperture shape information 120 is, for example, a binary image. Hereafter, the adsorption aperture shape information 120 may be referred to as the adsorption aperture shape image 120. The outline of the adsorption aperture shape image 120 is, for example, a square.

[0068] The suction aperture shape image 120 includes a region 121 corresponding to the suction aperture. Each pixel value in the suction aperture region 121 is, for example, "1". In the suction aperture shape image 120, each pixel value in areas other than the suction aperture region 121 is "0". The suction aperture region 121 can also be called a mask for the suction aperture. The suction aperture region 121 represents the shape of the suction aperture.

[0069] The adsorption opening equivalent region 121 has a shape corresponding to the shape of the adsorption opening. For example, the adsorption opening equivalent region 121 is circular. The diameter of the adsorption opening equivalent region 121 corresponds to the diameter of the adsorption opening. The center of the adsorption opening equivalent region 121 coincides with the center of the adsorption opening shape image 120. The center of the adsorption opening shape image 120, in other words, the center of the adsorption opening equivalent region 121, corresponds to the reference position of the holding part (in other words, TCP).

[0070] In this example, when the suction part 13b adsorbs and holds the object 30, the suction part 13b is less likely to interfere with other objects besides the object 30. Therefore, interference range information is not included in the suction holding part information. However, if the suction part 13b is likely to interfere with other objects besides the object 30 when adsorbing and holding the object 30, interference range information may be included in the suction holding part information.

[0071] In step s1, once the holding unit information is acquired, in step s2, the acquisition unit 2b acquires object information 150 relating to the shape of the object 30. The acquisition unit 2b acquires object information 150 based, for example, on a camera image 21 (also called the first camera image 21) obtained when the object 30 is photographed by the camera 20.

[0072] When step s2 is performed, the robot control unit 2a controls the posture of the arm 11 so that, for example, the object 30 on the workbench 40 can be photographed by the camera 20 from directly above. The control unit 2 then controls the camera 20 through the interface 4 to cause the camera 20 to photograph the object 30 from directly above. The first camera image 21 obtained by the camera 20 when the object 30 is photographed is input to the control unit 2. The acquisition unit 2b of the control unit 2 acquires object information 150 based on the first camera image 21.

[0073] The object information 150 represents, for example, the shape and position of the object 30. Figure 10 is a schematic diagram showing an example of object information 150. The object information 150 is, for example, a binary image. Hereafter, object information 150 may be referred to as object image 150. The image size of object image 150 is larger than the image sizes of the opening / closing range image 100, the open state shape image 110, and the suction opening shape image 120.

[0074] The object image 150 includes an object-corresponding region 151 that corresponds to the object 30. Each pixel value in the object-corresponding region 151 is, for example, "1". In the object image 150, each pixel value in areas other than the object-corresponding region 151 is "0". The object-corresponding region 151 can also be said to be, for example, a mask of the object 30. The object-corresponding region 151 represents the shape of the object 30.

[0075] The object-equivalent region 151 has a shape corresponding to the shape of the object 30. The image size of the object image 150 corresponds to the shooting range of the camera 20. The position of the object-equivalent region 151 in the object image 150 corresponds to the position of the object 30 in the shooting range of the camera 20. It can also be said that the position of the object-equivalent region 151 in the object image 150 represents the position of the object 30 in the shooting range of the camera 20. The positions of the multiple pixels that make up the object image 150 correspond to multiple positions in the actual work space where the robot 10 performs its work. Hereafter, the position of a pixel may be referred to as the pixel position.

[0076] The ratio of the image size of the object-corresponding region 151 to the image size of the finger-corresponding region 112 (see Figure 8) is consistent with the ratio of the size of the object 30 (more specifically, the size of the object 30 when viewed from directly above) to the size of the finger portion 13a (more specifically, the size of the finger portion 13a when viewed from the tip side).

[0077] Furthermore, the ratio of the image size of the object-equivalent region 151 to the distance between the two finger-equivalent regions 112 is consistent with the ratio of the size of the object 30 to the distance a1 between the two finger-parts 13a (in other words, the two finger-parts 13a in the open state) in the open-holding portion 13.

[0078] Furthermore, the ratio of the image size of the object-equivalent region 151 to the image size of the opening / closing range-equivalent region 101 (see Figure 7) is consistent with the ratio of the size of the object 30 to the size of the opening / closing range of the holding unit.

[0079] The acquisition unit 2b may acquire the object image 150 based on the color image (also called the first color image) included in the first camera image 21, or it may acquire the object image 150 based on the depth image (also called the first depth image) included in the first camera image 21.

[0080] For example, the acquisition unit 2b may perform segmentation to recognize the object 30 in the first color image using a machine learning model to generate an object image 150. As the machine learning model for segmentation, for example, Mask Scoring R-CNN may be used, Segment-Anything Model (SAM) may be used, or other learning models may be used. R-CNN is an abbreviation for Region Based Convolutional Neural Networks.

[0081] Furthermore, if the camera 20 can distinguish the color of the object 30 from other colors when viewing the object 30, the acquisition unit 2b may generate the object image 150 by applying a color filter to the first color image. Alternatively, the acquisition unit 2b may generate the object image 150 by binarizing the first depth image.

[0082] In step s2, when object information 150 is acquired, in step s3, the acquisition unit 2b acquires location information 160 regarding the location of the object 30. The acquisition unit 2b acquires location information 160 regarding the location of the object 30 based on, for example, a camera image 21 (also called a second camera image 21) obtained by the camera 20 capturing the location of the object 30. The location information 160 can also be said to be, for example, location environment information regarding the location environment of the object 30, or information representing the location environment of the object 30.

[0083] In this example, the placement destination for the object 30 is the placement jig 35. When step s3 is executed, the robot control unit 2a controls the posture of the arm 11 so that, for example, the placement jig 35 on the workbench 45 can be photographed from directly above by the camera 20. The control unit 2 then controls the camera 20 through the interface 4 to photograph the placement jig 35 from directly above. The second camera image 21 obtained by the camera 20, which shows the placement jig 35 (in other words, the placement destination), is input to the control unit 2. The acquisition unit 2b of the control unit 2 acquires placement destination information 160 based on the second camera image 21. In this example, the placement destination information 160 can be said to be placement jig information relating to the placement jig 35, or it can be said to be information representing the placement jig 35.

[0084] The placement information 160 represents, for example, the shape and position of obstacles present at the placement location. The placement information 160 can be said to be obstacle information representing the shape of the obstacles, or obstacle position information representing the location of the obstacles.

[0085] Figure 11 is a schematic diagram showing an example of placement information 160. Placement information 160 is, for example, a binary image. Hereafter, placement information 160 may be referred to as placement image 160. The image size of placement image 160 is, for example, the same as the image size of the target image 150.

[0086] In the placement image 160, the region 161 with a pixel value of "1" corresponds to the region of the obstacle at the placement location. Hereafter, region 161 may be referred to as the obstacle-corresponding region 161.

[0087] In the placement image 160, the pixel values ​​of all areas except the obstacle-corresponding area 161 are "0". The obstacle-corresponding area 161 can be described as, for example, a mask of the obstacle. The obstacle-corresponding area 161 represents the shape of the obstacle. The obstacle-corresponding area 161 can also be described as the area where the obstacle exists at the placement location of the object 30.

[0088] In this example, the frame-shaped portion 35b of the placement jig 35 becomes an obstacle. The obstacle-corresponding region 161 is an image corresponding to the frame-shaped portion 35b. The obstacle-corresponding region 161 represents the shape of the frame-shaped portion 35b. Specifically, the obstacle-corresponding region 161 represents the shape of the frame-shaped portion 35b when the placement jig 35 is viewed from above.

[0089] The obstacle-equivalent area 161 has a shape corresponding to the shape of the frame-shaped portion 35b (in other words, the obstacle). The image size of the placement image 160 corresponds to the shooting range of the camera 20. The position of the obstacle-equivalent area 161 in the placement image 160 corresponds to the position of the frame-shaped portion 35b (in other words, the obstacle) in the shooting range of the camera 20. The position of the obstacle-equivalent area 161 in the placement image 160 represents the position of the frame-shaped portion 35b in the shooting range of the camera 20. The positions of the multiple pixels constituting the placement image 160 correspond to multiple positions in the actual working space of the robot 10.

[0090] The acquisition unit 2b may acquire the placement image 160 based on the color image (also called the second color image) included in the second camera image 21, or it may acquire the placement image 160 based on the depth image (also called the second depth image) included in the second camera image 21.

[0091] For example, the acquisition unit 2b may perform segmentation to recognize the frame-shaped portion 35b in the second color image using a machine learning model to generate the placement destination image 160. As the machine learning model for segmentation, for example, Mask Scoring R-CNN may be used, Segment-Anything Model may be used, or other learning models may be used.

[0092] Furthermore, if the frame-shaped portion 35b can be distinguished from other colors when viewed from the camera 20, the acquisition unit 2b may generate the placement destination image 160 by applying a color filter to the second color image. Alternatively, the acquisition unit 2b may generate the placement destination image 160 by binarizing the second depth image.

[0093] In step s3, once the placement information 160 is acquired, in step s4, the determination unit 2c determines the holding position in which the holding unit 13 will hold the object 30 on the workbench 40, based on the holding unit information acquired in step s1 and the object information acquired in step s2. For example, the determination unit 2c determines the holding reference position (in other words, TCP) in which the holding unit 13 will hold the object 30, based on the holding unit information and the object information.

[0094] In step s4, the determination unit 2c performs, for example, convolution (also called convolution operation) of the open / close range image 100 included in the open / close / holding unit information and the object image 150. Specifically, the determination unit 2c uses the open / close range image 100 as a kernel (also called a mask or filter) to perform convolution of the open / close range image 100 onto the object image 150. The convolution is performed, for example, by taking the pixel at the upper left corner of the object image 150 as the starting position for the convolution and the pixel at the lower right corner as the goal position, and then performing a sum-of-products operation of pixel values ​​while moving the center of the open / close range image 100 (corresponding to the holding unit reference position) over all pixels of the object image 150 from the pixel at the upper left corner to the pixel at the lower right corner of the object image 150. The movement of the open / close range image 100 is performed, for example, in a raster-like manner. Specifically, the opening / closing range image 100 moves, for example, from the leftmost pixel to the rightmost pixel in the top row of the object image 150, then moves down one row and moves again from the leftmost pixel to the rightmost pixel. By repeating these movements, the opening / closing range image 100 moves over all pixels of the object image 150. In the sum-of-products operation of pixel values, the determination unit 2c multiplies each pixel value of the part of the object image 150 that overlaps with the opening / closing range image 100 (also called the overlapping part image) with each pixel value of the opening / closing range image 100, using the pixel values ​​at the same pixel position. Next, the determination unit 2c adds the obtained multiplicative values. Then, the determination unit 2c adopts the obtained added value (in other words, the sum-of-products calculation value) as the pixel value at the same pixel position as the center of the overlapping part image in the image obtained by the convolution operation of the opening / closing range image 100 and the object image 150. Furthermore, when the opening / closing range image 100 is moved on the object image 150, the pixel values ​​of areas in the object image 150 that do not overlap with the opening / closing range image 100 are padded.

[0095] Next, the determination unit 2c binarizes the image (also called the first convolution image) that has pixel values ​​obtained by the convolution operation of the open / closed range image 100 and the object image 150. The determination unit 2c sets all pixel values ​​that are greater than 0 in the first convolution image to "1" and binarizes the first convolution image. Hereafter, the binarized first convolution image will be called the first convolution binary image.

[0096] Figure 12 is a schematic diagram showing an example of the first convolutional binary image 170. In the first convolutional binary image 170, the region 171 with a pixel value of "1" corresponds to the range of positions of the holding unit 13 such that at least a part of the object 30 is located within the holding unit opening / closing range when the holding unit 13 attempts to hold the object 30. In other words, region 171 corresponds to the range of the holding unit reference position (in other words, TCP) such that at least a part of the object 30 is located within the holding unit opening / closing range when the holding unit 13 attempts to hold the object 30. Each position within region 171 corresponds to the holding unit reference position when at least a part of the object 30 on the workbench 40 is within the holding unit opening / closing range. A position in the actual work space corresponding to a position within region 171 can be said to be a position such that at least a part of the object 30 on the workbench 40 is within the holding unit opening / closing range when the holding unit reference position is located at that position.

[0097] Furthermore, in step s4, the determination unit 2c performs a convolution operation between the open state shape image 110 included in the open / closed holding unit information and the object image 150. Specifically, the determination unit 2c uses the open state shape image 110 as a kernel to convolve the open state shape image 110 into the object image 150, in the same manner as when the open / closed range image 100 is convolved into the object image 150.

[0098] Next, the determination unit 2c binarizes the image (also called the second convolution image) which has pixel values ​​obtained by the convolution operation of the open shape image 110 and the object image 150. The determination unit 2c sets all pixel values ​​that are greater than 0 in the second convolution image to "1" and binarizes the second convolution image. Hereafter, the binarized second convolution image will be called the second convolution binary image.

[0099] Figure 13 is a schematic diagram showing an example of a second convolutional binary image 180. In the second convolutional binary image 180, the region 181 with a pixel value of "1" corresponds to the range of positions of the holding part 13 where the two open finger parts 13a interfere with the object 30 when attempting to hold the object 30. In other words, region 181 corresponds to the range of reference positions of the holding part where the two open finger parts 13a interfere with the object 30 when attempting to hold the object 30. Each position within region 181 corresponds to the reference position of the holding part when the finger parts 13a interfere with the object 30. A position in the actual workspace corresponding to a position within region 181 can be said to be a position where the finger parts 13a interfere with the object 30 on the workbench 40 when the reference position of the open holding part 13 exists at that position.

[0100] Next, in step s4, the determination unit 2c generates an inverted second convolutional binary image by inverting each pixel value of the second convolutional binary image 180. Then, the determination unit 2c calculates the logical AND of the inverted second convolutional binary image and the first convolutional binary image. Hereafter, the image obtained as a result of calculating the logical AND of the inverted second convolutional binary image and the first convolutional binary image will be called the logical AND image 190.

[0101] Figure 14 is a schematic diagram showing an example of a logical AND image 190. The positions of the multiple pixels constituting the logical AND image 190 correspond to multiple positions in the actual working space of the robot 10. In the logical AND image 190, the region 191 with a pixel value of "1" corresponds to the range of holding positions (also called the holding-possible position range) in which the opening / closing holding unit 13 can hold the object 30 on the workbench 40 without interfering with the object 30. The holding-possible position range can also be said to be the range of the holding unit reference position in which the opening / closing holding unit 13 can hold the object 30. If the holding unit reference position exists at a position in the actual working space corresponding to a position within region 191, the holding unit 13 can hold the object 30. A position in the actual working space corresponding to a position within region 191 can be said to be a position in which the holding unit 13 can hold the object 30 when the holding unit 13 is located at that position. Hereafter, region 191 may be referred to as the holding-possible position range equivalent region 191.

[0102] In step s4, the determination unit 2c determines a certain position within the range of holdable positions 191 as the holding position where the holding unit 13 holds the object 30. More specifically, the determination unit 2c determines a certain position within the range of holdable positions 191 as the holding reference position where the holding unit 13 holds the object 30. This determines the holding position. The position within the range of holdable positions 191 where the holding unit 13 holds the object 30 can also be said to be the holding position on the image, or it can also be said to be the holding reference position on the image where the holding unit 13 holds the object 30. The determination unit 2c determines a certain position within the range of holdable positions 191 as the holding position on the image.

[0103] There are various methods by which the determination unit 2c can determine a position within the retainable position range equivalent region 191 as the retainable position on the image. For example, the determination unit 2c may determine the centroid of the retainable position range equivalent region 191 as the retainable position on the image. Alternatively, the determination unit 2c may convert the logical AND image 190 into a distance-converted image, and based on the distance-converted image, determine a point within the retainable position range equivalent region 191 that is far from the contour (in other words, edge) of the retainable position range equivalent region 191 as the retainable position on the image.

[0104] The center of the logical AND image 190 corresponds, for example, to the center of the object 30 when viewed from directly above. Therefore, if the center of the logical AND image 190, which is located within the area equivalent to the holdable position range 191, is selected as the holding position on the image, then when the object 30 held by the holding unit 13 is viewed from directly above, the reference position of the holding unit coincides with the center of the object 30. On the other hand, if a position shifted from the center of the logical AND image 190, which is located within the area equivalent to the holdable position range 191, is selected as the holding position on the image, then when the object 30 held by the holding unit 13 is viewed from directly above, the reference position of the holding unit is shifted from the center of the object 30.

[0105] Furthermore, if the end effector 12 is equipped with a suction holding unit 13, the determination unit 2c uses the suction aperture shape image 120 included in the suction holding unit information as a kernel and convolves the suction aperture shape image 120 onto the object image 150. The determination unit 2c then binarizes the image (also called the third convolution image) that has pixel values ​​obtained by convolving the suction aperture shape image 120 onto the object image 150. The determination unit 2c determines a certain position within the region where the pixel value in the binarized third convolution image is "1" as the holding position on the image.

[0106] Thus, since the determination unit 2c determines the holding position based on the holding unit information and the object information, it can determine a holding position in which the holding unit 13 is likely to succeed in holding the object 30.

[0107] In step s4, once the holding position is determined, in step s5, the determination unit 2c determines the placement area in which the holding unit 13 can place the object 30. In step s5, the determination unit 2c acquires placement information 200 regarding the shapes of the holding unit 13 and the object 30 when the holding unit 13 places the object 30 at the placement location, based on the holding unit information acquired in step s1 and the holding position determined in step s4 (also called the determined holding position). As described above, the determination unit 2c determines the holding position based on the holding unit information and the object information, so it can also be said that the determination unit 2c acquires placement information 200 based on the holding unit information and the object information. The determination unit 2c determines the placement area based on the acquired placement information 200 and the placement location information 160 acquired in step s3. Step s5 will be described in detail below.

[0108] The placement information 200 represents, for example, the state of the holding unit 13 and the object 30 when the holding unit 13, which holds the object 30 at the determined holding position, places the object 30 at the designated placement location. The placement information 200 can also be described as information based on the shape of the holding unit 13 and the object 30 when the holding unit 13, which holds the object 30 at the determined holding position, places the object 30.

[0109] Figure 15 is a schematic diagram showing an example of placement information 200 relating to the shapes of the opening / closing holding unit 13 and the object 30 when the opening / closing holding unit 13 places the object 30. The placement information 200 is, for example, a binary image. Hereafter, the placement information 200 may be referred to as the placement image 200.

[0110] The placement image 200 includes an object-corresponding region 201 that corresponds to the object 30 held by the holding unit 13 at the determined holding position. The placement image 200 also includes an opening / closing region 202 that shows how the opening / closing holding unit 13 opens and closes when the object 30 is placed.

[0111] The opening / closing region 202 can also be said to represent the way the two finger portions 13a open when the opening / closing holding portion 13 places the object 30. Alternatively, the opening / closing region 202 can also be said to represent the region that each finger portion 13a passes through when the two finger portions 13a open and place the object 30. The pixel values ​​of the object-corresponding region 201 and the opening / closing region 202 are "1". In the placement image 200, the pixel values ​​of the regions other than the object-corresponding region 201 and the opening / closing region 202 are "0".

[0112] The center of the placement image 200 corresponds to the determined holding position. More specifically, the center of the placement image 200 corresponds to the reference position of the holding unit 13 when it holds the object 30 (also called the reference position during holding).

[0113] Figure 16 is a schematic diagram showing an example of how to acquire the placement image 200. The determination unit 2c calculates the logical OR of, for example, the opening / closing range image 100 and the open state shape image 110. Here, the image obtained as a result of calculating the logical OR of the opening / closing range image 100 and the open state shape image 110 is called the logical OR image 195.

[0114] The center of the logical OR image 195 corresponds to the reference position of the holding part. The region 196 in the logical OR image 195 where the pixel value is "1" is the region that is the sum of the opening / closing range equivalent region 101 included in the opening / closing range image 100 and the opening / closing holding part equivalent region 111 included in the open state shape image 110.

[0115] Next, the determination unit 2c superimposes the object-equivalent region 151 included in the object image 150 onto the logical OR image 195 at a position corresponding to the determination and holding position. The logical OR image 195 with the object-equivalent region 151 superimposed becomes the placement image 200. The object-equivalent region 151 superimposed onto the logical OR image 195 becomes the object-equivalent region 201. Furthermore, in the region of the logical OR image 195 with the object-equivalent region 151 superimposed where the pixel value is "1", the region other than the object-equivalent region 151 (in other words, the object-equivalent region 201) becomes the opening / closing state region 202.

[0116] Here, within the region 191 corresponding to the range of holdable positions included in the logical AND image 190, a certain position determined as the holdable position on the image is called the holdable position equivalent position. The holdable position equivalent position can also be said to correspond to the reference position during hold. For example, if the centroid of the region 191 corresponding to the range of holdable positions is selected as the holdable position on the image, then the centroid of the region 191 corresponding to the range of holdable positions becomes the holdable position equivalent position. The holdable position equivalent position becomes the determined holdable position on the image (in other words, the reference position during hold on the image).

[0117] If the center of the logical AND image 190 corresponding to the center of the object 30 coincides with the position corresponding to the holding position, the determination unit 2c superimposes the object-corresponding region 151 onto the logical OR image 195 and combines them so that the center of the object-corresponding region 151 coincides with the center of the logical OR image 195. The placement images 200 shown in Figures 15 and 16 are examples of placement images 200 generated when the position corresponding to the holding position coincides with the center of the logical AND image 190. In the placement images 200 shown in Figures 15 and 16, the center of the object-corresponding region 201 coincides with the center of the placement image 200 corresponding to the determined holding position. The placement images 200 shown in Figures 15 and 16 can be said to show that the reference position during holding (in other words, the determined holding position) coincides with the center of the object 30.

[0118] On the other hand, if the center of the logical AND image 190 corresponding to the center of the object 30 is offset from the position corresponding to the holding position, the determination unit 2c superimposes the object-corresponding region 151 onto the logical OR image so that the offset of the center of the object-corresponding region 151 from the center of the logical OR image 195 matches the offset of the center of the logical AND image 190 from the position corresponding to the holding position. For example, consider the case where the center of the logical AND image 190 is offset by 2 pixels to the right and 3 pixels upward from the position corresponding to the holding position. In this case, the determination unit 2c superimposes the object-corresponding region 151 onto the logical OR image so that the center of the object-corresponding region 151 is offset by 2 pixels to the right and 3 pixels upward from the center of the logical OR image 195.

[0119] The positional relationship between the center of the placement image 200 and the object-equivalent region 201 corresponds to the positional relationship between the determined holding position and the object 30 when the holding unit 13 holds the object 30 at the determined holding position. Therefore, it can be said that the placement image 200 also shows the positional relationship between the determined holding position (in other words, the reference position during holding) and the object 30 when the holding unit 13 holds the object 30 at the determined holding position.

[0120] Figure 17 is a schematic diagram showing an example of a placement image 200 generated when the center of the logical AND image 190 is offset from the position corresponding to the holding position. In the placement image 200 shown in Figure 16, the center of the object-corresponding region 201 is offset from the center of the placement image 200 corresponding to the determined holding position. The placement image 200 shown in Figure 17 can be said to show that the reference position during holding (in other words, the determined holding position) is offset from the center of the object 30.

[0121] Furthermore, if the end effector 12 includes an adsorption and holding section 13, the determination section 2c generates placement information 200 based on the adsorption aperture shape information 120 included in the adsorption and holding section information and the determined holding position. For example, the determination section 2c superimposes the object-equivalent region 151 included in the object image 150 onto the adsorption aperture shape image 120 at a position corresponding to the determined adsorption position and synthesizes them.

[0122] If the center of the logical AND image 190 coincides with the position corresponding to the holding position, the determination unit 2c superimposes the object-corresponding region 151 onto the suction aperture shape image 120 so that the center of the object-corresponding region 151 coincides with the center of the suction aperture shape image 120. Figure 18 is an example of a placement image 200 generated when the end effector 12 is equipped with a suction holding unit 13 and the position corresponding to the holding position coincides with the center of the logical AND image 190.

[0123] In this example, the size of the suction opening is smaller than the size of the object 30. Therefore, when the object-equivalent region 151 is superimposed and combined with the suction opening shape image 120 so that the center of the object-equivalent region 151 coincides with the center of the suction opening shape image 120, the placement image 200 shows only the object-equivalent region 201, which is the object-equivalent region 151, and does not show the suction opening equivalent region 121 that was included in the suction opening shape image 120. Consequently, the placement image 200 shows only the shape of the object 30, out of the suction holding unit 13 and the object 30 when the suction holding unit 13 places the object 30.

[0124] On the other hand, if the center of the logical AND image 190 is offset from the position corresponding to the holding position, the determination unit 2c superimposes the object-corresponding region 151 onto the suction opening shape image 120 so that the offset of the center of the object-corresponding region 151 from the center of the suction opening shape image 120 matches the offset of the center of the logical AND image 190 from the position corresponding to the holding position. In this case, the placement image 200 may show a part of the suction opening equivalent region 121. If the placement image 200 shows a part of the suction opening equivalent region 121, it can be said that the placement image 200 shows the shape of the suction holding unit 13 (specifically the suction opening) and the object 30 when the suction holding unit 13 places the object 30.

[0125] When the determination unit 2c acquires the placement information 200, it determines the placement area based on the acquired placement information 200 and the placement destination information 160 acquired in step s3. In step s5, the determination unit 2c performs a convolution operation between the placement image 200 and the placement destination image 160, for example. Specifically, the determination unit 2c uses the placement image 200 as a kernel to convolve the placement image 200 into the placement destination image 160, similar to the case where the opening / closing range image 100 is convolved into the object image 150. Hereafter, the image having pixel values ​​obtained by the convolution operation between the placement image 200 and the placement destination image 160 is called the fourth convolution image.

[0126] Next, the determination unit 2c generates an inverted fourth convolution image 210 by inverting the pixel values ​​of the fourth convolution image. This inverted fourth convolution image 210 becomes placementable area information representing the placementable area. The inverted fourth convolution image 210 can also be called a placementable area image representing the placementable area. Hereafter, the inverted fourth convolution image 210 may be referred to as the placementable area image 210.

[0127] Figure 19 is a schematic diagram showing an example of a placeable area image 210 obtained based on the placement image 200 shown in Figure 15 and the placement destination image 160 shown in Figure 11. The placeable area image 210 is, for example, a binary image. The positions of the multiple pixels constituting the placeable area image 210 correspond to multiple positions in the actual working space of the robot 10.

[0128] In the fourth convolutional binary image that forms the basis of the placement area image 210, the areas with a pixel value of "1" correspond to the range of positions of the holding unit 13 (in other words, the range of the holding unit's reference position) in which, if the holding unit 13 attempts to place the object 30, at least one of the holding unit 13 or the object 30 interferes with an obstacle.

[0129] On the other hand, in the placementable area image 210, the area 211 with a pixel value of "1" corresponds to the range of positions of the holding unit 13 such that when the holding unit 13 places the object 30, the holding unit 13 and the object 30 do not interfere with obstacles at the placement site. More specifically, area 211 corresponds to the range of positions of the holding unit 13 such that when the holding unit 13, which holds the object 30, releases the holding of the object 30 and places the object 30, the holding unit 13 and the object 30 do not interfere with obstacles at the placement site. Area 211 can also be said to correspond to the range of reference positions of the holding unit such that when the holding unit 13 places the object 30, the holding unit 13 and the object 30 do not interfere with obstacles at the placement site. Area 211 can be said to correspond to the range of reference holding positions such that the holding unit 13 can place the object 30.

[0130] Each position within region 211 corresponds to a position of the holding unit 13 such that when the holding unit 13 places the object 30, the holding unit 13 and the object 30 do not interfere with any obstacles at the placement location. Each position within region 211 corresponds to a reference position of the holding unit 13 in which the object 30 can be placed. If a reference position of the holding unit exists in the actual workspace corresponding to a position within region 211, the holding unit 13 can place the object 30 in such a way that the holding unit 13 and the object 30 do not interfere with any obstacles. A position in the actual workspace corresponding to a position within region 211 can be said to be a position where, if the holding unit 13 places the object 30, the holding unit 13 and the object 30 are unlikely to interfere with any obstacles.

[0131] Hereafter, the range of positions of the holding unit 13 such that the holding unit 13 and the object 30 do not interfere with obstacles at the placement site when the holding unit 13 places the object 30 may be referred to as the placementable position range. The placementable position range can also be said to be the range of reference positions of the holding unit such that the holding unit 13 and the object 30 do not interfere with obstacles at the placement site when the holding unit 13 places the object 30. Furthermore, the region 211 may be referred to as the placementable position range equivalent region 211. The placementable position range equivalent region 211 can be said to be the placementable position range on the image.

[0132] The placement range determines the placement area in which the holding unit 13 can place the object 30 at the placement destination. Therefore, the placement range equivalent area 211 can be said to represent the placement area. The determination unit 2c determines the placement area by acquiring a placement area image 210 that represents the placement area.

[0133] Once the placement area is determined in step s5, in step s6, the determination unit 2c determines the placement position in the placement area where the holding unit 13 will place the object 30.

[0134] In step s6, the determination unit 2c determines a certain position within the area equivalent to the possible placement range 211 as the position of the holding unit 13 when the holding unit 13 places the object 30 (more specifically, the holding unit reference position). In other words, the determination unit 2c determines a certain position within the area equivalent to the possible placement range 211 as the position of the holding unit 13 on the image when the holding unit 13 places the object 30. By determining the position of the holding unit 13 when the holding unit 13 places the object 30, the placement position where the holding unit 13 places the object 30, in other words, the position where the object 30 is placed by the holding unit 13, is determined. Hereafter, the position of the holding unit 13 when the holding unit 13 places the object 30 may be called the placement holding unit position. The placement holding unit position can also be said to be the holding unit reference position when the holding unit 13 places the object 30.

[0135] There are various methods by which the determination unit 2c can determine a certain position within the area equivalent to the configurable position range 211 as the position of the placement and retention unit on the image. In the example of Figure 19, the determination unit 2c may, for example, determine the centroid of the area enclosed by the area equivalent to the configurable position range 211 as the position of the placement and retention unit on the image. Alternatively, the determination unit 2c may convert the configurable area image 210 into a distance-converted image, and based on the distance-converted image, determine a point far from or close to the contour (in other words, edge) of the area equivalent to the configurable position range 211 as the position of the placement and retention unit on the image.

[0136] In this way, the determination unit 2c determines the placement area in which the holding unit 13 can place the object 30 based on the object information, holding unit information, and placement destination information, thereby enabling the appropriate determination of the placement area. As a result, the placement position is determined from the determined placement area, enabling the holding unit 13 to place the object 30 appropriately and reducing the likelihood of failure in placing the object 30.

[0137] In step s6, once the placement position is determined, in step s7, the robot control unit 2a controls the robot 10 to cause the holding unit 13 to hold the object 30 at the holding position determined in step s4. In step s7, first, the robot control unit 2a controls the posture of the arm 11 so that the reference position of the holding unit is located at a position in the actual work space corresponding to a position within the holding range equivalent area 191, which has been determined as the holding position on the image. Then, the robot control unit 2a controls the end effector 12 to cause the holding unit 13 to hold the object 30. As a result, the holding unit 13 holds the object 30 at the holding position determined in step s4.

[0138] Once the holding unit 13 holds the object 30, in step s8, the robot control unit 2a controls the robot 10 to cause the holding unit 13 to place the object 30 at the placement position determined in step s6. In step s8, first, the robot control unit 2a controls the posture of the arm 11 so that the reference position of the holding unit is located at a position in the actual work space corresponding to a position within the placement range equivalent area 211, which has been determined as the placement holding unit position on the image. Then, the robot control unit 2a controls the end effector 12 to cause the holding unit 13 to release the holding of the object 30. As a result, the holding unit 13 places the object 30 at the placement position determined in step s6.

[0139] Furthermore, if there are multiple objects 30 on the workbench 40, the processing device 1 repeatedly executes the process consisting of steps s1 to s8, thereby allowing the robot 10 to place the multiple objects 30 on the workbench 40 one by one onto the workbench 45.

[0140] The order in which steps s1 to s8 are executed is not limited to the example above. For example, steps s1 and s2 may be executed at any time before steps s4 and s5. For example, step s1 may be executed before steps s4 and s5 and after step s2. Also, step s3 may be executed at any time before step s5. For example, step s3 may be executed between steps s4 and s5. Also, step s7 may be executed at any time between steps s4 and s8. For example, step s7 may be executed between steps s4 and s5.

[0141] <Other examples of methods for determining placement location> <Another example of the first> Depending on the content of the work to be performed on the object 30 after the robot 10 has placed the object 30, it may be desirable to place the object 30 in a location that is difficult to move after placement.

[0142] For example, consider the case shown in Figure 20, where the placement jig 35 has a large frame-shaped portion 35c and a small frame-shaped portion 35d on the base portion 35a. In this case, it may be desirable for the object 30 to be placed in the small frame-shaped portion 35d, as shown in Figure 20, rather than in the large frame-shaped portion 35c.

[0143] As another example, consider the case shown in Figure 21, where three rectangular parallelepipeds 35e are placed on the base portion 35a of the placement jig 35. In this case, the T-shaped object 30 may be placed as shown in Figure 21, or as shown in Figure 22. Depending on the work to be done on the object 30 after placement, it may be desirable to place the object 30 in a location where it is difficult to move after placement, as shown in Figure 22.

[0144] In this example, the determination unit 2c acquires constraint information, which represents the constraints imposed on the object 30 located in the placement area by surrounding objects (in other words, surrounding obstacles), based on the object information 150 and the placement location information 160. Then, the determination unit 2c determines the placement position based on the acquired constraint information. As a result, the object 30 is more likely to be placed in a location that is difficult to move after placement. In other words, the object 30 is more likely to be placed in a location that is constrained by its surroundings. This example will be explained in detail below.

[0145] The determination unit 2c generates a binary image 250 obtained by changing the pixel values ​​of the opening / closing state region 202 from "1" to "0" in the placement image 200 acquired based on the holding unit information and the object information 150.

[0146] Figure 23 is a schematic diagram showing an example of a binary image 250. Figure 23 shows a binary image 250 obtained by changing the pixel value of the opening / closing area 202 from "1" to "0" in the arrangement image 200 shown in Figure 17. The binary image 250 includes an object-corresponding area 251 corresponding to the object 30. The pixel value of the object-corresponding area 251 is "1". In the binary image 250, the pixel value of areas other than the object-corresponding area 251 is "0".

[0147] The center of the binary image 250 corresponds to the reference position of the holding unit. The positional relationship between the center of the binary image 250 and the object-equivalent region 251 corresponds to the positional relationship between the reference position of the holding unit (in other words, the determined holding position) and the object 30 when the holding unit 13 holds the object 30. Therefore, it can be said that the binary image 250 shows the positional relationship between the determined holding position and the object 30 when the holding unit 13 holds the object 30 at the determined holding position. Furthermore, the binary image 250 can be said to be object information (or object image) representing the object 30, or object shape information (or object shape image) representing the shape of the object 30.

[0148] Next, the determination unit 2c performs a convolution operation between the binary image 250 and the placement image 160. Specifically, the determination unit 2c uses the binary image 250 as a kernel and convolves it onto the placement image 160, similar to when the opening / closing range image 100 is convolved onto the object image 150. Hereafter, the image having pixel values ​​obtained by the convolution operation between the binary image 250 and the placement image 160 will be called the fifth convolution binary image 260.

[0149] Figure 24 is a schematic diagram showing an example of a fifth convolutional binary image 260. The positions of the multiple pixels constituting the fifth convolutional binary image 260 correspond to multiple positions in the actual working space. In the fifth convolutional binary image 260, the region 261 where the pixel value is "1" corresponds to the range of reference positions of the holding unit 13 where the object 30 interferes with an obstacle when the holding unit 13 attempts to place the object 30, focusing only on the object 30 in relation to interference with obstacles between the holding unit 13 and the object 30. Each position within region 261 corresponds to a reference position of the holding unit where the object 30 interferes with an obstacle when the holding unit 13 attempts to place the object 30. A position in the actual working space corresponding to a certain position within region 261 is a position where the object 30 interferes with an obstacle when the holding unit 13 attempts to place the object 30, given that a reference position of the holding unit exists at that position. Region 261 can also be said to represent the range where obstacles exist. Region 261 can also be described as representing the area where obstacles exist, based on the reference position of the holding part.

[0150] Next, the determination unit 2c performs a process to extract edges from the fifth convolution image 260 and obtains an edge binary image 270 representing those edges. Figure 25 is a schematic diagram showing an example of an edge binary image 270.

[0151] The positions of the multiple pixels that make up the edge binary image 270 correspond to multiple positions in the actual working space. In the edge binary image 270, the region 271 with a pixel value of "1" corresponds to the edge of the fifth convolutional binary image. The edge of the fifth convolutional binary image can also be said to be the contour of region 261. Hereafter, region 271 may be referred to as the edge equivalent region 271. The edge equivalent region 271 can also be said to be the region corresponding to the contour of region 261.

[0152] The edge-equivalent region 271 corresponds to the contour of the range of reference positions of the holding unit 13 where the object 30 would interfere with an obstacle if the holding unit 13 were to place the object 30. The position on the contour of the range of reference positions of the holding unit where the object 30 would interfere with an obstacle if the holding unit 13 were to place the object 30 can be seen as the reference position of the holding unit when the object 30 placed in the placeable area collides with an obstacle when it moves. Therefore, the edge-equivalent region 271 can also be said to represent the range of points where the object 30 placed in the placeable area collides with an obstacle when it moves.

[0153] The determination unit 2c acquires constraint information representing the constraints imposed by surrounding objects (i.e., obstacles) on the object 30 located in the placeable area, based on the edge binary image 270 and the placeable area image 210.

[0154] Here, each pixel that constitutes the area equivalent to the locatable position range 211 included in the locatable area image 210 is called a first pixel. Also, in the edge binary image 270, a pixel that is at the same position as the first pixel is called a second pixel. The edge binary image 270 contains multiple second pixels, each corresponding to one of the multiple first pixels that constitute the area equivalent to the locatable position range 211. Among the multiple second pixels, the second pixel of interest, or in other words, the second pixel to be explained, is called the second pixel of interest.

[0155] The determination unit 2c determines the shortest distance d from the position p of the target second pixel (also called the target second pixel position p) to the edge equivalent region 271 for each of the multiple directions. Specifically, the determination unit 2c determines the shortest distance d from the target second pixel position p to the edge equivalent region 271 for each of the multiple directions extending from the target second pixel position p, each of which has a different angle from one another.

[0156] For example, as shown in Figure 26, the determination unit 2c sets a direction 301 (also called the measurement direction 301) in which the angle θ from the reference line 300 passing through the target second pixel position p is. The reference line 300 is set, for example, in the horizontal direction (in other words, the row direction) of the edge binary image 270. The angle θ is set counterclockwise from the reference line 300. The determination unit 2c changes the angle θ from the reference line 300 for the measurement direction 301 within a range of 0 degrees to less than 360 degrees. Then, for each value of angle θ, the determination unit 2c finds the shortest distance d from the target second pixel position p to the edge equivalent region 271 in the measurement direction 301. In this way, the shortest distance d from the target second pixel position p to the edge equivalent region 271 in each of the multiple measurement directions 301 with different angles θ is found.

[0157] For example, if the angle θ changes from 0 degrees to 345 degrees in increments of 15 degrees, 24 shortest distances d can be determined, corresponding to 24 measurement directions 301.

[0158] As can be understood from the explanation so far, a certain position in the actual workspace corresponding to the target second pixel position p can be considered as the reference position of the holding unit 13 when the holding unit 13 places the object 30 in the placeable area. And, since the placement position of the object 30 within the placeable area is determined when the reference position of the holding unit 13 is determined when the holding unit 13 places the object 30 in the placeable area, it can be said that the target second pixel position p represents a certain position within the placeable area. Also, the edge equivalent area 271 represents the range of points where the object 30 placed in the placeable area will collide with an obstacle when it moves. Therefore, the shortest distance d from the target second pixel position p to the edge equivalent area 271 in a certain measurement direction 301 can be said to be distance information representing the distance the object 30, placed at a certain position within the placeable area, will move in a certain direction until it collides with an obstacle (in other words, surrounding objects). It can be said that the determination unit 2c acquires distance information for each of the multiple directions that represents the distance the object 30 in the placeable area will move in that direction until it collides with an obstacle. More specifically, when an object 30 is placed at a certain position within the placement area, the determination unit 2c acquires distance information representing the distance the object 30 will travel in each of the multiple directions extending from that position, each of which has a different angle to each other, until it collides with an obstacle after moving in that direction. If we call the direction in the actual workspace corresponding to the measurement direction 301 the second measurement direction, then the determination unit 2c acquires distance information representing the distance the object 30 will travel in each of the multiple second measurement directions, each of which has a different angle to each other, until it collides with an obstacle after moving in that second measurement direction within the placement area.

[0159] Hereafter, a certain position within the placement area corresponding to the target second pixel position p may be referred to as the target position in the placement area. The position in the actual workspace corresponding to the target second pixel position p can be said to be the reference position of the holding unit 13 when the holding unit 13 places the object 30 at the target position in the placement area.

[0160] The determination unit 2c acquires constraint information based on distance information acquired for multiple directions. For example, the determination unit 2c uses the sum of the shortest distances d acquired for multiple measurement directions 301 as constraint information representing the constraint on the object 30 located in the placement area by surrounding objects. The smaller the value of the constraint information obtained for the second target pixel (the sum of the shortest distances d in this example), the higher the constraint on the object 30 placed at the target position in the placement area by surrounding objects. In other words, the smaller the value of the constraint information obtained for the second target pixel, the more difficult it becomes for the object 30 placed at the target position in the placement area to move. The value of the constraint information can also be called the degree of constraint. The determination unit 2c may also use the average value of the shortest distances d acquired for multiple measurement directions 301 as constraint information.

[0161] Figure 27 is a schematic diagram showing an example of a histogram 310 of multiple shortest distances d acquired by the determination unit 2c. The horizontal axis of the histogram 310 represents the angle θ, and the vertical axis of the histogram 310 represents the shortest distance d.

[0162] The upper part of Figure 27 shows an example of how the object 30 is positioned in the positioning jig 35. In the positioning jig 35 shown in the upper part, a small frame-shaped part 35d is provided on top of the base part 35a. The object 30 is positioned inside the frame-shaped part 35d. The lower part of Figure 27 shows an example of a histogram 310 of the shortest distance d, which represents the distance the object 30, positioned as shown in the upper part, will travel before colliding with the obstacle, the frame-shaped part 35d. Since the object 30 positioned inside the frame-shaped part 35d cannot move, the shortest distance d in each direction is zero. Therefore, the constraint information value is zero.

[0163] Figures 28-30 are schematic diagrams showing other examples of histograms 310. Similar to Figure 27, the upper part of Figures 28-30 shows an example of how the object 30 is positioned on the positioning jig 35. The lower part of Figures 28-30 shows an example of a histogram 310 of the shortest distance d, which represents the distance the object 30 moves before colliding with a surrounding object when it moves as positioned as shown in the upper part.

[0164] In the placement jig 35 shown in Figure 28, four L-shaped sections 35g, which are rectangular parallelepipeds bent into an L-shape, are provided on the base section 35a. The object 30 is positioned in the center of the area enclosed by the four L-shaped sections 35g, in other words, in the center of the base section 35a. In this case, the movement of the object 30 is restricted to some extent all around, so the constraint information value is relatively small, as can be seen from the histogram 310 at the bottom of Figure 28.

[0165] The placement jig 35 shown in Figure 29 is the same as the placement jig 35 shown in Figure 28, but with the two L-shaped sections 35g on the right half removed. In this case, the object 30 is difficult to move to the left, but easy to move to the right, up, and down. As can be seen from the histogram 310 at the bottom of Figure 29, the constraint information values ​​become relatively large.

[0166] In the placement jig 35 shown in Figure 30, nothing is provided on the base portion 35a. Therefore, there are no obstacles at the placement site. In this case, the object 30 is not constrained by its surroundings and can move freely around its entire perimeter. As a result, as can be seen from the histogram 310 at the bottom of Figure 30, the value of the constraint information becomes quite large.

[0167] In step s6, the determination unit 2c acquires constraint information for each of the multiple second pixels included in the edge binary image 270. Next, the determination unit 2c identifies the constraint information with the smallest value among the multiple acquired constraint information. Then, the determination unit 2c determines the position of the first pixel at the same position as the second pixel corresponding to the constraint information with the smallest value as the position corresponding to the reference position of the holding unit 13 when the holding unit 13 places the object 30 (in other words, the placement holding unit position). It can also be said that the determination unit 2c determines the position of the first pixel at the same position as the second pixel corresponding to the constraint information with the smallest value as the placement holding unit position on the image. It can also be said that the determination unit 2c determines the position in the actual workspace corresponding to the position of the second pixel corresponding to the constraint information with the smallest value as the placement holding unit position. By determining the placement holding unit position (in other words, the reference position of the holding unit 13 when the holding unit 13 places the object 30), the placement position in which the holding unit 13 places the object 30 is determined.

[0168] In this way, by the determination unit 2c determining the placement position based on the constraint information, the object 30 is more likely to be placed in a location that is difficult to move after placement (in other words, a location that is constrained by its surroundings).

[0169] Furthermore, as in this example, the determination unit 2c can appropriately acquire constraint information by obtaining distance information representing the distance traveled by the object 30 located in the placement area until it collides with a surrounding object when it moves in that direction, based on the object information and the placement destination information, and by acquiring constraint information based on the distance information acquired for the multiple directions.

[0170] <Detailed explanation of an example of how to obtain distance information> Here, we will explain in detail an example of how to obtain the shortest distance d as distance information. Figures 31 to 34 are schematic diagrams illustrating an example of how to obtain the shortest distance d.

[0171] The edge-equivalent region 271 is not actually composed of a line, but rather of multiple pixels 272. Hereafter, the set of multiple pixels 272 that constitute the edge-equivalent region 271 will be referred to as set E.

[0172] When determining the shortest distance d, the determination unit 2c first sets a straight line 305 that passes through the target second pixel position p and forms an angle θ with the reference line 300 for the edge binary image 270, as shown in Figure 31.

[0173] Next, the determination unit 2c identifies a plurality of pixels 272 in set E whose distance from the line 501 is less than or equal to a first threshold. The first threshold may be set to, for example, a distance of two pixels. As the distance between a pixel 272 and the line 501, for example, the distance between the intersection point of the perpendicular line drawn from the pixel 272 to the line 501 and the line 501, and the pixel 272 is used. Hereafter, the set of a plurality of pixels 272 whose distance from the line 501 is less than or equal to the first threshold will be called set A. Also, the pixels 272 in set A, that is, the pixels 272 whose distance from the line 501 is less than or equal to the first threshold, will be called pixels 272a.

[0174] Next, the determination unit 2c determines the first angle from the reference line 300 for each pixel 272a in set A, which is the straight line connecting the pixel 272a and the target second pixel position p. The first angle is set counterclockwise, similar to the angle θ. Next, the determination unit 2c determines the absolute value of the difference between the first angle determined for each pixel 272a in set A and the angle θ. Then, the determination unit 2c defines set B as the set of pixels 272a in set A whose first angle corresponds to an angle for which the absolute value of the difference with angle θ is 90 degrees or less. Hereafter, pixels 272a in set B will be called pixels 272b. Figure 32 is a schematic diagram showing an example of pixels 272a in set A. Figure 32 also shows an example of pixels 272b in set B.

[0175] Next, the determination unit 2c divides the pixels 272b of set B into pixels 272b of set B0, pixels 272b of set B1, and pixels 272b of set B2. The pixels 272b of set B0 are the pixels 272b of set B that correspond to the first angle that coincides with angle θ. The pixels 272b of set B0 can also be said to be the pixels 272b of set B that lie on the line 305. The pixels 272b of set B1 are the pixels 272b of set B that correspond to the first angle that is greater than angle θ. The pixels 272b of set B2 are the pixels 272b of set B that correspond to the first angle that is smaller than angle θ. Hereafter, the pixels 272b of set B1 will be called pixels 272b1, and the pixels 272b of set B2 will be called pixels 272b2. Figure 33 is a schematic diagram showing an example of a pixel 272b in set B. Figure 33 also shows examples of pixel 272b1 in set B1 and pixel 272b2 in set B2.

[0176] Next, the determination unit 2c determines the distance 315 (see Figure 33) between pixels 271b1 and 271b2 that constitute each combination of pixels 272b1 in set B1 and pixels 272b2 in set B2. Then, the determination unit 2c identifies the combinations from all the combinations of pixels 272b1 in set B1 and pixels 272b2 in set B2 for which the distance 315 is less than or equal to a second threshold. The second threshold may be set to, for example, a distance of two pixels.

[0177] Hereafter, combinations corresponding to a distance 315 below the second threshold will be referred to as specific combinations. Furthermore, pixels 271b1 and 271b2 that constitute a specific combination will be referred to as pixels 271b1a and 271b2a, respectively.

[0178] Next, as shown in Figure 34, the determination unit 2c determines a straight line 311 for each specific combination, connecting the pixels 271b1a and 271b2a that constitute that specific combination. Then, for each determined straight line 311, the determination unit 2c determines the intersection point 312 between the straight line 311 and the straight line 305.

[0179] Next, the determination unit 2c identifies the position closest to the target second pixel position p from among the positions of each intersection point 312 and each pixel 271b of set B0. The determination unit 2c then takes the distance between the identified position closest to the target second pixel position p and the target second pixel position p as the shortest distance d from the target second pixel position p to the edge equivalent region 271 in the direction 301 that forms an angle θ with the reference line 300.

[0180] Figure 35 is a schematic diagram showing an example of an object image 150 for multiple types of objects 30. Figure 36 is a schematic diagram showing an example of a placement location image 160. Figure 37 is a schematic diagram showing an example of a placement position determined by the determination unit 2c for multiple types of objects 30.

[0181] Figure 35 shows, as examples of object images 150, object image 150a including an object-equivalent region 151a corresponding to the first type of object 30, object image 150b including an object-equivalent region 151b corresponding to the second type of object 30, object image 150c including an object-equivalent region 151c corresponding to the third type of object 30, and object image 150d including an object-equivalent region 151d corresponding to the fourth type of object 30. The obstacle-equivalent region 161 included in the placement destination image 160 (also called placement destination image 160a) shown in Figure 36 represents the shape of the obstacle at the placement destination.

[0182] The upper left of Figure 37 shows how, in the placement destination image 160a, an object-equivalent region 151a corresponding to the first type of object 30 is placed at a position corresponding to the placement location of the first type of object 30 determined based on the object image 150a and the placement destination image 160a.

[0183] In the upper right of Figure 37, the object-equivalent region 151b corresponding to the second type of object 30 is shown in the placement image 160a, at the position corresponding to the placement location of the second type of object 30 determined based on the object image 150b and the placement image 160a.

[0184] The lower left of Figure 37 shows how, in the placement destination image 160a, an object-equivalent region 151c corresponding to the third type of object 30 is placed at a position corresponding to the placement location of the third type of object 30 determined based on the object image 150c and the placement destination image 160a.

[0185] In the lower right of Figure 37, the object-equivalent region 151d corresponding to the fourth type of object 30 is shown in the placement destination image 160a, at the position corresponding to the placement location of the fourth type of object 30 determined based on the object image 150d and the placement destination image 160a.

[0186] As can be seen from Figure 37, each of the first to fourth types of objects 30 is positioned in a way that constrains it to its surroundings.

[0187] <Another example for the second object 30> If the placement position determined by the determination unit 2c for the object 30 is close to an obstacle, depending on the accuracy of the determination of the placement position, when the holding unit 13 attempts to place the object 30 at that position, the holding unit 13 or the object 30 may interfere with the obstacle, potentially resulting in failure to place the object 30. Also, depending on the accuracy of the conversion when converting the position on the image to the position in the actual workspace, when the holding unit 13 attempts to place the object 30 at the placement position determined by the determination unit 2c, the holding unit 13 or the object 30 may interfere with the obstacle.

[0188] Therefore, in this example, the determination unit 2c acquires uniformity information, which represents the uniformity of constraints imposed by surrounding objects on the object 30 located in the placeable area, based on the object information 150 and the placement location information 160. Then, the determination unit 2c determines the placement position based on the acquired uniformity information. This makes it less likely that a position close to surrounding objects (in other words, obstacles) will be determined as the placement position. The details of this example will be explained below.

[0189] The determination unit 2c, in the same manner as described above, determines the shortest distance d from the target second pixel position p to the edge equivalent region 271 for each of the multiple measurement directions 301. Then, the determination unit 2c acquires uniformity information based on the shortest distances d obtained for the multiple measurement directions 301. For example, the determination unit 2c obtains the chi-squared statistic x of the multiple shortest distances d obtained. ２ This is considered uniformity information.

[0190] Here, K represents the number of shortest distances d obtained by the determination unit 2c. Furthermore, di represents the i-th shortest distance d when the shortest distances d obtained by the determination unit 2c are numbered from 1 to K. E represents the average value of the K shortest distances d. x is the chi-squared statistic of the K shortest distances d. ２ The result is expressed by the following equation (1). The decision unit 2c uses equation (1) to obtain the chi-squared statistic x of the K shortest distances d. ２ We seek.

[0191]

[0192] The value of uniformity information obtained for the second target pixel (in this example, the chi-squared statistic x) ２ The smaller the value of ), the higher the uniformity of the constraints imposed by surrounding objects on the object 30 placed at the target position in the placement area. In other words, the smaller the value of uniformity information obtained for the second pixel of the object, the further the object 30 placed at the target position in the placement area will be from obstacles to a similar extent in all directions, and the less likely it is that obstacles will be located near the object 30.

[0193] Figures 38 and 39 are schematic diagrams showing an example of a histogram 310 of multiple shortest distances d acquired by the determination unit 2c. The upper part of Figures 38 and 39 shows an example of how the object 30 is positioned on the placement jig 35. In the placement jig 35 shown in the upper part of Figures 38 and 39, an annular part 35h is provided on the base part 35a. The object 30 is positioned inside the annular part 35h. The arrows shown in the upper part of Figures 38 and 39 indicate the distance the object 30 moves until it collides with the annular part 35h. The lower part of Figures 38 and 39 shows an example of a histogram 310 of the shortest distances d, which represents the distance the object 30 moves until it collides with the annular part 35h when positioned as shown in the upper part.

[0194] In the example shown in Figure 38, the object 30 is located in the center of the inside of the annular portion 35h, and the object 30 is roughly the same distance from the annular portion 35h all around. In this case, as shown in the lower histogram 310 of Figure 38, the multiple shortest distances d are uniform, and the uniformity information value is small. On the other hand, in the example shown in Figure 39, the object 30 is located near the annular portion 35h. In this case, as shown in the lower histogram 310 of Figure 39, the multiple shortest distances d are scattered, and the uniformity information value is large.

[0195] In step s6, the determination unit 2c acquires uniformity information for each of the multiple second pixels included in the edge binary image 270. Next, the determination unit 2c identifies the uniformity information with the smallest value from among the multiple acquired uniformity information. Then, the determination unit 2c determines the position of the first pixel at the same position as the second pixel corresponding to the uniformity information with the smallest value as the position corresponding to the reference position of the holding unit 13 when the holding unit 13 places the object 30 (in other words, the placement holding unit position). This determines the placement position in which the holding unit 13 places the object 30.

[0196] In this way, by determining the placement position based on uniformity information, the determination unit 2c makes it less likely that a position close to an obstacle will be determined as the placement position for the object 30. As a result, when the holding unit 13 places the object 30 at the placement position determined by the determination unit 2c, the holding unit 13 and the object 30 are less likely to interfere with the obstacle, making it easier to successfully place the object 30.

[0197] Furthermore, as in this example, the determination unit 2c can appropriately acquire uniformity information by obtaining distance information representing the distance traveled by the object 30 located in the placement area until it collides with a surrounding object when it moves in that direction, based on the object information and the placement destination information, for each of the multiple directions, and by acquiring uniformity information based on the distance information acquired for the multiple directions.

[0198] The determination unit 2c may determine the placement position based on constraint information and uniformity information. In this case, for example, the determination unit 2c may determine the constraint information value (e.g., the sum of K shortest distances d) and the uniformity information value (e.g., the chi-squared statistic x of the K shortest distances d) for each of the multiple second pixels. ２ The evaluation value obtained by adding the following values ​​may be acquired. The determination unit 2c then determines the position of the first pixel at the same position as the second pixel corresponding to the smallest evaluation value among the acquired evaluation values ​​as the position corresponding to the reference position of the holding unit 13 when the holding unit 13 places the object 30. This makes it difficult for the object 30 to move after placement and reduces the possibility that a location extremely close to an obstacle is determined as the placement position. Thus, it is possible to increase the constraint on the object 30 from its surroundings after placement while reducing the possibility of failure in placing the object 30. Note that the evaluation value may be obtained by weighting and adding the constraint information value and the uniformity information value.

[0199] Figure 40 is a schematic diagram showing an example of an object image 150. The object image 150 (also called object image 150e) shown in Figure 40 includes an object-equivalent region 151 (also called object-equivalent region 151e) corresponding to the fifth type of object 30.

[0200] Figure 41 is a schematic diagram showing an example of the placement position of the fifth type of object 30 (also called the placement position of the fifth type of object 30 based on constraint information) determined by the determination unit 2c based on the object image 150e and the placement destination image 160a shown in Figure 36, using the acquired constraint information. Figure 41 shows how the object-equivalent region 151e corresponding to the fifth type of object 30 is placed in the placement destination image 160a at the position corresponding to the placement position of the fifth type of object 30 based on constraint information.

[0201] Figure 42 is a schematic diagram showing an example of the placement position of the fifth type of object 30 (also called the placement position of the fifth type of object 30 based on the constraint information and uniformity information) determined by the determination unit 2c based on the object image 150e and the placement destination image 160a, after the determination unit 2c has acquired constraint information and uniformity information. Figure 42 shows how the object-equivalent region 151e corresponding to the fifth type of object 30 is placed in the placement destination image 160a at a position corresponding to the placement position of the fifth type of object 30 based on the constraint information and uniformity information.

[0202] In the example in Figure 41, the object 30 after placement is easily moved in a certain direction (a certain direction in the actual work space corresponding to the lower side of Figure 41). In contrast, in the example in Figure 42, the object 30 after placement is surrounded by obstacles on all sides, making it difficult to move the object 30 on all sides.

[0203] In the example above, the processing unit 1 generates the placement image 160 based on the camera image 21. However, the processing unit 1 may also generate a region 161 (also called a mask 161) in the placement image 160 where the pixel value is "1", based on instructions from the user. In this case, the user may specify the shape of the mask 161 to the processing unit 1 through an input unit (e.g., a mouse or touch sensor) provided by the processing unit 1.

[0204] Furthermore, in the above example, the object 30 is placed in a location where an obstacle exists, but it may also be placed in a location where there are no obstacles. In this case, the mask 161 included in the placement destination image 160 may represent not an obstacle, but an area where the placement of the object 30 is prohibited (also called a prohibited placement area) at the placement destination. The mask 161 can also be said to be a prohibited placement area equivalent area 161. The prohibited placement area equivalent area 161 represents the shape of the prohibited placement area at the placement destination (for example, the top surface of the workbench 45).

[0205] If the placement destination image 160 includes a region equivalent to the prohibited placement area 161, then in the placementable area image 210 (see Figure 19), the region 211 with a pixel value of "1" corresponds to the range of positions of the holding unit 13 that allow the holding unit 13 to place the object 30 in an area other than the prohibited placement area (in other words, an area where placement is permitted). The determination unit 2c can determine the placement position while avoiding the prohibited placement area by determining the position of the placement holding unit on the image from the region 211. As a result, the object 30 is placed in an area other than the prohibited placement area.

[0206] The shape of the area equivalent to the prohibited placement area 161 may be specified by the user. In this case, the user may specify the shape of the area equivalent to the prohibited placement area 161 to the processing device 1, for example, through the input unit provided by the processing device 1.

[0207] Furthermore, when the camera 20 captures an AR marker (AR is an abbreviation for Augmented Reality), a virtual area where placement is prohibited may appear in the camera image 21. In other words, the camera image 21 may include an image representing the area where placement is prohibited. In this case, the destination image 160 is generated based on the camera image 21 in which the area where placement is prohibited virtually appears, thereby generating a destination image 160 that includes an area equivalent to the area where placement is prohibited 161.

[0208] Alternatively, an object such as tape may be temporarily placed in the no-placement area, the temporarily placed object may be photographed by the camera 20, and the temporarily placed object may be removed after the camera 20 has taken the photograph. In this case, the processing unit 1 may generate the destination image 160 based on the camera image 21 obtained when the temporarily placed object is photographed by the camera 20. The mask 161 included in the destination image 160 generated by the processing unit 1 represents the object temporarily placed in the no-placement area, so the mask 161 can be said to be the area equivalent to the no-placement area 161. In this way, a destination image 160 including the area equivalent to the no-placement area 161 may be generated.

[0209] In the example above, a camera image 21 obtained from a camera 20 fixed to the end effector 12 is used to determine the placement position of the object 30. However, instead of camera image 21, a camera image obtained from a camera with a fixed position and fixed shooting range may be used. Also, a grayscale image may be used instead of a color image to determine the placement position of the object 30.

[0210] As described above, the processing apparatus has been explained in detail, but the above description is illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be combined and applied insofar as they do not contradict each other. And it is understood that countless examples not illustrated can be conceived without falling outside the scope of this disclosure.

[0211] This disclosure includes the following:

[0212] In one embodiment, (1) the processing apparatus includes an acquisition unit that acquires object information relating to the shape of an object to be held in a holding unit, holding unit information relating to the shape of the holding unit, and placement information relating to the placement location of the object, and a determination unit that determines a placement area in which the holding unit can place the object based on the object information, the holding unit information, and the placement location information acquired by the acquisition unit.

[0213] (2) The processing apparatus of (1) above, wherein the acquisition unit acquires object information based on a first camera image obtained by photographing the object.

[0214] (3) The apparatus according to (1) or (2) above, wherein the holding unit information includes holding range information that represents the range in which the holding unit can hold the object.

[0215] (4) The apparatus according to (3) above, wherein the holding part is capable of holding the object by opening and closing, the holding range information is opening and closing range information representing the opening and closing range of the holding part, and the holding part information includes the opening and closing range information and open state shape information representing the shape of the holding part in the open state.

[0216] (5) Any one of the processing devices described in (1) to (4) above, wherein the placement information includes obstacle shape information representing the shape of an obstacle that would hinder the placement of the object.

[0217] (6) The processing apparatus of (5) above, wherein the acquisition unit acquires the obstacle shape information based on a second camera image obtained by photographing the obstacle.

[0218] (7) A processing apparatus relating to any one of (1) to (6) above, wherein the determination unit acquires placement information relating to the shape of the holding unit and the object when the holding unit places the object, based on the object information and the holding unit information, and determines the placeable area based on the placement information and the placement destination information.

[0219] (8) The apparatus according to (7) above, wherein the determination unit determines a holding position in which the holding unit holds the object based on the object information and the holding unit information, and acquires the arrangement information relating to the shape of the holding unit and the object when the holding unit that holds the object at the holding position places the object, based on the object information and the holding unit information.

[0220] (9) Any one of the processing devices described in (1) to (8) above, wherein the determination unit determines the placement position in which the holding unit places the object from the placement area.

[0221] (10) The processing apparatus of (9) above, wherein the determination unit acquires constraint information representing the constraint on the object located in the placeable area by surrounding objects based on the object information and the placement location information, and determines the placement position based on the acquired constraint information.

[0222] (11) The processing apparatus according to (10) above, wherein the determination unit obtains distance information for each of the plurality of directions, based on the object information and the placement destination information, which represents the distance traveled by the object located in the placement area until it collides with the surrounding object when it moves in that direction, and obtains the constraint information based on the distance information obtained for the plurality of directions.

[0223] (12) Any one of the processing devices described in (9) to (11) above, wherein the determination unit acquires uniformity information representing the uniformity of constraints on the object located in the placeable area by surrounding objects based on the object information and the placement location information, and determines the placement position based on the acquired uniformity information.

[0224] (13) The processing apparatus according to (12) above, wherein the determination unit obtains distance information for each of the multiple directions, based on the object information and the placement destination information, which represents the distance traveled by the object located in the placement area until it collides with the surrounding object when it moves in that direction, and obtains the uniformity information based on the distance information obtained for each of the multiple directions.

[0225] In one embodiment, program (14) is a program that causes the computer device to function as one of the processing units described in (1) to (13) above.

[0226] 1 Processing unit 2b Acquisition unit 2c Determination unit 3a Program 13 Holding unit 21 Camera image 30 Object 100 Opening / closing range information 110 Open state shape information 120 Suction range information 150 Object information 160 Placement location information 200 Placement time information

Claims

1. A processing apparatus comprising: an acquisition unit that acquires object information relating to the shape of an object to be held in a holding unit, holding unit information relating to the shape of the holding unit, and placement information relating to the placement location of the object; and a determination unit that determines a placement area in which the holding unit can place the object based on the object information, holding unit information, and placement location information acquired by the acquisition unit.

2. The processing apparatus according to claim 1, wherein the acquisition unit acquires object information based on a first camera image obtained by photographing the object.

3. The apparatus according to claim 1 or claim 2, wherein the holding unit information includes holding range information representing the range in which the holding unit can hold the object.

4. The apparatus according to claim 3, wherein the holding portion is capable of holding the object by opening and closing, the holding range information is opening and closing range information representing the opening and closing range of the holding portion, and the holding portion information includes the opening and closing range information and open state shape information representing the shape of the holding portion in the open state.

5. A processing apparatus according to any one of claims 1 to 4, wherein the placement destination information includes obstacle shape information representing the shape of an obstacle that hinders the placement of the object.

6. The processing apparatus according to claim 5, wherein the acquisition unit acquires obstacle shape information based on a second camera image obtained by photographing the obstacle.

7. A processing apparatus according to any one of claims 1 to 6, wherein the determination unit acquires placement information relating to the shape of the holding unit and the object when the holding unit places the object, based on the object information and the holding unit information, and determines the placeable area based on the placement information and the placement destination information.

8. A processing apparatus according to claim 7, wherein the determination unit determines a holding position in which the holding unit holds the object based on the object information and the holding unit information, and acquires arrangement information relating to the shape of the holding unit and the object when the holding unit, which holds the object at the holding position, arranges the object, based on the object information and the holding unit information.

9. A processing apparatus according to any one of claims 1 to 8, wherein the determination unit determines an arrangement position in which the holding unit places the object from the arrangementable area.

10. A processing apparatus according to claim 9, wherein the determination unit acquires constraint information representing the constraint on the object located in the locatable area by surrounding objects, based on the object information and the location information, and determines the placement position based on the acquired constraint information.

11. The processing apparatus according to claim 10, wherein the determination unit acquires distance information for each of a plurality of directions, based on the object information and the placement destination information, which represents the distance traveled by the object located in the placement area until it collides with the surrounding object when it moves in that direction, and acquires the constraint information based on the distance information acquired for the plurality of directions.

12. A processing apparatus according to any one of claims 9 to 11, wherein the determination unit acquires uniformity information representing the uniformity of constraints on the object located in the locatable area by surrounding objects, based on the object information and the location information, and determines the placement position based on the acquired uniformity information.

13. The processing apparatus according to claim 12, wherein the determination unit acquires distance information for each of a plurality of directions, based on the object information and the placement destination information, which represents the distance traveled by the object located in the placement area until it collides with the surrounding object when it moves in that direction, and acquires uniformity information based on the distance information acquired for the plurality of directions.

14. A program for causing a computer device to function as a processing device according to any one of claims 1 to 13.

Images