Calibration of a robot-guided camera

Abstract

The invention relates to a method for calibrating a camera (21) guided by a robot (10) with the aid of a surroundings-side camera (22) and a robot-side reference (12). According to the method, in each of a number of different positions of the robot, an image of the reference (12) is determined with the aid of the surroundings-side camera (22), and an image of the surroundings-side camera (22) is determined with the aid of the robot-guided camera (21). A transformation between a robot-camera-fixed coordinate system (M) of the robot-guided camera and a robot-end-effector coordinate system (E) is determined on the basis of the determined images of the reference and of the surroundings-side camera. The invention also relates to a method for controlling the robot (10), to a system for calibration and / or control purposes, and to a computer program (product).

Description

[0001] 2024P00043 WO 1 / 20 Kuka Deutschland GmbH

[0002] Description

[0003] Calibration of a robot-guided camera

[0004] The present invention relates to a method and system for calibrating a camera guided by a robot, a method and system for controlling a robot using the calibrated camera, and a computer program or computer program product for carrying out a method described herein.

[0005] The object of the present invention is to improve the calibration of robot-guided cameras and / or the control of robots.

[0006] This task is accomplished by a method having the features of claim 1 or 2.

[0007] Claim 5, or a system with the features of claim 8 or 9, respectively, is solved. Claim 10 protects a computer program or computer program product for carrying out a method described herein. The dependent claims relate to advantageous embodiments.

[0008] According to one aspect of the present invention, a robot-guided camera (“robot-guided camera”) is calibrated using an environment-side camera and a robot-side reference.

[0009] In one embodiment, the robot has a stationary base and / or a flange, preferably distal; in another embodiment, it has an end or...

[0010] The invention comprises a tool flange, and / or at least three, preferably at least six, joints or (motion) axes adjustable by drives, preferably electric drives, of the robot, by which the flange is connected to the base in one embodiment, preferably a robot arm with the base, the flange and / or the at least three, preferably at least six, joints or (motion) axes. The present invention is particularly advantageous for such robots due to its kinematics. 2024P00043 WO 2 / 20 Kuka Deutschland GmbH

[0011] The robot-side reference can be, in particular, fixed to the robot, preferably to a distal flange of the robot, and in particular may include a distal flange of the robot and / or a tool arranged thereon, in one embodiment a (robot) gripper.

[0012] In one embodiment, the robot-guided camera is fixed to the robot, preferably to a distal flange of the robot and / or to a tool arranged on a distal flange of the robot.

[0013] The ambient camera is in one version fixed to the robot's environment, preferably to the robot's stationary base.

[0014] According to one embodiment of the present invention, in two or more different positions of the robot, each

[0015] - using the ambient camera, an image of the (robot-side) reference is determined; and

[0016] - using the robot-guided camera, an image of the surrounding camera was determined.

[0017] In one embodiment, the robot-side reference exhibits different poses in the various positions of the robot. A pose within the meaning of the present invention preferably comprises a one-, two-, or three-dimensional position and / or a one-, two-, or three-dimensional orientation, and is therefore preferably one-, two-, three-, four-, five-, or, particularly preferably, six-dimensional (“6-D”). A robot position within the meaning of the present invention can be characterized or defined in particular by the positions or coordinates of the joints or axes or drives of the robot.

[0018] According to one embodiment of the present invention, based on

[0019] - the images of the reference determined (in or for the various robot positions) and 2024P00043 WO 3 / 20 Kuka Deutschland GmbH

[0020] - the images of the surrounding camera determined (in or for the various robot positions)

[0021] a transformation (hereinafter, as with the other mathematical variables used here without loss of generality: 7EM) between - a robot end effector coordinate system (E), preferably three-dimensional and / or robot-fixed, preferably robot flange-fixed and / or reference-fixed or fixed with respect to a or the distal robot flange, in one embodiment end or tool flange, or the robot-side reference, particularly preferably a TCP coordinate system, and

[0022] - a robot camera-mounted, preferably three-dimensional, coordinate system (M) of the robot-guided camera

[0023] determined. This transformation can in particular be a transformation from the robot end effector coordinate system (E) to the robot camera fixed coordinate system (M) of the robot-guided camera or from the robot camera fixed coordinate system (M) of the robot-guided camera to the robot end effector coordinate system (E).

[0024] A transformation can, in particular, comprise a homogeneous and / or four-dimensional matrix and / or describe or specify a pose or a one-, two-, or three-dimensional position and / or a one-, two-, or three-dimensional orientation of one coordinate system relative to another coordinate system, especially a translational and / or rotational transformation from one to the other coordinate system. It can, for example, be a Denavit-Hartenberg matrix or a matrix according to one of the known modified Denavit-Hartenberg conventions, or it can comprise a 3x3 matrix R specifying a three-dimensional orientation, a 3x1 vector t specifying a three-dimensional position, as well as a 1x3 zero vector and a scalar 1.

[0025] By determining the transformation, the robot-guided camera is calibrated in one configuration. Accordingly, calibrating the robot-guided camera can include determining the transformation, in particular its 2024P00043 WO 4 / 20 Kuka Deutschland GmbH

[0026] Through this calibration based

[0027] - the images of the reference obtained using the ambient camera and

[0028] - the images obtained with the help of the robot-guided camera from the surrounding camera

[0029] is advantageous in one version

[0030] - no knowledge of the robot positions themselves and / or

[0031] - no correlation between camera images and known robot positions and / or

[0032] - No communication between robot (controller) and camera (application) is required. This reduces the effort and / or improves precision compared to known calibration methods.

[0033] In one embodiment, based on the determined images of the reference, at least one pose change A of the reference in an environment-fixed coordinate system (S) of the environment-side camera is determined as a result of the robot's change of position. In a further development, this pose change can comprise or be described by a transformation between a reference-fixed coordinate system (G) in one of the robot positions and this reference-fixed coordinate system (G) in another of the robot positions, in particular a combination or linking of the transformations 7G, ​​i between the reference-fixed coordinate system (G) and the environment-fixed coordinate system (S) in the respective robot position i, in particular according to:

[0034] A = (TG,2)- 1 • 7G, 1 (1)

[0035] The reference-fixed coordinate system (G) can be identical to the robot end-effector coordinate system (E) ((G) = (E)). Likewise, a transformation, preferably fixed or constant, between the reference-fixed coordinate system (G) and the robot end-effector coordinate system (E) can be considered or carried out, in an implementation based on a predefined numerical model of the reference or a known 2024P00043 WO 5 / 20 Kuka Deutschland GmbH

[0036] Transformation between the robot flange or robot end effector coordinate system (E) and the reference-fixed coordinate system (G). Accordingly, or more generally, the robot end effector coordinate system (E) and the reference-fixed coordinate system (G) can be interchangeable, possibly taking into account the corresponding transformation between the two coordinate systems, such that, for the sake of a more compact representation, only one of the two coordinate systems (E) or (G) is ever mentioned or specified.

[0037] In one embodiment, based on the images of the surrounding camera determined by the robot-guided camera, at least one pose change B of the surrounding camera in the robot-camera-fixed coordinate system of the robot-guided camera is determined as a result of the robot's change in position. In a further development, this pose change can be a transformation between one or the surrounding camera's fixed coordinate system (S) in one of the robot positions and this surrounding camera's fixed coordinate system (S) in another of the robot positions, in particular a combination or

[0038] Linking the transformations Ts,i between the environment-fixed coordinate system (S) and the robot-camera-fixed coordinate system (M) of the robot-guided camera in the respective robot position i, include or be described by this, in particular according to:

[0039] B = Ts, 2 - (Ts,i)-1 (2)

[0040] In one embodiment, the transformation TEM between the robot end effector coordinate system (E) and the robot camera-fixed coordinate system (M) of the robot-guided camera is determined based on these determined pose changes of the reference and the surrounding camera, in a further development according to:

[0041] A ■ TGM = TGM • B (3)2024P00043 WO 6 / 20 Kuka Deutschland GmbH

[0042] As already mentioned, the robot end effector coordinate system (E) and the reference-fixed coordinate system (G) can also be interchanged here, possibly taking into account the corresponding transformation between the two coordinate systems.

[0043] In one embodiment, determining the change in the pose of the reference in the environment-fixed coordinate system (S) of the ambient camera is based on a predefined numerical model of the reference and / or comprises determining poses of the reference in the environment-fixed coordinate system (S) of the ambient camera in the various robot positions i, preferably based on a predefined numerical model of the reference. These poses of the reference in the environment-fixed coordinate system (S) of the ambient camera in the various robot positions i can, in particular, be described by the corresponding transformations TG.

[0044] Additionally or alternatively, in one embodiment, determining the pose change of the surrounding camera in the robot-camera-fixed coordinate system (M) of the robot-guided camera is based on a predefined numerical model of the surrounding camera and / or comprises determining poses of the surrounding camera in the robot-camera-fixed coordinate system of the robot-guided camera in the various robot positions, preferably based on a predefined numerical model of the surrounding camera. These poses of the surrounding camera in the robot-camera-fixed coordinate system of the robot-guided camera in the various robot positions i can be described, in particular, by the corresponding transformations 7s, i.

[0045] This allows for improved calibration of robot-guided cameras and thus the control of robots using robot-guided cameras, in particular reducing effort and / or improving precision. 2024P00043 WO 7 / 20 Kuka Deutschland GmbH

[0046] Advantageously, a predefined numerical model of the surrounding camera and / or the reference is already available, for example for collision monitoring and / or avoidance or the like, and can thus be used.

[0047] In one iteration, the procedure comprises the following steps:

[0048] The robot is adjusted to the different (robot) positions, either by executing a predefined robot program or based on control commands entered online by an operator.

[0049] In each position, an image of the robot-side reference is acquired using the surrounding camera. Based on this, and preferably using a pose determination or estimation algorithm and / or based on the predefined numerical model of the reference, the pose (without loss of generality TG ) of the reference is determined in the surrounding-fixed coordinate system (S) of the surrounding camera for the respective robot position i.

[0050] Additionally, in the respective position, an image of the surrounding camera is determined using the robot-guided camera. Based on this, preferably using a pose determination or estimation algorithm, particularly preferably the same algorithm that is also used to determine the poses TG of the reference in the environment-fixed coordinate system of the surrounding camera, and / or based on the predefined numerical model of the surrounding camera, the pose Tsj of the surrounding camera is determined in the robot-camera-fixed coordinate system of the robot-guided camera for the respective robot position i.

[0051] Out of

[0052] TG,I • TGM • Ts,i = TG,2 • TGM • Ts, 2 (4)2024P00043 WO 8 / 20 Kuka Deutschland GmbH

[0053] surrendered

[0054] (7G, 2)' 1 • TG,I • 7GM = 7GM • Ts, 2 • (Ts,i)' 1 (5)

[0055] and with Eq. (1) and (2) then Eq. (3), where TG is the (determined) pose of the reference in the environment-fixed coordinate system (S) of the environment-side camera for the respective robot position i, Ts,i and the (determined) pose of the environment-side camera in the robot-camera-fixed coordinate system (M) of the robot-guided camera for the respective robot position i, Ts,i.in the respective robot position i, A the (relative) transformation of the reference from position i=2 to position i=1 in the environment-fixed coordinate system (S) of the environment-side camera, B the (relative) transformation of the environment-side camera from position i=1 to position i=2 in the robot-camera-fixed coordinate system (M) of the robot-guided camera, and TGM the transformation between the reference-fixed coordinate system (G) and the robot-camera-fixed coordinate system (M) of the robot-guided camera, in particular from the reference-fixed coordinate system (G) to the robot-camera-fixed coordinate system (M) or from the robot-camera-fixed coordinate system (M) to the reference-fixed coordinate system (G), and, as already mentioned several times, here too the robot end effector coordinate system (E) and the reference-fixed coordinate system (G) can be interchanged.can be determined, if necessary taking into account the corresponding transformation between these two coordinate systems. This transformation between the robot end effector coordinate system (E) and the reference-fixed coordinate system (G) can, for example, be determined from a predefined numerical model, such as a CAD model or the like, the reference, and / or the robot flange. The pose of the reference-fixed coordinate system (G) in the figures and / or in the environment-fixed coordinate system and / or relative to the reference can be determined by a known algorithm. Similarly, the pose of the environment-fixed coordinate system (S) of the surrounding camera in the figures and / or in the robot camera-fixed coordinate system can also be determined. 2024P00043 WO 9 / 20 Kuka Deutschland GmbH.

[0056] The coordinate system and / or relative to the surrounding camera can be determined by an algorithm that is known per se.

[0057] In qualitative or intuitive terms, the change in pose of the surrounding camera from the perspective of the robot-guided camera during the change in position corresponds to the inverse change in pose of the reference from the perspective of the surrounding camera during the change in position, taking into account the transformation 7GM between the robot-camera-fixed coordinate system (M) and the reference-fixed coordinate system (G), which can be identical to the sought-after transformation between the robot-camera-fixed coordinate system and the robot end-effector coordinate system or can be converted or transformed into it by a corresponding known transformation.

[0058] Using a robot-guided camera calibrated in this way, the robot can then be controlled particularly advantageously, as this reduces the effort required for calibration and / or improves precision in calibration and / or control.

[0059] Accordingly, in a further development, according to one embodiment of the present invention, the robot-guided camera is calibrated according to a method described herein for controlling a robot, and one or, preferably successively, several objects are detected using the calibrated camera, and the robot is controlled based on this detection. In a further development, the robot is controlled to perform a movement predetermined relative to the (respective detected) object, in a further development, to approach the (respective detected) object, and in a further development, to pick up the object. In one embodiment, a pose of the detected object is transformed from the robot-camera-fixed coordinate system (M) of the robot-guided camera into the robot end-effector coordinate system (E), and the robot is controlled to approach this pose, and in a further development, to pick up the object. 2024P00043 WO 10 / 20 Kuka Deutschland GmbH

[0060] The invention can be used particularly advantageously when picking up objects and / or when objects are arranged on or in a feeding device, or in a further development, when objects are present in a feeding container.

[0061] According to one embodiment of the present invention, a system, in particular in terms of hardware and / or software, in particular in terms of programming, is set up and / or has the following features for carrying out a method described herein:

[0062] - Means of determining an image of the robot-side reference using the environment-side camera and an image of the environment-side camera using the robot-guided camera in the various positions of the robot; and

[0063] - Means of determining a transformation between a robot camera-fixed coordinate system of the robot-guided camera and a robot end effector coordinate system based on the determined images of the reference and surrounding camera.

[0064] According to one embodiment of the present invention, the system has:

[0065] - Means of capturing at least one object using the calibrated camera;

[0066] and

[0067] - Means of controlling the robot based on this data collection.

[0068] In one version, the system or its means exhibit:

[0069] - Means for determining at least one pose change of the reference in an environment-fixed coordinate system of the environment-side camera as a result of the robot's change of position, based on the determined images of the reference, wherein determining the pose change of the reference in the environment-fixed coordinate system of the environment-side camera preferably comprises determining poses of the reference in the environment-fixed coordinate system of the environment-side camera in the various robot positions and / or is based on a predetermined numerical model of the reference; and / or

[0070] - Means of determining at least one pose change of the surrounding camera in the robot camera-fixed coordinate system of the robot-guided 2024P00043 WO 11 / 20 Kuka Deutschland GmbH

[0071] Camera as a result of the robot's change in position based on the images of the surrounding camera determined using the robot-guided camera, wherein determining the change in pose of the surrounding camera in the robot-camera-fixed coordinate system of the robot-guided camera preferably comprises determining poses of the surrounding camera in the robot-camera-fixed coordinate system of the robot-guided camera in the various robot positions and / or is based on a predetermined numerical model of the surrounding camera; and / or

[0072] - Means of determining the transformation between the robot-camera-fixed coordinate system of the robot-guided camera and the robot end effector coordinate system based on the determined pose changes of the reference and surrounding camera; and / or

[0073] - Means for controlling the robot based on the detection(s) of the object(s) using the calibrated camera, preferably for performing a movement specified relative to the (respective) object, in a version for approaching the (respective) object.

[0074] A system and / or means according to the present invention can be configured as hardware and / or software, in particular comprising at least one processing unit, preferably a microprocessor unit (CPU), graphics processing unit (GPU), or the like, preferably connected to a storage and / or bus system via data or signals, and / or comprising one or more programs or program modules. The processing unit can be configured to execute instructions implemented as a program stored in a storage system, to acquire input signals from a data bus, and / or to output signals to a data bus. A storage system can comprise one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be configured to embody the methods described herein.is able to execute, so that the processing unit can carry out the steps of such procedures and thus in particular the 2024P00043 WO 12 / 20 Kuka Deutschland GmbH.

[0075] The system can determine the transformation between the robot end effector coordinate system and the robot camera-fixed coordinate system of the robot-guided camera, or calibrate the robot-guided camera, or control the robot. A computer program product may, in one embodiment, include a storage medium, in particular a computer-readable and / or non-volatile medium, for storing a program or instructions, or with a program or instructions stored thereon. In one embodiment, the execution of this program or these instructions by a system or a controller, in particular a computer or an arrangement of several computers, causes the system or the controller, in particular the computer(s), to execute a procedure described herein or one or more of its steps, or the program or instructions are configured for this purpose.

[0076] In one embodiment, one or more, in particular all, steps of the procedure are fully or partially computer-implemented, or one or more, in particular all, steps of the procedure are fully or partially automated, in particular by the system or its means.

[0077] In one embodiment, the system includes the robot and / or the robot-guided camera and / or the environment-side camera and / or the robot-side reference.

[0078] Further advantages, features, and advantageous embodiments of the present invention will become apparent from the dependent claims and the exemplary embodiments. These are shown, in part schematically:

[0079] Fig. 1: a system for calibrating a robot-guided camera for controlling the robot using the camera according to an embodiment of the present invention; and

[0080] Fig. 2: A method for calibrating the camera for controlling the robot using the camera according to an embodiment of the present invention. 2024P00043 WO 13 / 20 Kuka Deutschland GmbH

[0081] Fig. 1 shows a system for calibrating a robot-guided camera 21 using an ambient camera 22 and a robot-side reference in the form of a (robot) gripper 12, which is arranged on a distal flange 11 of the robot, or for controlling the robot 10 using the robot-guided camera 21 according to one embodiment of the present invention. Fig. 2 shows a method for calibrating the robot-guided camera 21 for controlling the robot 10 using this camera 21 according to one embodiment of the present invention, which is carried out using a computer 1.

[0082] In step S10, the robot 10 is moved to the position shown in Fig. 1. In this position, in step S20, an image of the gripper 12 is acquired using the ambient camera 22, and an image of the ambient camera 21 is acquired using the robot-guided camera 21. In step S30, the robot 10 is moved to a position different from that shown in Fig. 1, in which its gripper 12 has a different position and orientation. In this position, in step S40, an image of the gripper 12 is again acquired using the ambient camera 22, and an image of the ambient camera 21 is acquired using the robot-guided camera 21.

[0083] In step S50, using a pose determination algorithm based on a CAD model of the gripper, 6D poses of the gripper 12 are determined in an environment-fixed coordinate system (S) of the environment-side camera 22, for example, described by corresponding transformation matrices 7G, ​​i, where i = 1, 2 denotes the respective robot position. Additionally, in step S50, using the same pose determination algorithm based on a CAD model of the environment-side camera 22, 6D poses of the environment-side camera 22 are determined in a robot-camera-fixed coordinate system (M) of the robot-guided camera 21, for example, described by corresponding transformation matrices 7s, i. 2024P00043 WO 14 / 20 Kuka Deutschland GmbH

[0084] In step S60, based on this, the transformation TEM between the robot camera fixed coordinate system (M) of the robot-guided camera and a robot end effector coordinate system (E) is determined according to Eq. (1) - (5).

[0085] The robot end effector coordinate system (E) can be identical to a reference-fixed coordinate system (G) and / or a TCP coordinate system. If the robot end effector coordinate system is not identical to the reference-fixed coordinate system (G), a known transformation between the robot end effector coordinate system or robot flange 11 and the reference-fixed coordinate system (G), or one determined based on CAD data, can be considered. For example, the TGM transformation can be determined first, and the TEM transformation derived from this, which can then be used to control the robot. Similarly, this transformation between the robot end effector coordinate system or robot flange 11 and the reference-fixed coordinate system (G) can already be considered in equations (1) - (5).

[0086] In step S70, the robot 10 is controlled to successively approach and pick up objects 2, 3, 4 located in a container 5. For each of these steps, a 6D pose of the corresponding object 2, 3, or 4 is transformed from the robot camera's fixed coordinate system (M) to the robot end effector coordinate system (E) using the determined transformation TEM of the (thereby) calibrated robot-guided camera 21, and the robot 10 is controlled to (approach this pose to) pick up this object.

[0087] In the present disclosure, “has an X” does not generally imply an exhaustive list, but is a shorthand for “has at least one X” and also includes “has two or more X” as well as “has Y in addition to X”. 2024P00043 WO 15 / 20 Kuka Deutschland GmbH

[0088] It can therefore be particularly advantageous to use more than the two positions mentioned and to determine the transformation by means of averaging, adjustment calculation, or the like.

[0089] Although exemplary embodiments were explained in the preceding description, it should be noted that a multitude of modifications are possible. Furthermore, it should be noted that the exemplary embodiments are merely examples and are not intended to restrict the scope of protection, applications, or structure in any way. Rather, the preceding description provides the skilled person with a guideline for implementing at least one exemplary embodiment, whereby various modifications, particularly with regard to the function and arrangement of the described components, can be made without departing from the scope of protection as defined by the claims and these equivalent combinations of features. 2024P00043 WO 16 / 20 Kuka Deutschland GmbH

[0090] List of reference signs

[0091] I Computer

[0092] 2, 3, 4 object

[0093] 5 containers

[0094] 10 robots

[0095] II distal (robot) flange

[0096] 12 grippers

[0097] 21 Robot-guided camera, stationary to distal flange and gripper 22 Surrounding camera

[0098] (E) Robot end effector coordinate system

[0099] (G) reference-fixed coordinate system

[0100] (M) robot camera fixed coordinate system

[0101] the robot-guided camera 21

[0102] (S) environment-fixed coordinate system

[0103] the surrounding camera 22

Claims

2024P00043 WO 17 / 20 Kuka Deutschland GmbH Patent claims 1. Method for calibrating a camera (21) guided by a robot (10) using an environment-side camera (22) and a robot-side reference (12), wherein: - in different positions of the robot - using the ambient camera (22) to take an image of the reference (12) and - using the robot-guided camera (21 ) to create an image of the surrounding camera (22) is determined; and - a transformation is determined between a robot camera-fixed coordinate system (M) of the robot-guided camera and a robot end effector coordinate system (E) based on the determined images of the reference and surrounding camera.

2. Method according to claim 1, characterized in that the reference has a distal flange (11) of the robot and / or a tool (12) arranged thereon and / or the robot-guided camera (21) is stationary thereon.

3. Method according to one of the preceding claims, characterized in that - based on the determined images of the reference, at least one change in the pose of the reference in an environment-fixed coordinate system (S) of the environment-side camera is determined as a result of the robot's change in position; - based on the images of the surrounding camera obtained with the aid of the robot-guided camera, at least one change in the pose of the surrounding camera in the robot-camera-fixed coordinate system is determined as a result of the robot's change in position; and 2024P00043 WO 18 / 20 Kuka Deutschland GmbH - the transformation between the robot camera-fixed coordinate system and the robot end effector coordinate system is determined based on the determined pose changes of the reference and surrounding camera.

4. Method according to one of the preceding claims, characterized in that - determining the pose change of the reference in the environment-fixed coordinate system includes determining poses of the reference (12) in the environment-fixed coordinate system (S) in the different robot positions and / or is based on a predefined numerical model of the reference; and / or - determining the pose change of the surrounding camera in the robot camera fixed coordinate system includes determining poses of the surrounding camera (22) in the robot camera fixed coordinate system (M) in the different robot positions and / or is based on a predefined numerical model of the surrounding camera.

5. Method for controlling a robot (10) using a camera (21) guided by the robot, wherein: - the robot-guided camera is calibrated according to a method according to one of the preceding claims; - at least one object (2, 3, 4) is captured using the calibrated camera; and - the robot is controlled based on this data collection.

6. Method according to claim 5, characterized in that - the robot is controlled based on the detection to perform a movement relative to the object as specified; and / or - the object is an object to be picked up by the robot and / or arranged on or in a feeding device (5). 2024P00043 WO 19 / 20 Kuka Deutschland GmbH 7. Method according to claim 6, characterized in that the robot is controlled to approach the object based on the detection.

8. System for calibrating a camera (21) guided by a robot (10), which is set up and / or comprises a method according to one of the preceding claims: - Means for determining an image of a robot-side reference (12) using an environment-side camera (22) and an image of the environment-side camera using the robot-guided camera in different positions of the robot; and - Means of determining a transformation between a robot camera fixed coordinate system (M) of the robot-guided camera and a robot end effector coordinate system (E) based on the determined images of the reference and surrounding camera.

9. System for controlling a robot (10) using a camera (21) guided by the robot, which is configured and / or comprises a method according to one of the preceding claims 5-7: - Means for detecting at least one object (2, 3, 4) using the calibrated camera (21); and - Means of controlling the robot based on this data collection.

10. Computer program or computer program product, wherein the computer program or computer program product, in particular stored on a computer-readable and / or non-volatile storage medium, contains instructions which, when executed by one or more computers or a system according to claim 8 or 9, cause the computer(s) or system to perform a method according to any one of claims 1 to 7.

Images