Method and device for demonstration-based programming of a robot operable in multiple control modes

Abstract

A method of programming an industrial robot (100), which comprises a robot manipulator (110) and a robot controller (120), comprising: recording movements of a robot manipulator during a demonstration-based programming session, for thereby obtaining a robot trajectory; capturing speech data while recording the movements of the robot manipulator; decomposing the captured speech data into a plurality of phases, wherein the robot trajectory has corresponding phases; providing an annotated robot trajectory by performing the following for at least one phase of the speech data: (i) using a natural-language model, parsing the speech data into at least one robot parameter value, and (ii) annotating a corresponding phase of the robot trajectory with the at least one robot parameter value; and generating a robot program on the basis of the annotated robot trajectory. The robot parameter value indicates a motion template to be used for realizing the robot trajectory, selected from a predefined set including a motion template in force-control mode and a motion template in position-control mode.

Description

METHOD AND DEVICE FOR DEMONSTRATION-BASED PROGRAMMING OF A ROBOT OPERABLE IN MULTIPLE CONTROL MODESTECHNICAL FIELD

[0001] The present disclosure generally relates to the field of robotic control, and specifically to demonstration-based programming of an industrial robot. More precisely, methods and devices are proposed herein which support decision-making on whether to operate the industrial robot in force-control mode or position-control mode. The decision may be expressed in terms of a motion template in which the robot is operated in either of these control modes.BACKGROUND

[0002] Two important branches of the field of robot control are human-to-robot instruction and robot teaching. Human-to-robot instruction addresses the problem of automatically generating robot motion from a non-technical, high-level description of a task such as “Assemble the car” or “Make me coffee”. In robot teaching, a robot program realizing the robot motion is automatically generated from a demonstration by an operator, who shows the robot how the task is to be carried out.Demonstration-based robot programming may include kinesthetic programming and so-called passive observation. In kinesthetic robot programming the robot handles the workpiece during the programming session, while in passive observation (robotless demonstration) a human operator who carries out the robot task may handle the workpiece in the robot’s stead, and the robot learns to imitate whatever handling the workpiece is exposed to.

[0003] Once an area exclusive to seasoned programmers of high technical expertise, the user base of demonstration-based robot programming has broadened lately thanks to the advent of Large Language Models (LLMs), and Generative Pretrained Transformers (GPTs) in particular. LLMs are increasingly being applied to robot control and robot programming. See for instance the applicant’s prior disclosure PCT / EP2023 / 087949, which proposed a speech-supplemented robot programming method based on the following workflow:A demonstration-based (in particular, a kinesthetic) programming session, during which movements of the robot manipulator are captured byproprioceptive sensors. The movements are recorded and saved as a robot trajectory.• A recording of speech data, which takes place during the demonstrationbased programming session.• By suitable prompting, an LLM is caused to parse the speech data into values of robot parameters, such as a tool state, a movement speed, a degree of compliance with the trajectory, a degree of movement precision, a choice of reference frame to express the movements in.• The robot trajectory is annotated with each robot parameter value at a point of the trajectory which corresponds to the time of utterance.• A robot program is generated as a sequence of commands which realizes the robot trajectory while applying the robot parameter values as modifiers.The programming method according to PCT / EP2023 / 087949 offers the operator a convenient, hands-free way of adding explanatory remarks to the recorded movements (purpose of a step, aspects to pay particular attention to), or of requesting certain robot parameter settings which cannot be seen or sensed during the demonstration (speed, precision) etc., with the ultimate aim of providing a usable robot program in shorter time. The programming workflow as such appears to be proper to the applicant.

[0004] Configuring contact-force regulation is a known difficulty in the programming of robot applications that include interaction with workpieces and other objects in the working environment. In particular, it may have to be configured whether the industrial robot is to be operated in position-control mode, which generally speaking has better precision, or force-control mode, which may be preferable for applications that involve contact forces between the robot and the environment, and delicate workpieces in particular.

[0005] At the core of this difficulty is the fact that, since the static forces sum to zero at each external interface of the robot, a proprioceptive sensor cannot capture whatever attempts the operator makes to demonstrate a desired magnitude of a contact force. It will not be felt by the gripper actuator that the operator pinches the gripper harder around a workpiece (e.g., to indicate an increased gripping force).Likewise, the operator’s intuitive attempt to ask for an increased downward force by pressing the end-effector against a table will not give rise to an increase in torque at the inner joints of the robot manipulator. In the absence of a dedicated external force sensor (contact force sensor) arranged in or at the external interface concerned, the contact force can only be determined in exceptional situations.

[0006] It would be desirable to automate or at least support the decision-making on whether to operate the industrial robot in force-control mode or position-control mode.SUMMARY

[0007] One objective of the present disclosure is to make available methods and devices which an operator can utilize to generate a robot program from a recorded robot trajectory and concurrently captured speech data, and which assist the operator with the decision-making on force-control mode vs position-control mode. A particular objective is to make available such methods and devices which provide this assistance by selecting a motion template in which the robot is operated in forcecontrol mode or a motion template in which the robot is operated in position-control mode. A further objective is to enable automated robot programming without having to adduce world knowledge about the robot task to be carried out. A further objective is to shorten the learning time that a newcomer needs in order to start using a robot programming device productively.

[0008] At least some of these objectives are achieved the invention as defined by the appended independent claims. The dependent claims relate to advantageous embodiments of the invention.

[0009] In a first aspect of the present disclosure, a method of programming an industrial robot, which comprises a robot manipulator and a robot controller, is provided. The method comprises: recording movements of a robot manipulator during a demonstration-based programming session, for thereby obtaining a robot trajectory; capturing speech data while recording the movements of the robot manipulator; decomposing the captured speech data into a plurality of phases (e.g., time segments, in particular contiguous time segments, wherein the robot trajectory has corresponding phases; providing an annotated robot trajectory; and generating a robot program on the basis of the annotated robot trajectory. The annotated robottrajectory is provided by performing the following operations for at least one phase of the speech data: parsing the speech data into at least one robot parameter value using a natural-language model, and then annotating the phase of the robot trajectory with the at least one robot parameter value. According to the first aspect, the robot parameter value indicates a motion template to be used for realizing the robot trajectory, wherein the motion template is selected from a predefined set including at least one motion template in which the robot is operated in force-control mode and at least one motion template in which the robot is operated in position-control mode.

[0010] The inventors have discovered a way of applying a natural-language model to the task of deciding whether to operate the robot in a force-control mode or a position-control mode when realizing the trajectory, wherein the model takes this decision on the basis of a recorded robot trajectory and concurrently captured speech data. The outcome of this decision is incorporated into the robot program, wherein the robot program may have the form of a sequence of robot commands (e.g., commands compliant with the RAPID™ language) which, when executed by the robot controller, cause the robot manipulator to realize the trajectory while applying the robot parameter values as modifiers to these robot commands. The inventors have empirically verified that the novel method performs well and provides useful results, also without supplying any world knowledge (a priori information) in addition to the speech data. In other words, the inventors have successfully specialized the framework known from PCT / EP2023 / 087949 into technology for automating or partially automating the decision-making on whether to operate the robot in a force-control mode or a position-control mode.

[0011] In some embodiments, the motion template includes at least one robot property which can be configured, such as movement speed, degree of compliance with the robot trajectory, degree of movement precision, gripping force of a robot tool, status of a robot tool, handling force, geometric movement constraints, force control vs position control. The concrete value of the property is indicated by a template argument. In these embodiments, the one or more robot parameter values parsed from the speech data include the template argument. This ability to adapt the motion templates can be applied more with greater flexibility and in a broader range of situations.

[0012] In some embodiments, the annotated robot trajectory is provided further on the basis of a natural-language description of a video of the programming session. In particular, the description may be parsed - in addition to the speech data - into said at least one robot parameter value. The description of the video may be generated using a vision-enabled language model (VLM) which is applied to the video. Accordingly, knowledge incorporated in the VLM will be transferred into the robot parameter values, which may be expected to make them more meaningful and / or pertinent to the robot program. The inventors have obtained satisfactory performance with a generic VLM, not specifically trained for robot-related processing.

[0013] In some embodiments, predefined heuristics are used to derive a new robot parameter value - or a modification of an already determined robot parameter value - from the robot trajectory. The predefined heuristics are adapted for locating one or more missing template arguments in a motion template and to predict contextually suitable values of the template arguments. The heuristics may be template-specific in the sense that each heuristic is specific to one or more motion templates in the predefined set of motion templates discussed above, and that the heuristic will be used in connection with those one or more motion templates only. Example heuristics will be disclosed below. Advantageously, if the natural-language model (and optionally the VLM) fail to find values of all template arguments, the predefined heuristics may fill these gaps in a contextually adequate way, so that a functioning robot program can be generated.

[0014] In a second aspect of the present disclosure, there is provided a programming device for facilitating programming of an industrial robot of the type discussed above. The programming device comprises memory and processing circuitry configured to carry out the method according to the first aspect.

[0015] The second aspect generally shares the effects and advantages of the first aspect, as reviewed above, and it can be put to practice with a corresponding degree of technical variation.

[0016] The present disclosure further relates to a computer program containing instructions for causing a computer, or the programming device in particular, to carry out the method according to the first aspect. The computer program may be stored or distributed on a data carrier. As used herein, a “data carrier” may be a transitory data carrier, such as modulated electromagnetic or optical waves, or a non-transitory datacarrier. Non- transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of magnetic, optical or solid-state type. Still within the scope of “data carrier”, such memories may be fixedly mounted or portable.

[0017] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to “a / an / the element, apparatus, component, means, step, etc.” are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, on which: figure 1 shows a work area, in which a robot manipulator operates under the control of a robot controller, and a programming device supporting demonstration-based robot programming; figure 2 is a flowchart of a robot programming method according to embodiments herein; figure 3 illustrates information flows during an execution of the method illustrated in figure 2, wherein the panels in the upper portion of figure 3 contain representative examples (or excerpts thereof) of the different types of information; and figures 4 and 5, respectively, are block diagrams of an example force-control arrangement and an example position-control arrangement for an industrial robot.DETAILED DESCRIPTION

[0019] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, on which certain embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thoroughand complete, and to fully convey the scope of all aspects of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.System overview

[0020] As explained initially, the present disclosure relates to the field of robot industrial control and specifically to demonstration-based programming of an industrial robot. In the terminology of the present disclosure, “industrial robot” is used in a broad sense, to cover in particular manufacturing robots, material-handling robots, assembly robots, cutting / welding robots, service robots, collaborative robots, hygiene robots, industrial robot tracks, industrial robot positioners. An industrial robot may be a stationary robot or a mobile robot, such as an automated guided vehicle (AGV), an autonomous mobile robot (AMR) or an autonomous mobile manipulator robot (AMMR). The term industrial robot covers the full range from lightweight robots designed to replace human manual work, over collaborative robots for supporting a human worker, all the way up to heavy-duty robots.

[0021] By way of illustration and not limitation, figure 1 shows an industrial robot100 made up of a robot manipulator no and a robot controller 120. The robot manipulator no and the robot controller 120 are joined by a wired or wireless bidirectional data connection, which conveys control signals, sensor data etc.

[0022] The robot manipulator 110 includes an arm, which extends from a base 112 and is composed of structural elements 114 and at least one linear or rotary joint 113. The arm may further include an end-effector 111 allowing it to carry tools by which it may interact with objects in the form of various workpieces 151, which are present in a work area 150 of the robot manipulator no. The workpieces are subject to manufacture, processing or handling by the robot manipulator no. The work area 150 may further include non-workpiece objects 152, such as containers, fixtures, robot positioners, separators, insulators, supports etc. These objects 152 maybe generic or they may be specifically adapted to the workpieces 151 handled by the robot manipulator 110. Apart from wear, staining etc., they are generally in the same condition at the beginning and end of a work cycle. The non-workpiece objects 152 could also be consumables or containers for consumables.

[0023] The arm of the robot manipulator no is movable by action of internal motors, drives or other sorts of actuators (not shown). The arm of the robotmanipulator no further includes transducers, encoders, sensors and other measuring equipment (not shown), from which the manipulator’s no current position, pose, technical condition, load etc. can be derived, to some degree of accuracy. The position, pose etc. of the robot manipulator no may in particular refer to a point on the arm, particularly to a tool-center point (TCP).

[0024] The actuators of the robot manipulator 110 can be controlled based on feedback from the measuring equipment, and in accordance with at least the following control concepts:- position control,- force control (or impedance control, or stiffness control).In broad terms, position-control mode signifies that the actuators are controlled in such manner that the robot manipulator no achieves a desired setpoint position xs(optionally subject to constraints on speed or acceleration), whereas force-control mode establishes a connection between the contact force and the position when the robot manipulator no makes contact with an object. The robot can conceptually ‘feel’ its surroundings, and it is able to apply a constant force on a surface, even if the exact position of the surface is not known. These characteristics of force control are likely to benefit use cases such as grinding, milling, polishing, deburring, deflashing and similar kinds of mechanical processing.

[0025] Position control and force control can be put into practice by analog or digital circuitry, or by software instructions executing in a robot controller 120, a microcontroller or another processor with suitable input and output capabilities. Although software-based implementations may be more widely practiced in industrial robot systems at the time of filing the present disclosure, the two control concepts will now be illustrated by two block diagrams where the blocks represent analog components.

[0026] Figure 4 is a block diagram of an example force-control arrangement 400 suitable for an industrial robot 100. Inputs to the force-control arrangement 400 are a setpoint force Fsand a setpoint position xs. In operation, a position error xs- x is mapped by a control gain 401 representing a mechanical impedance Z into a correction force Z x (xs— x). The force-control arrangement includes a control loop formed by a proportional regulator 410 in series with the robot manipulator no. Theproportional regulator 410 thus causes the robot manipulator no to exert a force F equal to Fs+ Z x (xs- x). It is understood that each of these setpoint quantities may be linear (force, position) or angular (torque, angle). Further, an industrial robot 100 with multiple degrees of freedom (DOFs) maybe controlled by as many copies of the force-control arrangement 400.

[0027] A force-control mode is implemented in versions of the applicant’s product RobotWare, including in version 6.14 and higher.

[0028] Figure 5 is a block diagram of an example position-control arrangement 500 for an industrial robot 100. The basic setpoint quantity is the setpoint position xsof the robot manipulator no, and the position-control arrangement 500 may further accept a setpoint speed vsand setpoint acceleration as. Further position-control arrangements, which accept a maximum speed and maximum acceleration, and which limit the movement of the robot manipulator no accordingly, are known in the literature. In the depicted position-control arrangement 500, if neither a setpoint speed nor an acceleration is provided, the robot manipulator 110 will be moved to the setpoint position xsas soon as possible in accordance with the control gain of a proportional regulator 510, which is arranged to minimize the position error xs— x at its input. The proportional regulator 510 and the robot manipulator no may be considered to form a position-control loop, which is an outer loop from a total of three cascaded control loops. On the intermediate level, a speed-control loop (or rpm-control loop) is formed by a proportional-integral-derivative controller 511 and the mass of the movable portion of the robot manipulator no for this DOF. An inner loop, consisting of a proportional-integral regulator 512 and the inductance 501 of a drive motor, is arranged to control the acceleration of the movable portion of the robot manipulator 110 for this DOF. The symbols u, i refer to actual values of tension and current of the drive motor, wherein the current may be considered to be the control signal.

[0029] Returning to figure 1, the positions, poses etc. of the robot manipulator no may be expressed with respect to one of multiple possible reference frames, including a first fixed reference frame O0defined with respect to a point in the work area 150, a second fixed reference frame (not shown) defined with respect to the base 112, a first reference frame O defined with respect to an initial position of a workpiece 151, or a second reference frame O2defined with respect to a position of a non -workpieceobject 152 in the work area 150. Here, the first fixed reference frame OQis defined with respect to the point in the work area 150 in the sense that its origin is situated in that point. Alternatively, a reference frame may be defined with respect to a reference point in the sense that its origin is situated a predefined translation from the reference point. The first and second fixed reference frames, as well as any further reference frames which are independent of the positions of objects 151, 152 in the work area 150, may be collectively referred to as neutral reference frames. All reference frames shown in figure 1 have a common orientation, i.e., their respective first, second and third axes are parallel. Without departing from the scope of the present disclosure, a further option is to use reference frames with mutually different orientations, including a reference frame which is oriented in accordance with a pose of an object 151, 152 in the work area 150 or another suitable reference objects. In particular, one may use a reference frame which has its origin situated in the TCP and is oriented parallel to the tool 111 of the manipulator no at all times.

[0030] The robot controller 120 comprises processing circuitry 121, a memory 122 and a communication interface 123. Example content of the memory 122 during operation may includeS: an operating system, basic settings, software implementing generic movements, sensing, self-monitoring, generically useful functionalities and services (all typically contributed by an original manufacturer), task- or role-specific configurations, configuration templates (typically contributed by a robot system integrator), and site-specific settings (typically contributed by an end user);C: robot programs or projects causing the robot manipulator no to perform useful or intended tasks in its work area 150.Processes that execute the robot programs may do so in accordance with the program-independent memory content S, e.g., by making calls to available functionalities, libraries, routines or parameter values therein. It is recalled that the memory 122 and processing circuitry 121 of the robot controller 120 maybe distributed and / or contain networked resources having a different physical localization than figure 1 suggests.

[0031] Each of the programs C may contain a plurality of movement instructions relating to locations such as points, poses, paths as well as modulated paths. Aprogram maybe a compiled executable (binary) or a script. A movement instruction relating to a modulated path may be expressed as - or may include - a process-on- path instruction. The programs C maybe created by an operator 190 with the aid of a robot programming device (programming station) or a general-purpose computer, or they may be created directly at the robot controller 120 if it has an operator interface. In the first two cases, versions of the programs C may be downloaded to the robot controller 120 over a wired or wireless connection or by being temporarily stored on a portable memory. A robot program maybe created by means of demonstration-based programming according to the teachings in the applicant’s prior disclosures PCT / EP2023 / 087949, PCT / EP2024 / 066288 and PCT / EP2024 / 078213.

[0032] As used herein, “demonstration-based” robot programming (or programming by demonstration [PbD], or programming from demonstration) includes a step where the operator performs a portion of a robot task or guides the robot manipulator such that it performs a portion of this robot task. The operator’s activities are imaged or, in other ways, observed by a programming device. Demonstration-based robot programming includes kinesthetic programming as a special case. The term kinesthetic programming - by allusion to the robot’s proprioceptive ability to sense its own position, pose and movements - is a programming approach in which the operator physically moves the robot manipulator to imitate the desired motions. The terms kinesthetic programming and lead-through programming are synonymous or at least partially overlapping in meaning. The state of the robot during a kinesthetic programming session is typically recorded by means of the robot’s onboard sensors, e.g., joint angles and torques. As used herein, kinesthetic programming also includes teleoperation as a special case, where the movement of the robot manipulator is controlled by an external input to the robot through a joystick, graphical user interface or other input means; kinesthetic programming by teleoperation does not require the operator to be present in the work area of the robot.

[0033] A dedicated programming device 160 is shown in the right-hand portion of figure 1. From the programming device 160, the created programs C can be transmitted to the communication interface 123 of the robot controller 120 and then stored in the memory 122 where they are available for execution. It is noted that the programming device 160 may be implemented as a set of collaborating components within the robot controller 120. In fact, the components can be shared with the robotcontroller 120, e.g., by using the processing circuitry 121 for the dual purposes of robot control and programming and / or using the memory 122 for the same dual purposes. In other words, the programming device 160 may constitute a portion of a multi-purpose device; it need not be a standalone device or a device with programming as its sole or main purpose.

[0034] The programming device 160 - which is shown as a standalone device in the non-limiting example of figure 1 - comprises processing circuitry 161, memory 162 with executable software 163, and at least one communication interface 164, 165. The programming device 160 may further comprise at least one operator interface 166, in particular a graphical operator interface or graphical user interface (GUI).

[0035] The programming device 160 may further have an imaging device (e.g., camera, video camera, depth camera, lidar, radar) 167 at its disposal, by which images or video of the demonstration-based programming session can be captured. In particular, the imaging device maybe a depth camera, lidar or radar, which in addition to - or instead of - the two-dimensional appearance of an object determines a depth coordinate of the object, e.g., by time-of-flight measurements, triangulation, reflection or other per se known techniques. An RGB-D camera is an example of a depth camera. The imaging device 167 may be a part of the programming device 160, or it maybe integrated into the industrial robot 100 and optionally be used for other tasks as well. More generally, an equivalent function may be achieved by any imaging device which is in rigid relationship with the robot base 112.

[0036] The programming device 160 is enabled to capture speech data, which could represent utterances or narration by the operator 190 during the programming session. For this purpose, the programming device 160 may utilize one or more acoustic transducers (microphones) 168 arranged in the vicinity of the operator’s 190 normal position. Alternatively, the speech data may be captured by a portable microphone or a headset worn by the operator 190. To parse speech data, the programming device 160 may utilize a natural-language model 171, which may be either stored internally or - as in the example configuration shown in figure 1 - may be available from a host computer or an external memory 170.

[0037] The natural-language model 171 stored in the external memory 170 may constitute a large language model (LLM), that is, a type of machine-learning algorithm which has been trained on very large datasets using deep learningtechniques to be able to perform natural-language processing (NLP) tasks. Example NLP tasks are recognizing, summarizing, translating, predicting and generating plausible textual content. A very large dataset in this sense may include of the order of one million parameters, such as tens of millions of parameters, such as hundreds of millions of parameters. An LLM may have a transducer architecture, particularly a transducer architecture with four cascaded key links when it processes the input data, namely: word embedding, position encoding, self-attention mechanism, feedforward neural network. At the time of filing this disclosure, noteworthy example LLMs include Bidirectional Encoder Representations from Transformers (BERT), Bard, BLOOM, Claude 2, various versions of Generative Pre-trained Transformer (GPT), Llama, PaLM 2, RoBERTa, T5, LaMDA, Turing NLG, Gemini 1.0. LLMs include, as a special case, multimodal models, such as Gemini 1.5. One benefit of an LLM is that the vocabulary is practically open-ended. The operator 190 can start using it without prior training. The demonstration-based programming can be carried out without requiring the operator to be extremely focused on using the right command words (or avoiding them) and / or syntax.

[0038] Optionally, to support some embodiments herein, the external memory 170 may further store a vision-enabled language model (VLM) 172. In alternative terminology, the VLM 172 may be referred to as a vision-language model. A VLM is machine-learning algorithm which has been trained on very large datasets to be able to perform processing which inputs or outputs combinations of textual and visual information. (It is emphasized that the training datasets need not be related to robot operation or to robot tasks, but the inventors have obtained advantageous results using generic VLMs.) A VLM is a multimodal model. A VLM may in particular be a vision-enabled LLM which, in addition to the above-reviewed characteristics of an LLM, is suitable for vision-centric tasks. Example types of VLM are contrastive-based models, models with masking objectives, generative-based models, or models from pretrained backbones. At the time of filing this disclosure, noteworthy example VLMs include versions of DALL-E, GPT-4 Vision, GPT-4 Turbo, GPT-40 and Gemini 1.5. At least DALL-E has functionalities for generating images from textual information (this is not a mandatory for implementing the present invention), and the further example models have functionalities for analyzing and understanding input images.

[0039] During a demonstration-based programming session, the programming device 160 has access to position data representing an actual position, a recorded position or recorded movements of the robot manipulator no, as well as speech data is captured while recording the movements of the robot manipulator. Optionally, according to some embodiments, the programming device 160 further has access to images or video of the robot programming session. The position data may for example be obtained through the intermediary of the robot controller 120, which monitors position data in the normal course of its operation; alternatively, the programming device 160 is granted access to corresponding signals from the transducers, sensors or other measuring equipment in the robot manipulator no.Robot programming method

[0040] Turning to the flowchart in figure 2, embodiments of a method 200 of programming an industrial robot 100 of the type depicted in figure 1 will now be described. Not all steps shown in figure 2 are necessarily carried out in all embodiments, wherein at least the steps indicated with dashed contour are optional. The output of the method 200 includes a robot program C to be executed by the robot controller 120 of the industrial robot 100 in figure 1. The robot program C maybe executable only by the robot controller 120 of the industrial robot 100 for which the programming method 200 was carried out. In some embodiments of the method 200, the robot program C can be executed also by further robot controllers of the same model as - or compatible with - the one for which the programming method 200 was carried out.

[0041] A robot program C may include a sequence of robot commands that cause the robot manipulator no to reproduce the trajectory. The robot commands may be selected from a predefined set of robot commands executable by the robot controller 120, such as commands compliant with the RAPID™ robot programming language.

[0042] The method 200 in figure 2 may be performed by a general-purpose processor and on the programming device 160 in particular. As mentioned, the programming device 160 may constitute a portion of a multi-purpose device. The method 200 may be considered to be a description of the behavior which the programming device 160 is configured for when active. The method 200 may as well correspond to the behavior of the robot controller 120 in a special programming mode, which the robot controller 120 can be requested to enter. Instructions for causing a computer - or theprogramming device 160 in particular - to carry out the method 200 maybe provided in the form of a computer program 163 (see figure 1).

[0043] The functioning and other characteristics of the method 200 may be better understood from the illustration in figure 3 of the information flows which occur during an execution of the method 200. Here, in addition to the reference numbers from figures 1 and 2, the following notation is used:301 demonstration-based robot programming session302 speech data303 video304 time-stamped speech data305 descriptions of the video306 robot parameter values307 robot trajectory308 robot trajectory annotated with robot parameter values309 robot program.The processes presented for the first time in figure 3 carry the following reference symbols:320 transcription of speech data330 heuristics340 conversion into robot commands.

[0044] A first step 210 of the method 200, which is executed during a demonstration-based robot programming session, includes recording movements of the robot manipulator no. On the basis of the recorded movements, a robot trajectory can be obtained, e.g., by combining the recorded movements. The robot trajectory 307 thus obtained may indicate the robot manipulator’s 110 position and / or pose as a function of time, wherein the position or pose maybe expressed in Cartesian or jointspace coordinates. The position or pose need not be indicated for all points in time. Alternatively, the trajectory 307 may be expressed as a number of reference times at which the robot manipulator no is to assume corresponding setpoint positions orsetpoint poses, wherein the robot manipulator no is free to have arbitrary positions or poses in the intervals between the reference times.

[0045] In a second step 211, speech data 302 is captured while the movements of the robot manipulator no are being recorded. For this purpose, the acoustic transducer 168 maybe utilized. The speech data 302 maybe stored in any suitable audio format, and at a bitrate considered adequate for the application at hand.

[0046] The speech data 302 may include utterances or narration by the operator 190 during the programming session. One the one hand, the speech data may act to confirm an activity which can in principle be inferred from the recorded trajectory 307. On the other hand, the speech data 302 may adduce information, such as a purpose or desired result of a certain sequence of movements, or a property / quality which is not derivable from the recorded trajectory 307 and / or not visible in a video of the programming section.

[0047] For instance, the speech data 302 may be used as a medium for indicating a setpoint speed of the robot manipulator no. This maybe helpful as it is, most often, practically impossible to perform the desired manipulator movements at the true speed without sacrificing accuracy and safety. The speed indications may be provided in terms of everyday speed-related vocabulary like “fast”, “slowly”, “quite slowly” or relative speed indications such as “faster”. Further, the speech data 302 may indicate a required degree of compliance with the recorded robot trajectory 307, such as: “Move the robot close to this object”, “Reorient the box handle towards the top”, “Reorient the object so it is aligned with the place position”, “Carefully proceed with the insertion”, “Distribute the paint evenly”. Further, the speech data 302 may indicate a required degree of movement precision or an acceptable geometric deviation from the trajectory, such as: “Move carefully downwards to do the insertion”, “I am moving the robot back to the home position”, “I will move the robot close to the object to grip”. Further, the speech data 302 may indicate, explicitly or implicitly, a reference frame in which the movements of the robot manipulator no are to be expressed, such as: “I am taking the robot back to the home position”, “I am approaching the red cube to pick up”, “I am inserting the tube in the hole”. Further still, the speech data 302 may indicate a handling force or a gripping force (or contact force), such as: “firmly”, “gently”, “hard”.

[0048] In a next step 213, the captured speech data 302 is decomposed into a plurality of phases. The phases may correspond to different time intervals of a captured audio track or different segments, and the segments may be contiguous in time. They should preferably correspond to different robot activities, so that parsed parameter values (see step 215) can be applied naturally. Normally, a time-uniform decomposition is not very useful, as it does not account for differences in duration of the different activities. The robot activities maybe sub-tasks (e.g., picking a cube, rotating a handle), steps explicitly mentioned by the operator 190 (e.g., “Now I will begin the process of moving backwards”) or robot operating modes (e.g., linear motion, motion parallel to the table, opening gripper, closing gripper), or various combinations of these. The phases may be annotated with respective human- intelligible labels. Optionally, the content of these labels is derived from the speech data, e.g., based on utterances by the operator 190 such as “The preheating is now complete, and I move the workpiece from the oven to the mold to begin the shaping.” Such a statement maybe interpreted as marking the end of a preheating phase and the start of a shaping or molding phase.

[0049] The decomposing into phases in step 213 may be carried out using the natural-language model 171, which may be selected as one of the examples reviewed above.

[0050] In the robot trajectory 307, it is possible to identify phases that correspond to the phases of the speech data 302 from step 213. The phases maybe corresponding with respect to the activity or subject-matter occurring therein. Alternatively, the corresponding phases in the trajectory 307 maybe contemporaneous with those of the speech data 302, in which case they can be identified by matching start and end times of the phase of the speech data 302 with time indications forming part of the recorded trajectory 307.

[0051] The execution flow of the method 200 then goes through the phases of the speech data 302 in the aim of providing an annotated robot trajectory 308. As long as there is at least one more phase to process - this is what is assessed in decision point 217 - the steps 215 and 216 are carried out in respect of the next phase.

[0052] In step 215, the present phase of the speech data 302 is parsed into at least one robot parameter value 306 using the natural-language model 171. The parsing may include transcribing the speech 302 data into timestamped text 304 (component320 in figure 3 represents the transcribing and timestamping), after which the natural-language model 171 is used for extracting robot parameter values from the timestamped text. Step 214 may return robot-parameter values pertaining to one or more robot parameters; further, step 214 may return one or more parsed robotparameter values for each of said robot parameter or robot parameters. The robot parameter may be, for example, a state of a tool carried by the robot manipulator, a movement execution parameter, a degree of compliance with the robot trajectory (e.g., to what extent smoothing, straightening or removal of sharp bends is permissible), a degree of movement precision, a reference frame, a drive system parameter, a motion template to be used for realizing the robot trajectory, or the like. The robot parameter shall apply locally, i.e., it applies to the present phase but not - at least not by default - to the full robot trajectory.

[0053] As used herein, a motion template is a preconfigured combination of robot commands (or a program snippet), which represents a sequence of movements and / or status changes of the robot manipulator no, typically with a time duration significantly less than it takes to complete a robot task, so that the motion templates maybe used as building blocks when carrying out the robot task. For example, a motion template maybe suitable for realizing linear motion, spiral search, hole insertion, polishing, welding, an act of opening or releasing a gripping tool. So-called reference movements in RAPID™ are motion templates in this sense.

[0054] A motion template may be configured for execution while the robot is operated in force-control mode, or while the robot is operated in position-control mode. A robot manipulator no with multiple degrees of freedom (DOFs) maybe configurable for force control or position control independently for each DOF. Reference is made to the research paper M. H. Raibert et al., “Hybrid Position / Force Control of Manipulators”, Transactions oftheASME, vol. 102 (1981), pp. 126-133, which refers to a so-called constraint frame, a matrix populated with binary values which specify for each linear or angular DOF whether this DOF shall be position- controlled, force-controlled or free (i.e., unconstrained).

[0055] A motion template may include at least one robot property which is configurable, such as movement speed (e.g., maximum linear or angular speed, exact linear or angular speed), degree of compliance with the robot trajectory (e.g., compliance with setpoint positions or poses), degree of movement precision, grippingforce of a robot tool, status of a robot tool (e.g., gripper open, gripped closed), handling force of the manipulator no (e.g., insertion force, screw torque), geometric movement constraints (e.g., center point and radius of a spiral movement, setpoint depth of an insertion operation) and the like. The concrete value of the property is indicated by a template argument. Optionally, the motion template may specify a default value to be applied in the absence of an explicitly given template argument.

[0056] In the method 200, the one or more robot parameter values parsed from the speech data 302 may include one or more template arguments. Optionally, any further template arguments which are needed in order to make the motion template well-defined maybe derived from the parsed robot parameter values 306 and / or the recorded trajectory 307. For this purpose, heuristics 330 maybe applied to the robot parameter values 306 or trajectory 307. The heuristics maybe encoded as software instructions (conventional programming), or they may be incorporated in a neural network which has been trained to make a set of robot parameters complete. In particular, such a neural network may have been trained to locate missing template arguments in a motion template and to predict contextually suitable values of these template arguments. More generally, the heuristics 330 may be used to derive a robot parameter value, or a modification of a robot parameter value, from the robot trajectory 307.

[0057] To illustrate, Tables 1 and 2 contain human-readable instructions of two example heuristics to be used, respectively, with a first motion template adapted for an insertion task and a second motion template for a picking task.

[0058] The shown example heuristics are template-specific in the sense that the instructions in Table 1 are to be used in a phase of the trajectory where the trajectory 307 is being realized by an insertion motion template, and the instructions in Table 2 are to be used in connection with a pick motion template. The heuristics according to Tables 1 and 2 maybe considered to codify practical experience of assigning workable parameter values (template arguments) to these motion templates when external instructions (e.g., detailed wishes of customer, complete measurements) are underdetermined. It will be within the abilities of those skilled in the art to formulate heuristics for further motion templates, by routine practice and possibly experimentation. Those skilled in the art will also be able to express the heuristicsaccording to Tables i and 2 in a computer-readable form which suits a concrete use case under consideration or, equivalently, to train a neural network to predict the outcome of the heuristics.

[0059] The teachings disclosed herein enable the use of a natural-language model 171 for parsing the speech data 302 into a novel type of robot parameter, namely, a robot parameter indicating whether to operate the robot in force-control mode or position-control mode. The natural-language model 171 will be fed with a prompt indicating which control modes are available, so that it can decide accordingly. These control concepts have been introduced above and exemplified with reference to figures 4 and 5. In particular, the novel type of robot parameter may indicate for two or more individual DOFs of the robot manipulator no whether the DOFs are to be position-controlled or force-controlled. If at least one DOF is configured as position- controlled or force-controlled, then a further DOF may further be configured as unconstrained. This may be expressed by a robot parameter which selects a corresponding motion template from a predefined set of motion templates, i.e., a motion template to be executed while the robot is operated (globally or for a particular DOF) in force-control mode or position-control mode. In the special case where force control vs position control is a configurable property of the indicated motion template, the robot parameter also includes a template argument which unambiguously indicates either force control or position control.

[0060] In step 216, a corresponding phase of the robot trajectory 307 is annotated with the robot-parameter values from step 216. The output of step 216 may include the extracted robot parameter values associated with the respective phases of the robot trajectory 307. Figure 3 shows an example appearance of the annotated robot trajectory 308, where the robot parameters and robot parameter values (preceded by dashes) are listed phase by phase. In “Phase 1”, the symbols pi and p4 denote predefined points in space. The “move” template represents an originally recorded movement of the manipulator no from pi to p4. The annotated trajectory 308 which forms the output of step 216 is to be provided in a form that can be handed over safely to the downstream processing steps without losing or corrupting the information. The output is preferably organized phase by phase, that is, the set of robot-parameter assignments for each new phase are contained in a new item in the output. In particular, the extracted robot parameter values may be formatted in accordance with adata serialization format segmented into phases. Examples data serialization formats are JSON (specified in The JSON Data Interchange Syntax, Standard ECMA-404, 2ndedition (2017-12)), YAML (specified in YAML Ain’t Markup Language (YAML™), version 1.2, revision 1.2.2 (2021-10-01)) and XML (specified in XML Signature Syntax and Processing, version 1.1, W3C Recommendation (201304-11)).

[0061] In a next step 218, a robot program 309 is generated (cf. component 340 in figure 3) on the basis of the annotated robot trajectory 308, in accordance with the robot parameter values 306. The robot program 309 is executable by the robot controller 120. The robot program 309 maybe expressed as a sequence of robot commands (e.g., RAPID™ commands) which, when executed by the robot controller 120, cause the robot manipulator no to realize the trajectory 301 while applying the robot parameter values as modifiers to these robot commands.

[0062] Step 218 may include a first substep of sampling the trajectory into a sequence of discrete points and a second substep of selecting robot commands that cause the robot manipulator 110 to move between each pair of consecutive discrete points. The sampling used in the first substep may be time-uniform sampling (constant step duration), space-uniform sampling (constant step length), or a non- uniform sampling algorithm with controlled deviation, such as Ramer-Douglas- Peucker. In a RAPID™ environment, the second substep may be performed so as to output instances of the command MoveL (cartesian linear motion) or MoveJ (jointspace linear motion) or a combination of these. Optionally, the step 218 may include a postprocessing substep applied to the sequence of generated robot commands, such as formatting the sequence into a predefined script format by appending a header, performing a consistency check, or the like. The execution of the method 200 may end at this point.

[0063] According to a further development, the robot programming method 200 further includes a step 212 of capturing a video (or video recording, or video sequence) 303 of the programming session. The video 302 maybe captured by means of the fixed imaging device 167 or by an imaging device (not shown) mounted on the robot manipulator no. Each frame of the video 303 maybe a visual image, a depth image or a combination of these. For the purposes of the present method 200, it is not essential that the video 303 have a high frame rate (e.g., a frame rate which is capable of capturing naturally-looking body movements). Rather, the video 303 maybe a sequence of sparse, even irregularly spaced, images of the work area 150. The above-discussed phases of the speech data 302 correspond directly to contemporaneous phases of the video 303.

[0064] In a subsequent step 214, the VLM 172 is prompted to generate a description 305 of the thus captured video 303, and the description 305 is utilized in providing the annotated robot trajectory in step 215 or 216. The VLM-generated description 305 may in particular relate to the movements of the robot manipulator no as seen in the video 303. The descriptions 305 maybe expressed in natural language. Alternatively, the descriptions 305 maybe expressed in latent space of the VLM 172, e.g., in the form of a bitstring or number which can be transferred to the next processing step 215 or 216. In embodiments where the natural-language model 171 and VLM 172 are implemented as a single multimodal model, the latent space is common to the natural-language model 171 and VLM 172, and the latent-space representation can be made available internally, without a need for conversion into a transferable format. A benefit of expression the descriptions 305 in latent space is a possible reduction of the influence from random variations in wording and vocabulary, which are inherent in natural language, so that ultimately the annotations of the trajectory become more deterministic.

[0065] The VLM-generated description 305 may be utilized by being parsed into robot parameter values 306 using the natural-language model 171. The naturallanguage model 171 maybe applied to a combination of the transcribed and timestamped speech data 304 and the VLM-generated description 305, as shown in figure 3. Alternatively, the VLM-generated description 305 is parsed separately, so as to obtain robot parameters in addition to those parsed from the speech data 302, or modifications to the robot parameters parsed from the speech data 302. In particular, the VLM-generated description 305 maybe parsed into a template argument referring to a motion template to be used for realizing the robot trajectory 307.

[0066] Optionally, the descriptions 305 carry timestamps, such that they can be associated with a corresponding point in, or phase of, the trajectory 307. Example neural networks suitable for use as the VLM 172 were reviewed above.

[0067] In some embodiments within this further development of the method 200, the step 214 is restricted to a set of extracted keyframes, in the sense that the descriptions 305 are generated for the keyframes but not for the remaining frames. Inparticular, the descriptions 305 may relate to movements which are deemed to have occurred in the intervals between the extracted keyframes, wherein the VLM 172 is prompted to describe the movement between a current keyframe and the preceding keyframe. According to one implementation, a video frame shall be identified as a keyframe if it depicts a significant visual change and / or it follows immediately after a significant visual change. A visual change may correspond to a local minimum or local decrease in a similarity of successive frames. Similarity may for example be quantified by a structural similarity index measure (SSIM), of which one definition is given in Wang, Z. et al., “Multiscale structural similarity for image quality assessment”, The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol. 2, pp. 1398-1402.

[0068] Restricting step 214 to keyframes (or keyframe intervals) limits the computational burden on the VLM 172 while enabling acceptable performance to be upheld. Further, the restriction to keyframes (or keyframe intervals) also acts as a downsampling or low-pass filtering; this tends to eliminate movements on a short time scale, which generally have a lower density of useful information. If the extracted keyframes carry timestamps, step 214 may include transferring the timestamps to the descriptions 305.Example

[0069] In an Example, a GoFa™ robot (5 kg payload version) manufactured by the applicant was taught to insert a mobile-device charger and insert it into an electric socket (“slot”). The robot was equipped with a 2FG7 parallel robot gripper manufactured by OnRobot, Odense, Denmark.

[0070] The teaching was carried out by demonstration-based robot programming in accordance with the described method 200, wherein operator-guided movements of the robot manipulator were recorded (realization of step 210) during which utterances by the operator 190 was captured (speech data; step 211). A transcription of the utterances is found in Table 3, to which a start time and end time of each word were added. Further, a depth video of the demonstration-based robot programming section was recorded (step 212) using a stationary depth camera directed at the work area 150 of the robot. The depth camera was a RealSense™ D405 camera manufactured by Intel, Santa Clara, California.

[0071] To carry out step 215, an LLM “gpt-4-32k” version 0613, hosted by Microsoft Azure™, was used as the natural-language model 171. The LLM has a so-called temperature setting representing the randomness of its output, i.e., how much ‘creative’ freedom the LLM has. The useful temperature values range from o to 1.0, where o represents deterministic output and 1.0 maximal randomness. The value 1.0 is oftentimes used for communication with users, especially in chatbots or entertainment applications. A temperature value of 0.2 was used in the present Example, which causes the LLM to provide a near-deterministic output (highest probability generation), while maintaining a certain randomness / variability of the LLM’s output. Because the same prompt leads to marginally different outputs of the LLM thanks to the residual randomness for 0.2, operators 190 has the option of performing re-generations based on identical input if the initial output is not according to their expectations.

[0072] The natural-language model 171 was given the prompt in Table 4, which requests it to select motion templates from the predefined set of seven listed motion templates i)-vii), and to assign values to their template arguments (“parameters”) as needed. The natural-language model 171 is further requested to add labels to each demonstrated task, add a task description, and to decompose the speech data into steps (“phases” in the sense of step 213). In other words, the steps 213, 215 and 216are performed by a single call to the natural-language model 171. The final portion of the prompt in Table 4 is the timestamped speech data in JSON format.

[0073] In response to this prompt, the natural-language model 171 provided the output shown in Table 5. The output outlines the sub-steps of the task demonstration and provides appropriate robot parameter values, similar to the per-phase annotated trajectory 308.

[0074] Based on this description, RAPID™ code for the demonstrated task was automatically generated (step 218). The RAPID™ code may have a similar format and significance as the program code 309. Excerpts of the resulting pick-and-place robot program are shown in Table 6. the somewhat unwieldy initial definitions of constantwaypoints (robtargets) Charger_Inser pi, Charger_Inser p2 etc. have been omitted for space reasons.

[0075] The RAPID code was then tested on the GoFa™ (5 kg) robot. While executing the generated RAPID program, the robot successfully carried out the demonstrated task, by only replicating from the demonstration the most relevant positions and orientations that the robot must go through. The robot successfully switched from position control into force control mode just before the insertion phase, and then went back to the position control mode in order to return to the home position. The results were considered satisfactory.

[0076] The aspects of the present disclosure have mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

Claims

CLAIMS1. A method (200) of programming an industrial robot (100), which comprises a robot manipulator (no) and a robot controller (120), the method comprising: recording (210) movements of a robot manipulator during a demonstration-based programming session, for thereby obtaining a robot trajectory (307); capturing (211) speech data (302) while recording the movements of the robot manipulator; decomposing (213) the captured speech data into a plurality of phases, wherein the robot trajectory has corresponding phases; providing an annotated robot trajectory (308) by performing the following for at least one phase of the speech data: using a natural-language model (171), parsing (215) the speech data into at least one robot parameter value (306); annotating (216) a corresponding phase of the robot trajectory with the at least one robot parameter value; and generating (218) a robot program (309) on the basis of the annotated robot trajectory, characterized in that the robot parameter value indicates a motion template to be used for realizing the robot trajectory, wherein the motion template is selected from a predefined set including at least one motion template in which the robot is operated in force-control mode and at least one motion template in which the robot is operated in position-control mode.

2. The method (200) of claim 1, wherein the robot parameter value further includes a template argument indicating a configurable property of the selected motion template.

3. The method (200) of claim 2, wherein the template argument is one or more of: movement speed, degree of compliance with the robot trajectory, degree of movement precision, gripping force of a robot tool, status of a robot tool, handling force, geometric movement constraints, whether to operate the robot in force-control or position-control mode.

434. The method (200) of any of the preceding claims, further comprising: capturing (212) a video (303) of the programming session; and generating (214), using a vision-enabled language model, VLM (172), a naturallanguage description (305) of phases of the video, which is utilized in providing the annotated robot trajectory.

5. The method (200) of claim 4, further comprising: parsing the VLM-generated description (305), in addition to the speech data (302), into said at least one robot parameter value (306).

6. The method (200) of claim 4, further comprising: parsing the VLM-generated description (305), in addition to the speech data (302), into a template argument indicating whether the robot is to be operated in force-control mode or positioncontrol mode.

7. The method (200) of any of the preceding claims, wherein said annotating (216) comprises deriving a robot parameter value, or a modification of a robot parameter value, from the robot trajectory (307) on the basis of predefined heuristics.

8. The method (200) of claim 7, wherein the predefined heuristics are adapted for locating one or more missing template arguments in a motion template and to predict contextually suitable values of the template arguments.

9. The method (200) of claim 7 or 8, wherein multiple predefined heuristics are selectively applied, each being specific to one or more motion templates in the predefined set.

10. The method (200) of any of the preceding claims, wherein the natural-language model (171) and the VLM (172) are implemented as distinct models.

11. The method (200) of any of claims 1 to 9, wherein the natural-language model and the VLM are implemented as a single multi-modal model.

12. A programming device (160) for facilitating programming of an industrial robot (100), which comprises a robot manipulator (no) and a robot controller (120), the programming device comprising memory (162) and processing circuitry (161) configured to: record movements of a robot manipulator during a demonstration-based programming session, for thereby obtaining a robot trajectory (307);44 capture speech data (302) while recording the movements of the robot manipulator; decompose the captured speech data into a plurality of phases, wherein the robot trajectory has corresponding phases; provide an annotated robot trajectory (308) by performing the following for at least one phase of the speech data: parse, using a natural-language model (171), the speech data into at least one robot parameter value (306); annotate the phase of the robot trajectory with the at least one robot parameter value; and generate a robot program (309) on the basis of the annotated robot trajectory, characterized in that the robot parameter value indicates a motion template to be used for realizing the robot trajectory, wherein the motion template is selected from a predefined set including at least one motion template in which the robot is operated in force-control mode and at least one motion template in which the robot is operated in position-control mode13. A computer program (163) comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method (200) of any of claims 1 to 11.

Images