Workflow-based optimization of robot operation sequences

Abstract

A device includes interface hardware, a memory, and processing circuitry. The interface hardware is configured to receive input parameterizing a robot operation sequence. The memory is configured to store data describing the robot operation sequence. The processing circuitry is communicatively coupled to the interface hardware and to the memory. The processing circuitry is configured to identify a variable associated with the robot operation sequence. The processing circuitry is further configured to assign multiple values to the variable, and to generate multiple workflows, where each respective workflow represents a respective run of the robot operation sequence using a corresponding value selected from the multiple values assigned to the variable. The processing circuitry is further configured to execute each respective workflow of the multiple generated workflows.

Description

WORKFLOW-BASED OPTIMIZATION OF ROBOT OPERATION SEQUENCESTECHNICAL FIELD

[0001] This disclosure generally relates to the technical field of robotics.BACKGROUND

[0002] Robotics technology refers to the design, construction, operation, and use of automated systems (“robots”) to perform a variety of tasks. The deployment of robots to perform many such tasks improves the accuracy, efficiency, and scalability of these operations. Robots are often programmable by way of programmable circuitry and / or fixed-function circuitry, and in many use case scenarios, can be programmed to carry out sequences of operations automatically.

[0003] Robots have been deployed in many industrial settings, such as in the fields of abrasives (e.g., sanding and smoothing), automotive manufacturing (e.g., to apply paints and protective coatings), hydraulics, dispensing technology, additive manufacturing, pharmaceuticals, manufacturing, electronics foundries, and many others. Often, the operation of these robots is controlled via a computing device to mechanical device interface, with operation instructions and (optionally) parameters being supplied via the computing device.SUMMARY

[0004] In cases of effectuating simple movements of a robotic modality, the computer-to-robot communication tends to be low dimensional and can thus be parameterized easily and entered via an input interface of the computing device (e.g., a command line in a user interface or “UI” in a human-machine interface scenario or an automated interface in a computer-to-computer interface scenario). However, as the movements of the robot become complex and / or the target item (or “part”) on which the robot operates has a more complex topography / ontology, these relatively simple input interfaces and / or operational commands no longer support higher dimensional movement trajectories and robot modality operations. As such, in controlling higher dimensionality movements and complex operations, commands to support the higher dimensionalities via a robot pendant, a Ul-provided input mechanism, or a command line interface becomes cumbersome and potentially futile, thereby limiting the ability to avail of the robot’s full operational potential. For instance, a robot pendant, which generally has the form factor of a handheld computing device, can process relatively simple instructions received via a UI. Pendants often enable users to physically drive a robot to points of interest and mark those points of interest as part of charting out a program. In scenarios in which paths need to be customized or otherwise adjusted dependingon context (e.g., in low-mix, high part variability scenarios), these pendants (or “teach pendants” as they are sometimes referred to) tend to be slow and cumbersome to work with. This process tends to be labor intensive and time intensive, creating significant scope for introducing operator-induced inaccuracies into the robotic process, thereby increasing resource usage as well as diminishing data precision.

[0005] In one example, this disclosure is directed to a device that includes interface hardware, a memory, and processing circuitry. The processing circuitry is communicatively coupled to the interface hardware and to the memory. The interface hardware is configured to receive input parameterizing a robot operation sequence. The memory is configured to store data describing the robot operation sequence. The processing circuitry is communicatively coupled to the interface hardware and to the memory. The processing circuitry is configured to identify a variable associated with the robot operation sequence. The processing circuitry is further configured to assign multiple values to the variable, and to generate multiple workflows, where each respective workflow represents a respective run of the robot operation sequence using a corresponding value selected from the multiple values assigned to the variable. The processing circuitry is further configured to execute each respective workflow of the multiple generated workflows.

[0006] In another example, this disclosure is directed to a method. The method includes receiving input parameterizing a robot operation sequence. The method further includes storing data describing the robot operation sequence. The method further includes identifying a variable associated with the robot operation sequence. The method further includes assigning multiple values to the variable. The method further includes generating multiple workflows, where each respective workflow represents a respective run of the robot operation sequence using a corresponding value selected from the multiple values assigned to the variable. The method further includes executing each respective workflow of the multiple generated workflows.

[0007] In another example, this disclosure is directed to a non-transitory computer-readable storage medium encoded with instructions. The instructions, when executed, cause processing circuitry of a computing device to perform operations. The operations include receiving input parameterizing a robot operation sequence. The operations further include storing data describing the robot operation sequence. The operations further include identifying a variable associated with the robot operation sequence. The operations further include assigning multiple values to the variable. The operations further include generating multiple workflows, where each respective workflow represents a respective run of the robot operation sequence using a corresponding value selected from the multiple values assigned tothe variable. The operations further include executing each respective workflow of the multiple generated workflows.

[0008] Aspects of this disclosure are directed to computer-robot interfaces that enable extraction of complex operational parameter data and enable delivery of operational commands that are based on these complex parameters to a robot controller. One example of a complex parameter set that can be ingested by the systems of this disclosure is topographical data describing facets of a part surface over which the robotic modality navigates and upon which the robot modality performs operations. The systems of this disclosure are configured to ingest geometric models as input data and generate robot / part calibration and / or robot navigation path data as output.

[0009] By generating calibration and / or navigation path data using a model-based approach, the systems of this disclosure provide various technical improvements in the technical field of robotics. As one example, the systems of this disclosure improve data precision by enabling robot operation using complex input parameter sets. For instance, the systems of this disclosure enable a Ul-driven control of a robot that includes complex sequences of operations implemented on a complex part configuration. As another example, the systems of this disclosure reduce complexity by implementing a UI that ingests no-code input that is processed to output the complex robot operation sequences on potentially complex part configurations described above. In this way, the systems of this disclosure enable users to plan and efficiently communicate high-dimensional, part-topography-aware paths via an easy-to-use UI.

[0010] Further aspects of this disclosure are directed to implementing an automated robot testing environment via the UI described above. The automated testing techniques of this disclosure target optimization with respect to robot operation via the Ul-based input mechanisms of this disclosure. For example, the automated testing configurations of this disclosure may enable a base test using a base set of parameters and may enable automated or manual variation of these parameters to generate alternative test results to aid in identifying optimal parameter selection. By enabling potentially large numbers of tests using a set of “seed” parameters and variations generated therefrom, the automated testing systems of this disclosure provide various technical improvements in the technical field of robotics. For instance, the automated testing techniques of this disclosure provide improved data precision by enabling identification of optimal parameter sets for use in a deployment scenario. As another example, the automated testing techniques of this disclosure provide increased scalability by enabling testing of numerous parameter profiles with a low set of input data in the form of a base parameter set.BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a conceptual diagram illustrating an example implementation of a system of this disclosure.

[0012] FIG. 2 is a block diagram illustrating an example implementation of a computing device configured to perform functionalities of this disclosure.

[0013] FIG. 3 is a screenshot illustrating a workflow setup of this disclosure.

[0014] FIG. 4 is a screenshot illustrating a robot path construction step of this disclosure.

[0015] FIG. 5 is a flowchart illustrating an example process that systems of this disclosure may perform in accordance with aspects of this disclosure.DETAILED DESCRIPTION

[0016] FIG. 1 is a conceptual diagram illustrating an example system 2 of this disclosure.System 2 illustrates an example of a user interface (UI) driven system that enables extraction of complex operational parameter data and enable delivery of operational commands that are based on these complex parameters to a robot controller. System 2 includes computing device 6 and robot controller 10. Computing device 6 may include, be, or be part of one or more of a variety of types of devices including desktop computers, laptop computers (including so-called “netbooks” and “ultrabooks”), mobile phones (such as so-called “smartphones”), wearable computing devices (such as so-called “smartwatches,” “smartglasses,” etc.) personal digital assistants (PDAs), tablet computers, convertible laptop / tablet computers, interactive televisions, gaming consoles, digital media hub, and / or various others.

[0017] Computing device 6 is configured to output user interface (UI) 4. In various examples, aspects of computing device 6 may output UI 4 for display, whether locally at computing device 6 or at a decoupled location in a distributed manner. In these examples, UI 4 may be displayed by a variety of display devices, such as monitors or televisions, or by input / output capable devices such as a touchscreen or a presence-sensitive display device. Computing device 6 may configure UI 4 to include different user interface controls, text, images, or other graphical elements. It will be appreciated that computing device 6 may operate UI 4 as an interactive graphical user interface (GUI). For instance, computing device 6 may update UI 4 to reflect input data received by computing device 6. Updating UI 4 may generally refer to the process of changing the contents of UI 4, which may be displayed to a user.

[0018] In the example of FIG. 1, computing device 6 is communicatively coupled with robot controller 10 via network 8. Network 8 may, in various examples, represent or include a private network associated with an association (e.g., a robotics operation network, etc.) orother entity or grouping of entities. In other examples, network 8 may represent or include a public network, such as the Internet. Although illustrated as a single entity in FIG. 1 purely for ease of illustration, it will be appreciated that network 8 may include a combination of multiple public and / or private networks. For instance, network 8 may represent a private network implemented using public network infrastructure, such as a virtual private network (VPN) tunnel implemented over the Internet. As such, network 8 may comprise one or more of a wide area network (WAN) (e.g., the Internet), a LAN, a VPN, and / or another wired or wireless communication network. Network 8 may include wired and / or wireless network components that conform to one or more standards, such as via Ethernet®, WiFi™, Bluetooth®, 3G, 4G LTE, 5G, and the like.

[0019] Again, one of the devices with which computing device 6 may be communicatively coupled via network 8 is robot controller 10. Robot controller 10 may, in various implementations, include any combination of one or more processors, one or more field programmable gate arrays (FPGAs), one or more application specific integrated circuits (ASICs), and one or more application specific standard products (ASSPs). Robot controller 10 may also include memory, including one or both of both static (e.g., hard drives or magnetic drives, optical drives, FLASH memory, EPROM, EEPROM, etc.) and / or dynamic (e.g., RAM, DRAM, SRAM, etc.), or any other computer-readable storage device or non- transitory computer readable storage medium capable of storing instructions that cause the processing circuitry (e.g., fixed-function circuitry and / or programmable processing circuitry) to operate a robotics modality in accordance with one or more of the techniques described in this disclosure.

[0020] As such, robot controller 10 may represent hardware or a combination of hardware and software operable to effectuate instructions received over network 8 from computing device 6. In various examples, robot controller 10 may be deployed directly at a robot operation site or may be located remotely from the site at which the associated robotic modality performs the end operations of system 2. In other examples consistent with embodiments of this disclosure, various aspects illustrated in FIG. 1 may be implemented purely digitally, such as in a simulated environment. For instance, one or more of robot controller 10, robot arm 12, part 14, and / or end-effector 16 may be implemented virtually in a simulation environment.

[0021] In the example of FIG. 1, robot controller 10 is communicatively coupled to robot arm 12. In some examples, such as in the implementation shown in FIG. 1, robot arm 12 may include multiple arm sections connected by joints that enable a hinging motion or the like. A distal end of robot arm 12 is coupled to an end effector 16. End effector 16 may be coupled to the distal end or distal tip of robot arm via various types of functional connectionmechanisms. Examples of these functional connectors may may include various mechanical and electrical means to functionally connect the end effector 16 to the distal end of robot arm 12. The combination of robot arm 12 and end effector 16 may be referred to herein as a “robot system.” In some examples, end effector 16 may be decoupled from, and therefore, at such time, not part of the robot system. As such, the term “robot system” can be used to referred to a “part in hand” configuration (in which robot arm 12 is configured to contact part 14) or a “tool in hand” configuration (in which robot arm 12 is coupled to end effector 16, and end effector 16 is configured to contact part 14).

[0022] In some examples, the functional connector may include any suitable fastening component configured to mechanically couple end-effector 16 to the distal end of robot arm 12 as well as any suitable communications hardware infrastructure operable to communicate electrical signals between robot arm 12 and end effector 16. In some such examples, the functional connector may also include electrical transmission hardware configured to conduct electrical power between robot arm 10 and end effector 16. Aspects of the operation of endeffector 16 are controlled by the locomotion of robot arm 12. For instance, the locomotion of robot arm 12 may adjust the position, orientation, movement trajectory, etc. of end effector 16.

[0023] Robot controller 10 may control the locomotion of robot arm 12 in accordance with one or more command programs. A command program may be a sequence of instructions with any required parameters provided, which, when executed, manifests as a locomotive process of robot arm 12. Robot controller 10 may control the locomotion of robot arm 12 such that end effector 16 navigates about the surface of part 14. While the non-limiting use-case illustrated in FIG. 1 shows part 14 as being an auto part (e.g., a bumper), it will be appreciated that system 2 can be deployed to operate on a variety of objects that can be alternative examples of part 14.

[0024] Robot controller 10 may execute a command program to control the locomotion of robot arm 12 such that end effector 16 can control a tool used to contact and move around the surface of part 14 to prepare the surface of part 14. Examples of surface preparation operations that end effector 16 may be used to perform on part 14 may include scuffing, general abrasive processes, sanding, polishing, grinding, abrasion testing, planar surface and / or complex surface abrasion, ultrasonic welding, etc. In some examples, robot controller 10 may include an optional power interface to a power source thereof to provide power to end-effector 16 in the form of electricity, pneumatic pressure, etc.

[0025] Computing device 6 may implement techniques of this disclosure to extract complex operational parameter data and deliver an operational command set formed from thesecomplex parameters to robot controller 10 for execution. For example, computing device 6 may ingest, via UI 4, a complex parameter set comprising topographical data describing facets of the surface of part 14. According to aspects of this disclosure, computing device 6 may also ingest geometric models (e.g., representing part 14 and / or end effector 16 as well as for components not shown in the use case scenario of FIG. 1) as input data. Using these inputs, computing device 6 may generate calibration information (e.g., relative poses) with respect to end effector 16 and part 14 as output and may also generate navigation path data for robot arm 12 as output. Computing device 6 may communicate these outputs over network 8 to robot controller 10 in the form of a command program, which robot controller 10 can execute at one or more appropriate start times to implement a surface preparation process with respect to part 14.

[0026] By generating calibration and / or navigation path data using the geometric model-based techniques of this disclosure, computing device 6 and system 2, as a whole, provide various technical improvements in the technical field of robotics. As one example, system 2 delivers enhanced data precision by enabling robot controller 10 to operate robot arm 12 according to input parameter set of scalable complexity. For instance, computing device 6 operates UI 4 to implement Ul-driven control of robot arm 10 according to potentially complex sequences of operations implemented on varying configurations of part 14, even if the configurations are relatively complex.

[0027] As another example, system 2 may reduce resource and bandwidth consumption by implementing UI 4 in a way that enables computing device 6 to ingest no-code input and use the no-code input to generate a command program that incorporates complex robot operation sequences based on potentially complex surface configurations associated with part 14. In this way, system 2 enables users of computing device 6 to plan and efficiently communicate high-dimensional, topography-aware paths with respect to part 14 via UI 4 in an easy-to-use manner.

[0028] Further aspects of this disclosure are directed to implementing an automated robot testing environment via UI 4. The automated testing techniques of this disclosure may be run in either a simulated or virtual environment implemented by computing device 6 and / or computing environments or may be carried out directly via robot arm 12 by communicating the test procedures as command programs to robot controller 10. The automated test procedures (or “workflows”) of this disclosure target optimization with respect to the operation of robot arm 10 via UI 4.

[0029] According to some examples of the automated testing techniques of this disclosure, computing device 6 may configure a base test using a base set of parameters and mayimplement automated or manually guided variation of these parameters to generate alternative test results to aid in identifying optimal operating parameter sets. By enabling potentially large numbers of tests using a set of “seed” parameters and variations generated therefrom, aspects of system 2 may implement the automated testing systems of this disclosure to provide various technical improvements in the technical field of robotics.

[0030] For instance, aspects of system 2 may implement the automated testing techniques of this disclosure to enhance data precision by enabling identification of optimal parameter sets for use in a scenario in which robot arm 12 and end effector 16 are deployed to prepare part 14. As another example, aspects of system 2 may implement the automated testing techniques of this disclosure to increase system reliability and scalability by enabling rapid testing of numerous parameter profiles with a small set of input data in the form of a base parameter set.

[0031] FIG. 2 is a block diagram illustrating an example implementation of computing device 6. While FIG. 2 shows one implementation of computing device 6 that is consistent with aspects of this disclosure, it will be appreciated that other architectures (whether singledevice or distributed architectures) of computing device 6 are consistent with aspects of this disclosure, as well.

[0032] In the example of FIG. 2, computing device 6 includes one or more processors 22 and memory 20. In some examples, memory 20 and processors 22 may be integrated into a single hardware unit, such as a system on a chip (SoC) or integrated circuit (IC). Each of processors 22 may comprise one or more of a multi -core processor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field- programmable gate array (FPGA), processing circuitry (e.g., fixed function circuitry, programmable circuitry, or any combination of fixed function circuitry and programmable circuitry) or equivalent discrete logic circuitry or integrated logic circuitry. Memory 20 may include any form of memory for storing data and executable software instructions, such as random-access memory (RAM), read-only memory (ROM), programmable read only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read only memory (EEPROM), and flash memory.

[0033] Memory 20 and processor(s) 22 provide a computer platform for executing operation system 28. In turn, operating system 28 provides a multitasking operating environment for executing one or more software components 30. As shown, processors 22 connect via an input / output (I / O) interface 24 to external systems and devices via one or more communicative networks, such as any one or more of a data-enabled telephony network (such as a cellular data network), a wide-area network (such as the Internet), a publicnetwork (such as the Internet), a private network, such as a local-area network (LAN) and / or an enterprise network, or any other type of network that enables data communications, or any combination of any two or more of the networks listed above. I / O interface 24 may incorporate network interface hardware, such as one or more wired and / or wireless network interface controllers (NICs) for communicating via communications link 18.

[0034] In the particular example of FIG. 2, communications link 18 represents one or more network-enabled communicative connections, such as a link to one or more packet-switched networks or other communicative networks. Communications link 18 may represent any of a data-enabled telephony network (such as a cellular data network), a wide-area network (such as the Internet), a public network (such as the Internet), a private network, such as a localarea network (LAN) and / or an enterprise network, or any other type of network that enables data communications, or any combination of any two or more of the networks listed above.

[0035] Communications link 18 represents a communicative connection implemented over a communicative network that may include, be, or be part of one or more wired connections (e.g., an Ethernet® connection), wireless connections (e.g., a Wi-Fi™ connection) or a combination of both wired and wireless communicative connections. Communications link 18 couples computing device 6 to robot controller 10. For instance, communications link 18 may represent a communication pathway implemented over network 8 or components thereof between computing device 6 and robot controller 10.

[0036] In the example illustrated in FIG. 2, I / O interface 24 also facilitates communication between computing device 6 and a system that trains and retrains AI / ML model 16 via communications link 18. Communications link 18 may represent any of the network-based connections listed above, or may represent one or more local connections, such as a connection to the system that trains and retrains AI / ML model 16 via a local area network (LAN) and / or personal area network (PAN) connections, such as one or more of near-field communication (NFC) pairings, Bluetooth® pairings, Zigbee® pairings, or the like. Again, communications link 18 may communicatively couple computing device 6 (by way of I / O interface 24) to a robot controller 10. In some examples, computing device 6 may invoke I / O interface 24 to signal a command program synthesized from input received via UI 4 over communications link 18 to robot controller 10 for execution via robot arm 12. In some examples, computing device 6 may invoke I / O interface 24 to signal a test suite (or “workflow”) synthesized from input received via UI 4 over communications link 18 to robot controller 10 for execution via robot arm 12.

[0037] Bus 26 provides inter-component connectivity between processors 22, memory 20, andI / O interface 24 in the implementation shown in FIG. 2. Bus 26 may represent a half-duplexor full-duplex bus that provides data transfer capabilities between two or more of processors 22, memory 20, I / O interface 24, and / or any other hardware components of computing device 6. Bus 26 may represent a system bus or a computer bus of various types, including one or more bus networks. Regardless of the topology implemented, bus 26 may, in various examples, incorporate various types of inter-component connectivity hardware such as those conforming to any of first generation, second generation, third generation, or fourth generation bus or bus network technology as set forth by the Institute of Electrical and Electronics Engineers (IEEE), and / or other bus or bus network technologies defined in developing or later-adopted standards.

[0038] Software components 30 of computing device 6, in the particular example of FIG. 2, include model analysis unit 30A, optimization unit 30B, path generation unit 30C, parameter processing unit 30D, ray tracing unit 30E, and calibration unit 30F. In some example approaches, one or more of software components 30 represent executable software instructions that may take the form of one or more software applications, software packages, software libraries, hardware drivers, and / or Application Program Interfaces (APIs). In some use-case scenarios, one or more of software components 30 may, when executed, cause computing device 6 to output data and / or receive input data via I / O interface 24.

[0039] Aspects of memory 20 that provide non-volatile storage and / or long-term storage support local storage of data repositories 32. In the example of FIG. 2, data repositories 32 include geometric models 34, robot operation parameters 36, pose information 38, and traversal paths 40. One or more of software components 30 may invoke processors 22 and memory 20 to access one or more of data repositories 32 to retrieve data for various purposes, or to store data generated by one or more of software components 30. In various examples, model analysis unit 30A and / or optimization unit 30B (or other aspects of computing device 6) may use one or more of geometric models 34 and generating one or more poses (to be stored to pose information 38) from the one or more of geometric models 34.

[0040] Software components 30 may implement read / write capabilities with respect to data repositories 32, such as to access and use information available from data repositories 32 and / or to modify information currently stored to data repositories 32. In implementations in which computing device 6 represents a distributed computing system, one or more of data repositories 32 may be positioned at a remote location from processors 22, and software components 30 may, in these implementations, access data repositories 32 using NIC hardware of I / O interface 24.

[0041] I / O interface 24 may include or be coupled with peripherals that provide data input and output capabilities to computing device 6. For instance, the I / O peripherals may implementUI 4 in a way that outputs data (e.g. in the form of a graphical user interface or “GUI”) to a user and receives input data via UI 4. For instance, computing device 6 may invoke I / O interface 24 to receive, via UI 4, one or more robot operation parameters. In turn, processors 22 may execute parameter processing unit 30D of software components 30. Parameter processing unit 30D may classify the robot operation parameters, which may, in various examples, pertain to one or more of paint repair, adhesive dispensing, ultrasonic welding, or surface abrasion with respect to part 14.

[0042] In other examples, parameter processing unit 30D may classify certain parameters as closed-loop feedback received from sensor hardware coupled (whether directly or indirectly) to robot arm 12. In any event, parameter processing unit 30D may store the parameters received via I / O interface 24 to robot operation parameters 36 of data repositories 32 implemented by memory 20. Additionally, computing device 6 may invoke I / O interface 24 to receive a geometric model describing a surface topography of part 14. In turn, processors 22 may store the received geometric model to geometric models 34 of data repositories 32 implemented by memory 20. In some examples, the geometric model may represent a three- dimensional (3D) geometric model describing the surface topography of part 14. At execution, parameter processing unit 30D may operate UI 4 in a way that provides a convenient set of widgets that enable users to define process parameters (e.g. a subset of robot operation parameters 36) to be input along with topographical data (e.g., in the form of one or more of geometric models 34) to enable the outlining of a robotic process plan.

[0043] Processors 22 may execute calibration unit 30F of software components 30. Calibration unit 30F may use the 3D geometric model of part 14 stored to geometric models 34 and the parameters stored to robot operation parameters 36 to calibrate pose information of robot arm 12 with respect to part 14. For instance, calibration unit 30F may form a calibrated pose that describes relative positioning and orientation of the distal tip of robot arm 12 (taking into consideration various facets of end effector 16) relative to the positioning and orientation of part 14. In some examples, calibration unit 30F may also form the calibrated pose based on the first point of contact on the surface of part 14 with respect to end effector 16. Calibration unit 30F may store the calibrated pose to pose information 38 of data repositories 32 implemented by memory 20. In this way, computing device 6 implements the techniques of this disclosure to form calibrated pose information of a robot with respect to a part using a 3D geometric model describing the topography of the part and one or more robot operation parameters to calibrate one of the poses stored to pose information 38.

[0044] Additionally, processors 22 may execute path generation unit 30C to implement robot path traversal formation techniques of this disclosure. Upon invocation, path generation unit30C may access the calibrated pose saved to pose information 38 for use in the path formation techniques of this disclosure. For instance, path generation unit 30C may extract the calibrated pose from pose information 38, and use the calibrated pose in combination with the 3D geometric model describing the surface topography of part 14 stored to geometric models 34 and the parameters stored to robot operation parameters 36 to form a traversal path of robot arm 12 (e.g., with facets of end effector 16 considered) with respect to part 14.

[0045] In a non-limiting use-case example pertaining to FIG. 1, path generation unit 30C may use data extracted from geometric models 34, robot operation parameters 36, and pose information 38 to form a traversal path for robot arm 12 such that end effector 16 effectuates a preparation process over the surface topography specific to a bumper that is a non-limited example of part 14. In some examples, the parameters that path generation unit 30C extracts from robot operation parameters 36 are associated with one or more of paint repair, adhesive dispensing, ultrasonic welding, or surface abrasion with respect to part 14. In some examples, path generation unit 30C may string together one or poses from pose information 38 (e.g., whether they are generated poses, calibrated poses, or any combination thereof) into a continuous path to be saved to traversal paths 40.

[0046] In other examples, the parameters that path generation unit 30C extracts from robot operation parameters 36 reflect closed-loop feedback received via I / O interface 24 from sensor hardware coupled (whether directly or indirectly via end effector 16) to robot arm 14. In these examples, the closed-loop feedback may be associated with one or more of paint repair, adhesive dispensing, ultrasonic welding, or surface abrasion. Other robotic processes may also have their results and / or performance metrics reflected in the closed-loop feedback.

[0047] As described above, calibration unit 30F and path generation unit 30C utilize a 3D geometric model representing the surface topography of part 14 as an input to execute their respective operation sets. To use the 3D geometric model representing the surface topography of part 14, calibration unit 30F and path generation unit 30C may invoke model analysis unit 30A. Model analysis unit 30A may, upon execution, implement one or more processes that extract and decompose the 3D geometric model into data that calibration unit 30F and / or path generation unit 30C can use as input to their respective operation sets. In some non-limiting examples, model analysis unit 30A may perform decomposition operations such as point cloud generation. For instance, model analysis unit 30A may generate a point cloud representing the surface topography of part 14 using the 3D geometric model extracted from geometric models 34 of data repositories 32.

[0048] By generating point clouds that represent the surface topography of part 14, model analysis unit 30A expands the functionality of UI 4 to enable support of higher-dimensional repair trajectories with respect to part 14. Moreover, these functionalities executed via model analysis unit 30A enable path generation unit 30C to project trajectories on curved surfaces, thereby populating traversal paths 40 with data that is usable by robot controller 10 even in scenarios in which part 14 has a curved (and potentially complex) surface topography. Because point clouds are composed of potentially hundreds if not thousands of numerical data points, model analysis unit 30A enables path generation unit 30C to form a traversal path that could be potentially prohibitive to define via a human-defined command. As such, model analysis unit 30A implements the techniques of this disclosure to enable interfacing in which topographical data can be extracted from a 3D geometric model and delivered to robot controller 10 in combination with process-defining input data in the form of a robotic process plan that begins from a starting point defined by the calibrated pose information formed by calibration unit 3 OF and stored to pose information 38.

[0049] In some examples, the user input received via UI 4 to define the robotic process plan of robot arm 12 and end effector 16 with respect to part 14 may include one or more tracing inputs. For instance, computing device 6 may implement UI 4 in a way that enables a user to trace aspects of the robotic process plan onto a visual rendering of the 3D geometric model via graphical interactive capabilities. In these examples, processors 22 may invoke ray tracing unit 30E. Ray tracing unit 30E, upon execution, may run processes that enable interpret the ray tracing inputs provide with respect to the 3D geometric model into aspects of the robotic process plan to be provided to robot controller 10. In this way, ray tracing unit 30E enables point & click type interactions with geometric models 34 via UI 4. For example, ray tracing unit 30E may enable a user to operate UI 4 to point and click on a rendering of any of geometric models 34 and indicate directly on the rendering of the 3D geometric model at which a particular robotic operation should take place. In one nonlimiting use-case scenario, ray tracing unit 30E may process user input that indicates where on an auto part (e.g., a car hood) that robot arm 12 and end effector 16 should perform a sanding, wiping or polishing operation. In general, software components 30 operate UI 4 in a way that enables users to import and manipulate geometric models 34 using operations such as dragging, zooming, and panning operations available via computer-aided manufacturing (CAM) and / or computer-aided design (CAD) systems. Additionally, software components 30 may operate UI 4 in a way that enables users to calibrate parts to align in space with a selected reference frame.

[0050] In conjunction, software components 30 may utilize data repositories 32 such that UI 4 is a no-code GUI that a user can operate to formulate a robotic process plan according to which robot arm 12 and end effector 16 to prepare part 14 in a variety of ways and using potentially complex sequences of operations. Through the no-code GUI of UI 4, software components 30 enable a user to calibrate pose information 38 with respect to part 14 and a tool (e.g., end effector 16) relative to the robot, to generate conformal paths on a part model of geometric models 34, to optimize a toolpath using a path planner with user-specified parameters for dynamic constraints and time-reparameterization, and to generate process parameters for applications such as paint repair, adhesive dispensing, etc. based on user inputs.

[0051] As described above, system 2 (and components thereof, such as computing device 6) implement techniques of this disclosure to enable users to interact with complex robotic systems in convenient ways. In a particular non-limiting use case, system 2 directly facilitates discovery of automated paint repair process plans implemented with respect to part 14. According to additional aspects of this disclosure, system 2 may provide users a sandbox environment for innovation by which users can quickly develop and test new ideas.

[0052] Computing device 6 may implement various testing and optimization techniques of this disclosure to provide, via UI 4, an environment in which developers and application engineers can collaborate to identify relevant process parameters of robot operation parameters 36. In some use case scenarios, many parameters may need to be introduced to provide users with fine-grained control over the configuration of a robot process plan.

[0053] As the number of input parameters increases, it becomes more difficult to ascertain which parameters are truly important, and how to efficiently tune those that are important in order to meet process demands. Furthermore, parameters may need to be tuned based on varying conditions at customer sites. For example, in the context of a robotic auto part preparation application, input parameters must be adapted on site to account for varying clear-coat formulations, variable paint cure-times, customers preferences regarding consumables, etc.

[0054] According to the testing and optimization techniques of this disclosure, computing device 6 may reduce or potentially eliminate the need for trial -and-error based optimization techniques. According to trial-and-error based optimization techniques, users must load processes one by one and edit each process manually. After editing, users must compile each plan, deliver each compiled plan to the associated process controller, and then trigger a ‘run’ command. This process is very manual in nature, and thus slow, cumbersome, and error prone. As the burden of generating data this way is relatively heavy, process experimentation is slowed, and as a result, many processes remain suboptimal.

[0055] Computing device 6 may implement the testing and optimization techniques to improve the efficiency and data precision of these optimization operations. By enabling automation of complex testing workflows, computing device 6 may implement the testing and optimization techniques of this disclosure to reduce resource expenditure by reducing the number of manual iterations in trial-and-error implementations, and to improve data precision by reducing the error-proneness of trial-and-error implementations.

[0056] Computing device 6 may invoke optimization unit 3 OB to deliver automated solutions that alleviate the burden of generating data for process optimization via execution of a testing suite framework. The framework provided by optimization unit 30B enables developers to use UI 4 to define configurable, application-specific tests, and also enables users to configure and run those tests in a streamlined fashion.

[0057] For instance, optimization unit 30B may ingest an operation sequence that a user provides via UI 4 in a single-click fashion. In turn, optimization unit 30B may save the operation sequence (or “workflow”) to workflows 42 of data repositories 32. One example of a five-step workflow that optimization unit 30B may ingest via UI 4 and store to workflows 42 includes the following five steps: (i) load the desired part calibration; (ii) perform a sampling routine at a specified location on a surface of the part; (iii) compile a plan using configured process parameters and sample information from (ii); (iv) deliver the compiled plan to robot controller 10; and (v) trigger the robotic process plan to run on hardware. In this example, the particular workflow of workflows 42 is tested at the processing site, using robot controller 10. In other examples, computing device 6 may run simulation software to implement one or more of workflows 42 in a simulated environment to provide feedback via UI 4.

[0058] According to some examples of this disclosure, optimization unit 30B may enable users to configure multiple tests simultaneously by targeting input parameters to examine and finetune. According to one such example, optimization unit 30B may implement the following process: (i) configure a test with a base set of input parameters; (ii) target a subset of input parameters to vary in an experiment; (iii) target a number of points on a surface where testing should take place; (iv) automatically generate a large set of tests from user input received at steps (i)-(iii) ; and (v) in response to a single-click user input provided via UI 4, instigate all relevant tests on hardware. Again, while this particular example is implemented in the processing environment, computing device 6 may, in other examples, implement this testing and optimization process in a simulated environment.

[0059] In any of the above cases, optimization unit 30B may enable the user of computing device 6 to identify optimal or improved parameters by way of running any of workflows 42.In this way, optimization unit 3 OB improves data precision and system performance by enabling optimal parameter selection for use in a runtime scenario.

[0060] In cases in which aspects of system 2 run large sets of workflows in simulation, optimization unit 30B may implement techniques of this disclosure to enable collection of data that can be useful to help tune process parameters, even without running on hardware. For example, as described with respect to feature detection and avoidance in the context of automotive paint repair, large scale studies have also been conducted to determine how a given set of input parameters will affect vehicle coverage (where system 2 is allowed to process and where system 2 deems it to be too dangerous or otherwise inappropriate to do so), and this is done without performing any actual on-hardware tests.

[0061] In some examples, scores can be assigned to each replicate. In these examples, optimization unit 30B may implement more intensive optimization techniques to more completely automate the optimization process. These include gradient based methods, stochastic optimization methods (e.g., Monte-Carlo / Genetic Algorithms), reinforcement learning, and others. In this manner, optimization unit 30B may provide data and assign scores using a variety of algorithms in an efficient way.

[0062] As more process knowledge is gained and the variability of conditions affecting the process at customer sites is understood, the set of experiments to be run can be reduced and can be assembled for rapid in-situ process optimization. In the context of auto part processing, for example, optimization unit 30B may implement one or more of workflows 42 to truly be a customized offering. In this manner, optimization unit 30B of computing device 6 enables system 2 to provide testing capabilities (in both hardware-hosted and simulator- hosted environments) that enable parameter refinement with respect to robot operation parameters 36, thereby reducing error and improving data precision with respect to a final robotic process plan executed via robot controller 10.

[0063] FIG. 3 is a screenshot 44 illustrating a workflow setup of this disclosure. Screenshot 44 is an instance of UI 4 showing a paint repair-based use case in which a user wishes to investigate the effect of the variable “process time” by executing sixteen (16) different runs, with each run varying in time between four (4) seconds and eight (8) seconds using the functionalities provided by optimization unit 30B and workflows 42. To enable the user to set up the described workflow, optimization unit 30B may dynamically update UI 4 in response to receiving, via I / O interface 24, a series of user inputs that define a robotic process plan.

[0064] However, in accordance with the workflow setup aspects of this disclosure, optimization unit 30B may operate UI 4 in a way that enables the user to indicate via user input that theuser would like to record test parameter performance (e.g., by way of a right-click and marking ‘Record MetaData’) across the particular workflow 42. It will be appreciated that the “Record MetaData” label for the activation UI element is only one non-limiting example. In other examples consistent with this disclosure, this UI element may have a different label such as “Track as Variable” or the like. Additionally, optimization unit 30B may output a visual indication (e.g., by highlighting the parameter selected for investigation) to indicate that optimization unit 30B is tracking that particular parameter as an experiment variable. Optimization unit 3 OB may then operate UI 4 to enable the user set up a workflow of workflows 42 by defining a sequence of operations.

[0065] The user may define one or more parameters (including the parameter(s) that forms the experiment variable for the particular workflow 42) in parameter panel 46. Additionally, the user may provide a defect location and other position-based parameters via model image area 48. In this manner, optimization unit 30B enables users to operate UI 4 to experiment with different parameters and different values for a common parameter, while leveraging the tendency of many testing workflows to be similar to one another in structure. Moreover, the scalability of workflows 42 with respect to operation set size enables optimization unit 3 OB to provide greater automation capability for quality assurance (QA) purposes.

[0066] Optimization unit 3 OB utilizes user inputs to construct workflows 42 in a way that enables users to combine operations in a fully automated, end-to-end fashion, thereby obviating the need to consistently drive each process step manually. Workflows 42 can be persisted (i.e. saved and loaded) and can be executed with a single click by leveraging nonvolatile storage aspects of memory 20 that implement data repositories 32. Optimization unit 30B implements workflows 42 to accomplish the objective of helping users such as application engineers operate more efficiently.

[0067] The use of workflows 42 has been shown to yield a fourfold (4x) speedup in the QA process (in one example, reducing from four weeks to one week). Workflows 42 enable stress testing of system 2 in a quick and consistent manner. These efficiencies introduced by optimization unit 30B and workflows 42 are particularly useful when considering the rich and granular parameter control that parameter processing unit 30D and robot operation parameters 36 offer via UI 4.

[0068] For instance, a single tab of UI 4 may alone expose approximately a hundred different knobs to enable granular tuning of a wide variety of robot operation parameters 36. Additionally, in subsequently invoked tabs, users can tune knobs that vary force, rpm, and other parameters in very customized ways, thereby introducing an even greater variety of tunable dimensions. As such, the input space to computing device 6 is relatively largebecause of the rich functionality provided by software components 30 with respect to developing and maintaining data repositories 32 with code abstraction and sharing. Optimization unit 3 OB implements workflows 42 to help users simplify, identify critical process parameters, and optimize process inputs to improve the performance of system 2 at runtime.

[0069] Upon finishing the setup of one of workflows 42, optimization unit 3 OB may automatically scan all workflow inputs for variables with metadata, which represent variables previously marked for an experiment. If optimization unit 3 OB detects any metadata, optimization unit 3 OB may invoke a “Parameter Assignment” window via UI 4 in which these variables may be assigned by the user. In the particular use case shown in FIG. 3, the user may assign sixteen (16) values between four (4) second and eight (8) seconds. After receiving a ‘configure’ command, optimization unit 3 OB may automatically generate sixteen (16) workflows and may enable the user to execute sixteen (16) different runs, each with the variable slightly altered, all via a single click.

[0070] In some examples, optimization unit 30B may run the same procedure with multiple parameters marked. In this case, the mechanism of how multiple workflows are generated is controlled by user input. Currently values may be ‘zipped’ together (e,g., 8 values - 8 values -> 8 workflows), combined in matrix format (8 values - 8 values -> 64 workflows), or manually entered by the user. Other techniques that may be implemented by optimization unit 30B may incorporate randomization and other statistical approaches including experimental design (DOE).

[0071] In sum, optimization unit 30B may implement workflows 42 to help make customized applications that improve power and flexibility for the user. Workflows 42 help the user exercise that power to incorporate process parameters to optimal values identified through experimentation. Workflows 42 can also help in other areas such as algorithm evaluation and quality assurance. As such, optimization unit 30B may implement workflows 42 to provide high-level automation in system 2 and naturally find use when any user wishes to operate the system.

[0072] The automated testing environment implemented by optimization unit 30B provides several technical advantages, two non-limiting examples of which are described herein. A first example is that the techniques implemented by optimization unit 30B provides system integration testing. For instance, the system integration testing improvement provided by the techniques implemented by optimization unit 30B enable testing of a system as a whole for quality assurance purposes. As a second example, the techniques implemented by optimization unit 30B enable optimal parameter identification. Optimization unit 30Bimplements the techniques of this disclosure to enable exploration, in an automated fashion, of a space of input parameters and lock in on parameters that are determined to be optimal based on a set of application-specific metrics. These optimal parameters can then be used in a deployment environment. The optimal parameter identification enabled by optimization unit 3 OB is also significant because what is deemed optimal at one factory may not be optimal at another. As such, it is important to be able to quickly and efficiently perform input optimization in varying contexts, as can be accomplished by way of the workflowbased optimization techniques implemented by optimization unit 3 OB.

[0073] FIG. 4 is a screenshot 50 illustrating a robot path construction step of this disclosure. To operate UI 4 at the stage of screenshot 50, a user may provide an input (e.g., in the form of a click) via path construction panel 52. As some examples, the user may select one or more process inputs by working through the sidebar menus indicated by the tabs located at the top of path construction panel 52. The tabs shown in FIG. 4 are non-limiting examples of parameter classes that computing device 6 enables a user to tune by interacting with UI 4. In some examples, parameters chosen from the sidebar(s) of path construction panel affect how a target topography (e.g., a surface topography of part 14) is identified via click inputs.Examples of parameters that may affect how a target topography is identified include feature detection, surface normal offset, and others.

[0074] User input (e.g. single-click or double-click input) provided in topography panel 54 may enable the user to operate UI 4 in a way that targets the topography of the part (e.g., part 14) illustrated by a 3D mesh representing the selected 3D geometric model. As such, topography panel 54 is a portion of UI 4 that enables a user to provide a 2D input projected onto a part surface model. A user selection provided via topography panel 54 may result in a single point area where a process or process step is to occur, or an entire path along part 14 taken from a series of click inputs provided via topography panel 54.

[0075] As such, the use of UI 4 in the stage of screenshot 50 can be described by the process flow of process input design being provided to both a target topography identification step and a process planner step. The target topography identification step may provide its output also to the process planner step. In turn, the process planner step may output the robotic process plan.

[0076] FIG. 5 is a flowchart illustrating an example process 60 that system 2 may perform in accordance with aspects of this disclosure. Process 60 may begin with model analysis unit 30A importing a part model (56). For instance, model analysis unit 30A may load a 3D geometric model accessed from geometric models 34. Calibration unit 30F may calibratepart 14 (58). For instance, calibration unit 30F may calibrate a portion of pose information 38 of robot arm 12 with respect to part 14.

[0077] Computing device 6 may form an application-specific process design (62). For instance, the application-specific process design may reflect a robotic process plan to be implemented via robot controller 10, robot arm 12, and end effector 16 with respect to the surface topography of part 14. Optionally, computing device 6 may run a simulation of the application-specific process design (64). The optional nature of the simulation in process 60 is shown by way of a dashed-line border in FIG. 5.

[0078] Computing device 6 may sync the plan to a hardware controller (66). For instance, computing device 6 may communicate the robotic process plan as a command program over network 8 to robot controller 10. Robot controller 10 may execute the plan (68). For instance, robot controller 10 may operate robot arm 12 according to the command program to effectuate the robotic process plan over the surface topography of part 14.

[0079] In the present detailed description of the example embodiments, reference is made to the accompanying drawings, which illustrate specific embodiments in which the invention may be practiced. The illustrated examples are not intended to be exhaustive of all embodiments according to the invention. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present invention. The detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0080] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified in all instances by the term “about” or “approximately” or “substantially.” Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0081] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” encompass embodiments having plural referents, unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term “or” is generally employed in its sense including “and / or” unless the content clearly dictates otherwise.

[0082] It is to be recognized that depending on the example, certain acts or events of any of the methods described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the method). Moreover, in certain examples, acts or events may be performed concurrently, e.g.,through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

[0083] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors, including one or more microprocessors, CPUs, GPUs, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), processing circuitry (e.g., fixed function circuitry, programmable circuitry, or any combination of fixed function circuitry and programmable circuitry), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components. The term “processor” or “processing circuitry” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit comprising hardware may also perform one or more of the techniques of this disclosure.

[0084] Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure. In addition, any of the described units, modules or components may be implemented together or separately as discrete but interoperable logic devices. Depiction of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

[0085] The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer-readable media.

[0086] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

CLAIMS1. A method comprising: receiving input parameterizing a robot operation sequence; storing data describing the robot operation sequence; identifying a variable associated with the robot operation sequence; assigning multiple values to the variable; generating multiple workflows, wherein each respective workflow represents a respective run of the robot operation sequence using a corresponding value selected from the multiple values assigned to the variable; and executing each respective workflow of the multiple generated workflows.

2. The method of claim 1, further comprising receiving at least two respective values of the multiple values, wherein assigning the multiple values to the variable comprises assigning each of the two or more values received by the interface hardware to the variable.

3. The method of claim 1, further comprising receiving a seed value associated with the variable, wherein assigning the multiple values to the variable comprises generating two or more of the multiple values using the seed value.

4. The method of claim 3, wherein generating the two or more of the multiple values using the seed value comprising varying the seed value according to a fixed increment.

5. The method of claim 1, wherein executing each respective workflow of the multiple generated workflows comprises executing each respective workflow of the multiple generated workflows in a simulation environment.

6. The method of claim 1, wherein executing each respective workflow of the multiple generated workflows comprises: signaling each respective workflow of the multiple generated workflows as a respective robot control plan to a robot controller; and causing the robot controller to execute each respective robot control plan associated with the multiple generated workflows on hardware.

7. The method of claim 1, wherein the input parameterizing the robot operation sequence comprises at least one of parameter data associated with the robot operation sequence or a part calibration associated with the robot operation sequence.

8. The method of claim 1, wherein the variable parameter is a first variable parameter of a plurality of variable parameters associated with the robot operation sequence, the method further comprising assigning respective multiple values to each respective variable parameter of the plurality of variable parameters, and wherein each respective workflow represents a respective run of the robot operation sequence using a corresponding combination of values selected from each respective set of multiple values assigned to each respective variable parameter of the plurality of multiple variable parameters.

9. The method of claim 1, further comprising obtaining respective performance characteristics associated with the execution of each respective workflow.

10. The method of claim 9, wherein the performance characteristics associated with the execution of each respective workflow comprise one or more performance metrics associated with the execution of each respective workflow.

11. The method of claim 9, wherein the performance characteristics associated with the execution of each respective workflow comprise path information associated with the execution of each respective workflow.

12. The method of claim 9, further comprising identifying an optimal value for the variable parameter from among the multiple values assigned to the variable parameter based on the respective performance metrics associated with the execution of each respective workflow of the multiple generated workflows.

13. The method of claim 1, wherein the input describing the robot operation sequence describes one or more of a part import, a part calibration, a controller configuration import, a target area identification, a sampling, a path generation, a path syncing to a controller, or a run preparation associated with the robot operation sequence.

14. A device comprising: interface hardware configured to receive input parameterizing a robot operation sequence; a memory configured to store data describing the robot operation sequence; andprocessing circuitry communicatively coupled to the interface hardware and the memory, the processing circuitry being configured to: identify a variable associated with the robot operation sequence; assign multiple values to the variable; generate multiple workflows, wherein each respective workflow represents a respective run of the robot operation sequence using a corresponding value selected from the multiple values assigned to the variable; and execute each respective workflow of the multiple generated workflows.

15. An apparatus comprising: means for receiving input parameterizing a robot operation sequence; means for storing data describing the robot operation sequence; means for identifying a variable associated with the robot operation sequence; means for assigning multiple values to the variable; means for generating multiple workflows, wherein each respective workflow represents a respective run of the robot operation sequence using a corresponding value selected from the multiple values assigned to the variable; and means for executing each respective workflow of the multiple generated workflows.

Images