Human robot teaming device

Abstract

A human teaming device includes a robot, a 3-D scanner, a projector, and a controller operably coupled to the 3-D scanner, the projector, and the robot. The controller comprises at least one processor, and memory having instructions stored thereon to execute a plan generation process. When executed by the at least one processor, the controller activates the 3-D scanner to scan the target object, creates a 3-D design model of the target object defining a virtual target object, perform registration between the scanned target object and the 3-D design model, generate a work function plan using the 3-D design model, and provide the visualization of the work function plan using the projector by aligning and projecting the virtual target object onto the target object. A confirmation and verification step is provided and the work function is executed with the robot when the confirmation and verification step is approved.

Description

MCC Ref. No 10025-339W01HUMAN ROBOT TEAMING DEVICE CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U. S. Provisional Patent No. 63 / 790,934, filed on April 18, 2025, the contents of which are incorporated herein by reference in its entirety.FIELD OF THE DISCLOSURE

[0002] The present disclosure relates generally to a human robot teaming device.BACKGROUND

[0003] Human-robot teaming device is increasingly recognized as a critical enabler for improving productivity and safety in modem manufacturing environments. Automated manufacturing processes such as riveting and drilling are fundamental for assembling components in industries like automotive and aerospace. These operations demand extensive programming that involves precise scaling, meticulous calibration, and rigorous evaluations to maintain quality standards. The calibration and continuous adjustments required for these tasks not only demand significant expert labor but also incur high time and cost expenditures. As a result, there is a pressing need for advanced techniques that can streamline these processes and reduce the dependency on manual intervention.SUMMARY

[0004] In some aspects, the disclosure relates to a human robot teaming device configured to be positioned within a workspace having a target object. Tire human robot teaming device includes a robot having one or more robotic arms configured to perform a work function, a 3-D scanner configured to selectively scan the target object, a projector configured to project a visualization of the work function plan onto an area of the workspace, and a controller operably coupled to the 3-D scanner, the projector, and the robot. The controller includes at least one processor, and memory having instructions stored thereon to execute a plan generation process that, when executed by the at least one processor, cause the controller to: activate the 3-D scanner to scan the target object, create a 3-D design model of the target object, the 3-D design model defining a virtual target object; perform registration between the scanned target object from the 3-D scanner and the 3-D design model defining the virtual target object, generate a work function plan using the 3-D design model of the virtual target object, provide the visualization of the work function plan using the projector, wherein the visualization includesMCC Ref. No 10025-339W01aligning and projecting the virtual target object onto the target object, provide a confirmation and verification step for an operator of the device, and execute the work function with the robot when the operator approves the confirmation and verification step,

[0005] In some aspects, the disclosure relates to a plan generation process for a human robot teaming device via a controller, wherein tire controller includes at least one processor, and memory having instructions stored thereon that configured to execute the plan generation process. The plan generation process includes scanning a target object with a 3-D scanner of the human robot teaming device, creating a 3-D design model of the target object, the 3-D design model defining a virtual target object; performing registration between the scanned target object from the 3-D scanner and the 3-D design model of the virtual target object; generating a work function plan using the 3-D design model defining the virtual target object, providing a visualization of the work function plan using a projector, wherein the visualization includes aligning and projecting the virtual target object onto the target object, providing a confirmation and verification step for an operator of the device, and executing die work function with the robot when the operator approves the confirmation and verification step.

[0006] This summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the devices or processes described herein will become apparent in die detailed description set forth herein, taken in conjunction with the accompanying figures, wherein like reference numerals refer to like elements.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 is a perspective view of a human robot teaming device according to an implementation of the disclosure.

[0008] FIG. 2A is a perspective view of die human robot teaming device performing a calibration method.

[0009] FIG. 2B is a schematic view of a controller of the human robot teaming device.

[0010] FIG. 3 is a flow chart of a plan generation process of the controller of the human robot teaming device.

[0011] FIG. 4 illustrates a registration and visualization process of the human robot teaming device.MCC Ref. No 10025-339W01

[0012] FIG. 5 A is a perspective front view of a target object.

[0013] FIG. 5B is a perspective front view of the target object of FIG. 5A with a virtual 3-D target object projected onto the target object.

[0014] FIG. 5C is a perspective view of the human robot teaming device performing a work function on the target object.

[0015] FIG. 6 illustrates target objects, scanned 3-D design models of the target objects, registration of the target objects, and projection of the virtual 3-D target objects onto the target object.

[0016] FIG. 7 is a graph illustrating a learning process of a reinforcement learning-based point cloud alignment framework.

[0017] FIG. 8 is a graph illustrating alignment errors when compared to iterations when using an Iterative Closest Point (I CP) algorithm for different types of materials.

[0018] FIG. 9 is a graph illustrating alignment errors compared to training episodes from the reinforcement learning-based point cloud alignment framework from the human robot teaming device for different types of materials.

[0019] FIG. 10 is a graph illustrating registration errors using the ICP algorithm, a general and distinctive (GeDi) based deep learning framework, and the reinforcement learning-based point cloud alignment framework.

[0020] FIG. 11 is a scattered plot of error visualization between target and projected points. DETAILED DESCRIPTION

[0021] Before turning to the figures, which illustrate the exemplary’ implementations in detail, it should be understood that the present application is not limited to the details or methodology set forth in the description or illustrated in the figures. It should also be understood that the terminology is for the purpose of description only. Like reference numerals in the figures may represent and refer to the same or similar element, feature, or function.

[0022] Recent research has shown that integrating visual cues, such as projecting digital imagery on a shared teaming workspace, can significantly improve coordination andMCC Ref. No 10025-339W01interaction between human operators and robotic systems. Tire projection of a CAD model overlays onto physical objects in automated manufacturing offers a transformative solution to the limitations of static references and rigid pre-programmed instructions. Projecting virtual design information and robotic operation plans onto real-world workpieces creates a continuously shared reference framework that both human operators and robotic systems can access in real-time. Such visualization not only enhances situational awareness but also enables adjustments to ongoing tasks while reducing the need for labor-intensive programming and extensive manual calibration. This dynamic integration of virtual and physical environments facilitates more efficient task coordination, improved precision, and ultimately significant savings in labor, time, and cost. Tire approach is particularly promising for complex manufacturing sectors like automotive and aerospace, where adaptability and precision are paramount.

[0023] FIG. 1 illustrates a human robot teaming device 100 of the current disclosure. The human robot teaming device 100 includes a robot 104, a 3-D scanner 108, a projector 112, and a controller 150. In some implementations, the robot 104, the 3-D scanner, and the projector 112 may be coupled to a single structure or may be positioned in a real-world environment adjacent each other. Tire device 100 may be positioned in a real -world environment having a target object 116 positioned within a workspace of the real -world environment. In some implementations, the robot 104 may be a stable or mobile robot platform (e.g., as illustrated in FIG. 1).

[0024] Tire robot 104 further includes one or more robotic arms 120 having a tool 124a and / or a gripper 124b coupled thereto. The robot arms 120 may work in unison to grab, support, and / or perform a work function on the target object 116 (e.g., drill, cut, couple the target object to another object, or the like). In other implementations, the robot 104 may be alternatively constructed to perform the desired work function.

[0025] Tire 3-D scanner 108 may be an industrial scanner or camera with sub-millimeter accuracy. In some implementations, the 3-D scanner 108 and the projector 112 may be formed as a single component capable of performing each function, lire 3-D scanner 108 is configured to scan the target object 116 and develop a 3-D model of the target object 116 (e.g., a virtual 3-D target object 122, FIG. 4). The projector 112 may be a laser projector with low projection distortion. Low projection distortion reduces distance distortion within a projection area 128. Generally low projection distortion is accomplished through mathematical transformations thatMCC Ref. No 10025-339W01strategically distribute distortion across the projection. As described in more detail below, the projector 112 is configured to display the 3-D design model created by a software (e.g., AutoCAD, or the like), which is aligned with the target object 116 by the registration algorithm in the controller 150.

[0026] As discussed in more detail herein controller 150 is operably coupled to the 3-D scanner 108, the projector 112, and the robotic arms 120 of the robot 104. The controller 150 is configured to activate the 3-D scanner 108 to scan the target object 116. lire controller 150 is configured to employ a point-cloud registration with reinforcement learning, which enables precise alignment between the 3-D design model and the target object 116. As the 3-D design model is created, the controller 150 is configured to activate the projector 112 to display the 3- D design model on the target object 116 via the projector 112. In some implementation, the controller 150 executes a plan generation process 300 (described in detail below), which is verified by an operator 118 within the workspace.

[0027] Now with reference to FIG. 2A, prior to executing the plan generation process 300, the device 100 is calibrated. As shown in FIG. 2, the 3-D scanner 108 defines a first coordinate system and the projector 112 defines a second coordinate system that is offset from the first coordinate system. Since the coordinate systems are offset, calibration is crucial to ensure precise measurements and interactions without distortion for the projector 112. Therefore, the calibration method needs to consider the intrinsic parameters for both devices and the extrinsic parameters that define their spatial relationship. In the illustrated implementation, the calibration method includes activating the projector 112 (e.g., via the controller 150) to exhibit a rectified image 132 on a surface 136 within the human-robot workspace. The calibration method may be programed within the controller 150 of the robot 104, For example, the rectified image 132 may be a checkerboard image. Tire calibration method may be automated by the controller 150 such that settings of the projector and scanner are adjusted until the checkerboard image is not distorted (e.g., as shown in FIG. 2). In some implementations, the surface 136 may include a 2-D rectified image 132 defined thereon. In such an implementation, the 3-D scanner 108 may scan the rectified image 132 defined on the surface 136 and the projector 112 may¬ display a virtual rectified image 132 over the image on the surface 136.Controller ArchitectureMCC Ref. No 10025-339W01

[0028] Referring now to FIG. 2B, a schematic diagram of the controller 150 is shown according to an example implementation. As shown in FIG. 2B, the controller 150 includes a processing circuit 155 having a processor 160 and a memory’ device 165, and a control system 170. The controller 150 includes a user input circuit 175 structured to receive user inputs, a scanner circuit 180 structured to control the scanner 108, a projector circuit 185 structured to control the projector 112, a robot circuit 190 structured to control movement of the robotic arms 120, a registration and visualization circuit 195 structured to perform a registration and visualization process 200 (FIG. 4), a plan generation circuit 205, and a communications interface 215. Generally, tire controller 150 is structured to receive user inputs from the operator or an external device (e.g., a computer) and control the system s of the human robot teaming device 100 to scan a target object, create a 3-D design model, perform registration, generate a work function plan, and provide visualization of the work function plan.

[0029] In one configuration, the circuits of the control system 170 are in the form of machine or computer-readable media that is executable by a processor, such as processor 160. As described herein, the machine-readable media facilitates performance of certain operations to enable reception and transmission of data. For example, tire machine-readable media may- pro vide an instruction (e.g., command, etc.) to acquire data. In this regard, the machine- readable media may' include programmable logic that defines the frequency of acquisition of the data (or, transmission of tire data). The computer readable media may include code written in any programming language. The computer readable program code may be executed on one processor, multiple co located processors, multiple remote processors, or any combination of local and remote processors. Remote processors may be connected to each other through any type of network (e.g., CAN bus, etc.).

[0030] In another configuration, the circuits of the control system 170 are implemented as hardw are units, such as electronic control units. As such, the circuits of the control system 170 may be implemented as one or more circuitry' components including, but not limited to, processing circuitry', network interfaces, peripheral devices, input devices, output devices, sensors, etc. In some implementations, the circuits of the control system 170 may take the form of one or more analog circuits, electronic circuits (e.g., integrated circuits (IC), discrete circuits, system on a chip (SOCs) circuits, microcontrollers, etc.), telecommunication circuits, hybrid circuits, and any other type of “circuit.” In this regard, the circuits of the control system 170 may include any type of component for accomplishing or facilitating achievement of theMCC Ref. No 10025-339W01operations described herein. For example, a circuit as described herein may include one or more transistors, logic gates (e.g., NAND, AND, NOR, OR, XOR, NOT, XNOR, etc.), resistors, multiplexers, registers, capacitors, inductors, diodes, wiring, and so on). Hie circuits of the control system 170 may also include programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like. The circuits of the control system 170 may include one or more memory' devices for storing instructions that are executable by the processor(s) of the circuits of the control system 170. The one or more memory devices and processor(s) may have the same definition as provided below with respect to the memory device 165 and processor 160. In some hardware unit configurations, the circuits of the control system 170 may be geographically dispersed throughout separate locations. Alternatively and as shown, the circuits of the control system 170 may be implemented in or within a single unit / housing, which is shown as the controller 150.

[0031] In the example shown, the controller 150 includes the processing circuit 155 having the processor 160 and the memory’ device 165, The processing circuit 155 may be structured or configured to execute or implement the instructions, commands, and / or control processes described herein with respect to the circuits of the control system 170. The depicted configuration represents the circuits of the control system 170 as machine or computer-readable media. However, as mentioned above, this illustration is not meant to be limiting as the present disclosure contemplates other implementations where the circuits of the control system 170, or at least one circuit of the circuits of the control system 170, is configured as a hardware unit. All such combinations and variations are intended to fall within the scope of the present disclosure.

[0032] The hardware and data processing components used to implement the various processes, operations, illustrative logics, logical blocks, modules and circuits described in connection with the implementations disclosed herein (e.g., the processor 160) may be implemented or performed with a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or, any conventional processor, or state machine. A processor also may be implemented as a combination ofMCC Ref. No 10025-339W01computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, the one or more processors may be shared by multiple circuits (e.g., the circuits of the control system 170 may comprise or otherwise share the same processor which, in some example implementations, may execute instructions stored, or otherwise accessed, via different areas of memory). Alternatively or additionally, the one or more processors may be structured to perform or otherwise execute certain operations independent of one or more co-processors. In other example implementations, two or more processors may be coupled via a bus to enable independent, parallel, pipelined, or multi-threaded instruction execution. All such variations are intended to fall within the scope of the present disclosure.

[0033] The memory device 165 (e.g., memory memory unit, storage device) may include one or more devices (e.g., RAM, ROM, Flash memory, hard disk storage) for storing data and / or computer code for completing or facilitating the various processes, layers and modules described in the present disclosure. The memory device 165 may be communicably connected to the processor 160 to provide computer code or instructions to the processor 160 for executing at least some of the processes described herein. Moreover, the memory device 165 may be or include tangible, non-transient volatile memory or non-volatile memory. Accordingly, the memory device 165 may include database components, object code components, script components, or any other type of information structure for supporting the various acti vities and information structures described herein.

[0034] Tlie user input circuit 175 is structured to receive user inputs corresponding starting the scanner 108, the projector 112, and / or performing a work function with the robotic arms 120. As described herein, the user may also provide an input into the user input circuit 175 to confirm the projected image by the projector 112 is correct. This may be referred to herein as a confirmation and verification step for an operator of the device 100. The user input circuit 175 provides the specific instructions to the circuits to device to perform the functions defined herein. In some implementations, the user input circuit 175 may provide feedback to the operator related to the progress of the operation.

[0035] The scanner circuit 180 is in communication with the scanner 108 to scan a target object in response to the input provided by the user input circuit 175. The registration and visualization circuit 195 is structured to create a 3-D design model of the target object, which defines aMCC Ref. No 10025-339W01virtual target object. Further, the registration and visualization circuit 195 performs registration between the scanned target object from the 3-D scanner and the 3-D design model to ensure precise alignment between the virtual target object and the target object. The registration process is described in more detail herein.

[0036] Once the registration process is complete, the plan generating circuit 205 is structured to generate a work function plan using the 3-D design model of the virtual target object. The plan generating circuit 205 is also structured to provide a visualization of the work function plan using the projector. The visualization includes aligning and projecting the virtual target object onto the target object. Once the process is complete the operator may provide input into the user input circuit to perform the confirmation and verification step. Once the confirmation and verification step is complete, the robot circuit 190 is structured to execute the work function.

[0037] While various circuits with particular functionality are shown in FIG. 2B, it should be understood that the controller 150 may include any number of circuits for completing the functions described herein. For example, the activities and functionalities of the circuits of the control system 170 may be combined in multiple circuits or as a single circuit. Additional circuits with additional functionality may also be included. Further, the controller 150 may further control other activities beyond the scope of the present disclosure. In some implementations, the circuits described herein may include one or more processing circuits comprising one or more memory’ devices coupled to one or more processors, the one or more memory devices configured to store instructions thereon that, when executed by the one or more processors, cause the one or more processors to perform the operations performed herein and described with reference to circuits.

[0038] As mentioned above and in one configuration, the “circuits” may be implemented in a machine-readable medium for execution by various types of processors, such as the processor X of FIG. 13. An identified circuit of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions, which may, for instance, be organized as an object, procedure, or function. Nevertheless, the executables of an identified circuit need not be physically located together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the circuit and achieve the stated purpose for the circuit. Indeed, a circuit of computer readable program code may be a single instruction, or many instructions, and may even be distributed over several different codeMCC Ref. No 10025-339W01segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within circuits, and may be implemented in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, and may exist, at least partially, merely as electronic signals on a system or network.

[0039] While the term “processor” is briefly defined above, the term “processor” and “processing circuit” are meant to be broadly interpreted, In this regard and as mentioned above, the “processor” may be implemented as one or more general-purpose processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other suitable electronic data processing components structured to execute instructions provided by memory. Tire one or more processors may take the form of a single core processor, multi-core processor (e.g., a dual core processor, triple core processor, quad core processor, etc.), microprocessor, etc. In some implementations, the one or more processors may be external to the apparatus, for example the one or more processors may be a remote processor (e.g., a cloud based processor). Alternatively or additionally, the one or more processors may be internal and / or local to the apparatus. In this regard, a given circuit or components thereof may be disposed locally (e.g., as part of a local server, a local computing system, etc.) or remotely (e.g., as part of a remote server such as a cloud based server). To that end, a “circuit” as described herein may include components that are distributed across one or more locations.

[0040] Implementations within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to can or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause aMCC Ref. No 10025-339W01general purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

[0041] Now with reference to FIGS. 3-6, the plan generation process 300 performed by the controller 150 is illustrated. The plan generation method includes calibrating the device 100 (Step 310), which is described in detail above with respect to FIG. 2 A. Once the device 100 is calibrated, controller 150 executes the plan generation method and using the registration and visualization circuit 195 enters a registration and visualization process 200 (FIG. 4). During the registration process 200, scanner circuit 180 activates the 3-D scanner 108 to scan the target object 116 (Step 320). Once the target object 116 is scanned, the registration and visualization circuit 195 of the controller 150 creates a 3-D design model of the target object 116. The 3-D design model defines a virtual target object 122. In the illustrated implementation, the target object 116 is scanned on the background surface and the registration and visualization circuit 195 of the controller 150 uses background segmentation to separate the virtual target object 122 from the background surface to form the 3-D design model of the virtual target object 122 (Step 320 of registration, FIG. 4). After the virtual target object 122 is removed from the background surface, the registration and visualization circuit 195 of the controller 150 performs registration between the 3-D design model, as designed in AutoCAD or similar software, and the scanned target object from the 3-D scanner (Step 330).

[0042] During the scanning and registration steps (Steps 320 and 330), the registration and visualization circuit 195 of the controller 150 implements a reinforcement learning protocol 335 to ensure alignment between the virtual target object 122 and the target object. The reinforcement learning-based point cloud alignment framework (FIG. 6) may be implemented by the registration and visualization circuit 195 of the controller 150 to iteratively refine transformation parameters based on feedback from the environment. In some implementations, the reinforcement learning-based point cloud alignment framework simulates an alignment task by generating a target point cloud from real sensor data and a source point-cloud by applying a known or random transformation. The reinforcement learning-based point cloud alignment framework is described in more detail below.

[0043] Once the registration is completed, the plan generating circuit 205 of the controller 150 is structured to generate a work function plan using the 3-D design model of the virtual 3-D target object 122 (Step 340). In some implementations, the target object 116 may be an object that already has the desired work function performed. For example, the target object 116 mayMCC Ref. No 10025-339W01include a series of apertures, slots, or the like drilled or cut out of the target object. In other implementations, the target object 116 may be devoid of any work function (e.g., a blank piece). In such an implementation, the 3-D design model may be manipulated on an external device (e.g., a computer or the like) such that the desired apertures, slots, or the like are added to the 3-D design model. In some implementation, the user input circuit 175 of the controller 150 may include a display that allows the operator to generate the plan directly on the device 100,

[0044] In order to accurately display the 3-D design model of the target object 116, an inverse perspective projection may be used to project the virtual target object 122 onto the target object 116. To accurately project the reference virtual 3D model onto the target object 116, a pose of the virtual model may be adjusted to align with that of the target object. In some implementations, the reinforcement learning-based point cloud alignment framework is used by the controller 150 for extracting robust local 3D descriptors from point clouds to enhance registration.

[0045] Once the plan generation is completed, the projector circuit 185 of the controller 150 is structured to provide a visualization of the plan generation using the projector 112 (Step 350). During the visualization process, the 3-D design model of the virtual target object 122 projected is aligned with the target object 116 (e.g., FIG. 5B). In addition, the visualization process may provide additional animations related to the steps of the plan (e.g., the work the robot 104 will be performing) such as the drill position, diameter of the hole being drilled, and the progress of the work function (FIG. 5B). In such an implementation, the projector circuit 185 may display the animations related to the work plan onto the works space adjacent to the target object 116 (e.g., as shown in FIGS. 5B and 5C).

[0046] Once the visualization is on the target object 116, the operator 118 in the workspace performs a confirmation and verification step (Step 360) into the user input circuit 175. During the confirmation and verification step, the operator 118 can verify that the projection of the virtual 3-D target object 122 is consistent with the target object 116. Additionally, the operator 118 can verify that the proposed work function plan is correct (e.g., drill holes are in the correct location, or the like). In some implementations, a prompt may be displayed on the robot 104 and an input from the operator 118 may approve or reject the displayed work function plan. If the operator 118 rejects the displayed work function plan, the operator 118 may instruct the robot 104 to restart registration (e.g., return to Step 320). For example, the operator may provide a first input to the user input circuit 175 of the controller 150 to restart performingMCC Ref. No 10025-339W01registration between the scanned target object from the 3-D scanner 108 and the 3-D design model of the virtual target object 122 if the projection of the virtual target object 122 is not aligned with the target object 116. Alternatively, the operator may provide a second input to the user input circuit 175 of the controller 150 to instruct the robot 104 to replan the plan generation step (e.g., return to Step 340) if the position of the work function (e.g., aperture, slot, or the like) is in an incorrect position. For example, the registration and visualization circuit of the controller 150 may regenerate the work function plan using the 3-D design model of the virtual target object if the position of the work function is in an incorrect position.

[0047] If the operator approves that the projection of the virtual 3-D target object 122 is consistent with the target object 116 and the proposed work function by the robot 104 is correct, the robot 104 will execute the work function (Step 370). In some implementations, the work function plan may include multiple steps (e.g., multiple apertures, slots, or the like that will be formed on the target object 116). In such an implementation, the plan generation process 300 may return to Step 360 to allow the operator 118 to confirm and verify each step prior to the robot 104 executing the work function.

[0048] Now with reference to FIGS. 6-11, experimental data for the reinforcement learning-based point cloud alignment framework and the plan generation process 300 is illustrated. FIG.7 illustrates the performance of the reinforcement learning-based point cloud alignment framework over 500 episodes using a reward function that penalizes misalignment and excessive update magnitudes. Initially, the cumulative reward is highly negative, reflecting large errors and corrective actions. As training progresses, the reward steadily increases and converges to zero, indicating that the reinforcement learning-based point cloud alignment framework has effectively minimized the error,

[0049] In the illustrated implementation, the reinforcement learning-based point cloud alignment framework is used to determine the optimal transformation parameters that align the virtual target object 122 with the scanned point-cloud of the physical object. An example of the reinforcement learning-based point cloud alignment framework is described in detail below.

[0050] The CAD model of the target object 116 is represented as a point set S —1. The corresponding real object is captured by the 3-D scanner 108, producinga point cloud The unknown rigid-body transformation from tire CADMCC Ref. No 10025-339W01coordinate frame to the world frame is denoted by T* — j E SE (3) where R* E SO (3)and t* E B represent a rotation matrix and a translation vector, respectively. At a discrete time step t, the alignment algorithm maintains an estimated pose. For example, at a discrete time R tstep t, the alignment algorithm maintains an estimated pose Tt={.10 1.

[0051] The CAD points transformed by this estimate are expressed as St=To quantify the pose discrepancy between the estimate and the ground truth, the transformation error is defined in the Lie algebra se(3), et--- Lo,g(T*(Tt)“1)v(Equation 1) where Log (.) denotes the matrix logarithm mapping from SE(3) to se(3) and (,)v, is the vee- operator that extracts the 6-D twist coordinates. Tire scalar pose error is then computed as Et— 11 ||2. The objective of the alignment process is to drive Ettoward zero such that the surface Stbecomes geometrically coincident with P and visually consistent under projection. With these geometric definitions in place, the misalignment between the digital and physical objects can be expressed as a pose error in the Lie algebra, enabling a rigorous mathematical formulation for alignment refinement.

[0052] Accurate pose refinement requires a structured representation of the current alignment state, a principled formulation of corrective pose adjustments, and objective measures that quantify the agreement between the transformed CAD model and the physical object. Corrective pose adjustments are expressed as six-dimensional increments in the Lie algebra se(3), Afy = where wtand vtdenote rotational and translational updates,respectively. The updated transformation is obtained using the exponential map, Tt+1= exp (Afy ) Tt(Equation 2) with (. ) Ttdenoting the skew-symmetric operator. This compact representation enables smooth, incremental refinement of the pose estimate, forming the basis of the iterative alignment process. In other words, this step is part of the reinforcement learning based point cloud alignment framework.

[0053] To evaluate alignment quality, both geometric and projection-based criteria are used. The geometric consistency between the transformed CAD points Stand the scanned point cloud P is quantified using the bidirectional Chamfer distance, Dch(st, P) —2. Projection-space consistency isassessed by comparing CAD-projected edges and silhouettes to those captured from theMCC Ref. No 10025-339W01physical object. Let Etcadand Etimgdenote the projected CAD edges and observed image-space edges, respectively. The edge alignment error is defined as Ledge(t) =(u), where DT is the distance transform of the observed edges.Silhouette overlap is measured using intersection-over-union, LIoU(t) = 1 -These measures are combined into the instantaneous alignment cost LIoU= α1Dch(St, P) + α2Ledgethe overall objective over a horizon T - Thisunified formulation establishes the relationship between how it updates the estimated pose and how alignment quality is quantitatively measured, setting the foundation for the reinforcement¬ learning framework introduced in the next section.

[0054] Projecting the 3D CAD model onto a non-planar or curved physical object (e.g., a target object 116) requires a transformation that compensates for depth-induced distortions. Without such compensation, projected edges and surfaces appear warped, misaligned, or scaled inconsistently across the object. To ensure that the projected imagery remains geometrically faithful to the underlying surface, an inverse perspective projection model is incorporated into the formulation, The goal of inverse perspective projection is to compute a per-pixel correction that accounts for surface depth, enabling the projector to render a distortion-free overlay consistent with the physical geometry. A calibrated camera and a calibrated projector be represented by their intrinsic matrices Kch, Kp∈ ℝ3×3, and their relative pose may be defined by a rigid transformation

[0055] For a transformed CAD point x' = Rtxi+ tt∈ ℝ3and the camera projection is given by up~ Kcx'. To render the CAD point using the projector 112, the controller 150 must account for the surface point’s actual depth. The corresponding projector space coordinate is computed by the controller as up~ Kp(Rpcx' + tpc) (Equation 3).

[0056] Inverse perspective projection corrects distortion by explicitly incorporating depth information from the scanned point cloud P. For example, y e P denote the 3D surface point associated with projector pixel up. The corrected projection ray is expressed as ũp= Kp(Rpcy + tpc) (Equation 4), where ũpis the inverse perspective projection-corrected projector coordinate. This mapping ensures that the projected CAD model conforms to the trueMCC Ref. No 10025-339W01depth of the surface of the target object 116. This prevents scale distortion, edge warping, and alignment drift across curved or uneven regions.

[0057] The inverse perspective projection correction is incorporated into the alignment pipeline by rendering all visual cues (edges, silhouettes, and depth-based overlays) using Equation 4. This produces a projection that is geometrically consistent with the target object’s physical surface, enabling accurate visual guidance during manufacturing tasks that rely on CAD overlays. Accurate CAD-to-object alignment and distortion-free projection require explicit modeling of the errors that arise throughout the registration, calibration, and projection processes. These errors propagate through the pipeline and directly influence both 3D geometric consistency and 2D visual fidelity.

[0058] This subsection summarizes the dominant error sources and presents a single combined error model used in the remainder of the paper. Three principal classes of error affect the end- to-end alignment and projection process: a) registration error, calibration error, and projection error.

[0059] Registration error arises from discrepancies between the CAD-derived point cloud and the scanned physical object due to sensor noise, sampling density differences, partial observations, and initial alignment biases. Let pibe a CAD point and qi* its corresponding surface point from the scanner 108. Given a candidate rigid transformation (R, t), the transformed point is ĝi= Rpi+ t and the residual is= q̂i− qi*. The registration cost is

[0060] Calibration error arises from inaccuracies in the intrinsic and extrinsic parameters of the 3-D scanner 108. Let u* denote the expected projector pixel for a world point pj' 'orldunder ideal calibration. Using intrinsics Kprojand extrinsics Rproj, tprojthe projected pixel is uj= Π(Kproj, Rproj, tproj, pjworld) with projection operator II( ■). The calibration cost is Ecal—

[0061] Projection error persists even under accurate calibration when projecting onto curved or non-planar surfaces. After applying inverse perspective projection, corrected projector coordinates Uj are obtained and the residual distortion is rj= ũj− uj*. The projection cost isMCC Ref. No 10025-339W01

[0062] The unified error is expressed as Etotal= wregEreg+ wcalEcal+ wprojEproj(Equation 5) where wreg, wcal, wproj> 0 balance geometric alignment, parameter calibration, and projection fidelity.

[0063] A unified computational framework enables accurate, distortion-free projection of complex 3D CAD models onto real physical components in collaborative manufacturing environments. The framework integrates three processes that are often treated separately in prior work: 3D point-cloud registration, camera-projector calibration, and depth-aware projection modeling. These components are combined into a coherent pipeline capable of producing spatially consistent overlays on non-planar surfaces. Unlike conventional visual guidance systems that rely on simple geometric primitives or assume planar projection geometry, the proposed approach explicitly models the geometric relationship between CAD representations, scanned object surfaces, and projector optics, and incorporates inverse perspective projection to correct perpixel depth distortion. This systematic integration allows the system to maintain geometric fidelity under variations in object shape, surface curvature, and viewing angle, enabling the projection of rich visual content such as full CAD models, depth-color maps, and manufacturing plans. The resulting methodology addresses a critical gap in projection-assisted manufacturing by providing a robust and generalizable solution for aligning digital and physical geometry’ with the precision required for human-robot collaborative tasks. Fig. 2 illustrates the complete CAD-to-object overlay pipeline used in this work. A 3D depth sensor captures the physical component and background removal yields a clean target point cloud.

[0064] In parallel, a source point cloud is generated from the CAD model. An initial rigid alignment is computed using TCP to obtain a reasonable starting pose, after which a reinforcement learning policy performs iterative pose refinement by applying incremental translation and rotation updates. Finally, the refined pose is used to render the CAD model through the calibrated camera- projector pair, and inverse perspective projection compensates for depth-induced distortion so that the overlay remains spatially consistent even on non-planar surfaces.

[0065] The alignment task introduced herein is formalized as a sequential decision-making problem under uncertainty. This section below details the reinforcement learning (RL)MCC Ref. No 10025-339W01framework used to learn an optimal policy for performing adaptive and robust CAD-to-object alignment in the presence of sensing noise, occlusions, and calibration drift.

[0066] The environment may be modeled as a Markov Decision Process (MDP) M = (O, A, T, r, γ) where 0 denotes the observation space, cA the continuous action space, T the transition dynamics, r the scalar reward, and y e (0, 1) the discount factor. At each time step t, the controller 150 receives an observation ote: 0, selects an action at∈∈ A according to its stochastic policy πθ(at|ot), and updates the estimated pose through equation 2.

[0067] The observation vector encodes geometric and visual information that characterizes the current alignment configuration, ot= φ(St, P, Et, St) where (. ) is a feature encoder that combines geometric distances, surface-normal statistics, projected-edge consistency, and silhouette overlap. These features provide the necessary cues for evaluating how well the transformed CAD model matches the scanned object. The reward encourages improvements in alignment quality, rt=(Equation 6). The controller 150 seeks to maximize the expected discounted return, J(π) =≤ C̄ where ct≥ 0 represents instantaneous safety costs (e.g., joint limit or collision violation).

[0068] At each iteration, the current pose estimate Ttis applied to the CAD model to obtain St. The transformed model is rendered using the calibrated camera-projector geometry defined herein. For example, the forward projector mapping (equation 3) generates nominal projection coordinates, which are corrected using the inverse mapping (equation 4) to account for surface depth. From the rendered CAD model, projected edge maps E_t^cad and silhouette masks S_t^cad are extracted. These visual features are compared with observed image-space edges and silhouettes captured from the physical object. The resulting projection space discrepancies contribute to the alignment cost Lt, which directly determines the reward signal in (equation 6). Thus, projection is not only a visualization mechanism but also an active feedback component within the pose refinement loop. FIG. 7 illustrates the number of episodes for a discounted reward of the reinforcement learning-based point cloud alignment framework to reach a desired amount.

[0069] For example, FIG. 7 illustrates tire reinforcement learning agent s performance over 500 episodes using a reward function that penalizes misalignment and excessive update magnitudes. The episode reward is initially strongly negative because both registration residuals and action magnitudes are large at the beginning of training. Over successiveMCC Ref. No 10025-339W01episodes, the reward increases toward zero as the agent learns to reduce alignment cost while producing smaller, more stable pose updates. The rolling average highlights the overall improvement trend despite the high-variance fluctuations that arise from randomized initial misalignment, partial observations, and stochastic policy exploration. This behavior indicates that the learned policy progressively stabilizes the refinement process and converges to a regime with low residual misalignment and minimal corrective motion.

[0070] FIG. 8 illustrates the number of iterations of an Iterative Closest Point (ICP) algorithm to reach an error threshold of 4.0 mm for the objects shown in FIG. 6. As shown in FIG. 8, the ICP algorithm requires approximately 100 iterations to reach the error threshold of 6.72 ± 2.43 mm on average for different types of materials (e.g., plastic, ceramic, wood, and metal),

[0071] FIG. 9 illustrates the number of iterations for the reinforcement learning-based point cloud alignment framework to reach a sub-millimeter value. As shown in FIG. 9, the reinforcement learning-based point cloud alignment framework requires approximately 30 iterations to reach the error threshold of the ICP algorithm. Additionally, after 100 iterations, the error threshold for the target objects shown in FIG. 6 are between 0.70 ± 0.23 mm on average. As such, the experimental results demonstrate that the reinforcement learning-based framework significantly improves the fidelity of the digital-to-physical projection and significantly improves point-cloud registration performance, particularly for small objects.

[0072] The error comparison of the reinforcement learning -based point cloud alignment framework is graphically illustrated in FIG. 10. In particular, the reinforcement learning -based point cloud alignment framework executed by the controller 150 has errors of 0.61 mm, 0.92 mm, 1.92 mm, 0.69 mm for four different objects tested. Each value was compared to the ICP algorithm and a general and distinctive (GeDi). In each case, the reinforcement learning-based point cloud alignment framework executed by the controller 150 significantly improves the alignment of the respective objects.

[0073] As such the reinforcement learning-based registration error remains consistently below the ICP error threshold, demonstrating the effectiveness of the refinement process. The qualitative presentation of the point-cloud registration is illustrated in the point-cloud registration part (middle) of FIG. 6. In the visualizations, the red point clouds denote the source generated from the 3D CAD model, whereas the green point clouds represent the target scanned using the 3D depth sensor. The visual differences between ICP-based and reinforcementMCC Ref. No 10025-339W01learning-based point-cloud registration are evident. The experimental results demonstrate that the reinforcement learning approach achieved superior alignment between the target and source point clouds.

[0074] Now with reference to FIG. 11, a scattered plot of error visualization between target and projected points is illustrated. To determine a quantitative assessment of the accuracy of the plan generation process 300 for point projections, random points on the target were selected. The random points were assumed as target points and were projected onto different types of surfaces. The scatter plot illustrates 40 random (x,y) coordinates of the target points. The coordinates of the projected points were computed and projected target onto different types of surfaces. The resulting RAISE error was 4.94 mm, the average error was 4.76 mm, and the percentage of error was 0.68%.

[0075] Although the figures show a specific order of method steps, the order of the steps may differ from what is depicted. It is to be understood that the methods and systems are not limited to specific synthetic methods, specific components, or to particular compositions. It is also to be understood that the terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting.

[0076] As used in the specification and the appended claims, the singular forms “a,” “an” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” or “substantially” one particular value, and / or to “about” another particular value. When such a range is expressed, another implementation includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by use of the antecedent “about,” it will be understood that the particular value forms another implementation. It will be further understood that the endpoints of each of the ranges are significant both in relation to the other endpoint, and independently of the other endpoint.

[0077] Throughout the description and claims of this specification, the word “comprise” and variations of the word, such as “comprising” and “comprises,” means “including but not limited to,” and is not intended to exclude, for example, other additives, components, integers or steps. “Exemplary” means “an example of’ and is not intended to convey an indication of a preferred or ideal implementation. “Such as” is not used in a restrictive sense, but for explanatory purposes.MCC Ref. No 10025-339W01

[0078] Disclosed are components that can be used to perform the disclosed methods and systems. These and other components are disclosed herein, and it is understood that when combinations, subsets, interactions, groups, etc. of these components are disclosed that while specific reference of each various individual and collective combinations and permutation of these may not be explicitly disclosed, each is specifically contemplated and described herein, for all methods and systems. This applies to all aspects of this application including, but not limited to, steps in disclosed methods. Thus, if there are a variety of additional steps that can be performed it is understood that each of these additional steps can be performed with any specific implementation or combination of implementations of the disclosed methods.

[0079] It is important to note that the construction and arrangement of the device as shown in the various exemplary implementations is illustrative only. Additionally, any element disclosed in one implementation may be incorporated or utilized with any other implementation disclosed herein. Although only one example of an element from one implementation that can be incorporated or utilized in another implementation has been described above, it should be appreciated that other elements of the various implementations may be incorporated or utilized with any of the other implementations disclosed herein.

Claims

MCC Ref. No 10025-339W01WHAT IS CLAIMED IS:

1. A human robot teaming device configured to be positioned within a workspace having a target object, the human robot teaming device comprising:a robot having one or more robotic arms configured to perform a work function; a 3-D scanner configured to selectively scan the target object;a projector configured to project a visualization of the work function plan onto an area of the workspace; anda controller operably coupled to the 3-D scanner, the projector, and the robot, the controller comprises at least one processor, and memory having instructions stored thereon to execute a plan generation process that, when executed by the at least one processor, cause the controller to:activate the 3-D scanner to scan the target object,create a 3-D design model of the target object, the 3-D design model defining a virtual target object;perform registration between the scanned target object from the 3-D scanner and the 3-D design model defining the virtual target object,generate a work function plan using the 3-D design model of the virtual target object,provide the visualization of the work function plan using the projector, wherein the visualization includes aligning and projecting the virtual target object onto the target object, provide a confirmation and verification step for an operator of the device, and execute the work function with the robot when the operator approves the confirmation and verification step.

2. The human robot teaming device of claim 1, wherein the memory is further configured to store instructions thereon that, when executed by the at least one processor, cause the at least one processor to: implement a reinforcement learning-based point cloud alignment framework, and wherein the reinforcement learning-based point cloud alignment framework uses an iterative process of adjusting transformation parameters based on feedback from the workspace.

3. The human robot teaming device of claim 2, wherein the transformation parameters include a rotation matrix Rtand a translation vector tt.MCC Ref. No 10025-339W014, The human robot teaming device of claim 3, wherein the target object is scanned on a background surface, and wherein the memory is further configured to store instructions thereon that, when executed by the at least one processor, cause the at least one processor to use background segmentation to separate the virtual target object from the background surface to form the 3-D design model of the virtual target object.5, The human robot teaming device of claim 4, wherein after the virtual target object is removed from the background surface, wherein the memory is further configured to store instructions thereon that, when executed by the at least one processor, cause the at least one processor to perform registration between the 3-D design model of the virtual object and the scanned target object from the 3-D scanner.6, The human robot teaming device of claim 1, wherein the memory’ is further configured to store instructions thereon that, when executed by the at least one processor, cause the at least one processor to display an inverse perspective projection to project the virtual target object onto the target object.

7. The human robot teaming device of claim 1, wherein the memory' is further configured to store instructions thereon that, when executed by’ the at least one processor, cause the at least one processor to display animations related to steps of the work function plan, and w herein tire animations include one or more of a drill position, a diameter of a hole being drilled during the work function, or a progress of the work function.

8. The human robot teaming device of claim 1, wherein during the confirmation and verification step, the operator verifies that the projection of the virtual 3-D target object is consistent with the target object and the displayed work function plan is correct.

9. The human robot teaming device of claim 1 wherein the memory is further configured to store instructions thereon that, when executed by the at least one processor, cause the at least one processor to display a prompt on the robot, and wherein an input from the operatorMCC Ref. No 10025-339W01is provided to the controller to approve or reject the displayed work function plan during the confirmation and verification step.

10. The human robot teaming device of claim 9, wherein the operator provides a first input to instruct tire controller to restart performing registration between the scanned target object from the 3-D scanner and the 3-D design model of the virtual target object if the projection of the virtual target object is not aligned with the target object.

11. The human robot teaming device of claim 10, wherein the operator provides a second input to regenerate the work function plan using the 3-D design model of the virtual target object if the position of the work function is in an incorrect position.

12. The human robot teaming device of claim 1, wherein the controller includes a user input circuit structured to receive user inputs, a scanner circuit structured to control tire 3-D scanner, a projector circuit structured to control the projector, a robot circuit structured to control movement of the robotic arms, a registration and visualization circuit structured to perform the registration between the scanned target object from the 3-D scanner and the 3-D design model and provide the visualization of the w ork function plan, a plan generation circuit, and a communications interface,13. A plan generation process for a human robot teaming device via a controller, wherein the controller comprises at least one processor, and memory having instructions stored thereon that configured to execute the plan generation process, the plan generation process comprising:scanning a target object with a 3-D scanner of the human robot teaming device; creating a 3-D design model of the target object, the 3-D design model defining a virtual target object;performing registration between the scanned target object from the 3-D scanner and the 3-D design model of the virtual target object;generating a w ork function plan using the 3-D design model defining the virtual target object,providing a visualization of the work function plan using a projector, wherein the visualization includes aligning and projecting the virtual target object onto the target object, providing a confirmation and verification step for an operator of the device, andMCC Ref. No 10025-339W01executing the work function with the robot when the operator approves the confirmation and verification step.

14. The plan generation process of claim 13, further comprising implementing a reinforcement learning-based point cloud alignment framework, and wherein the reinforcement learning-based point cloud alignment framework uses an iterative process of adjusting transformation parameters based on feedback from a workspace.

15. The plan generation process of claim 13, further comprising displaying an inverse perspective projection to project the virtual target object onto the target object.

16. The plan generation process of claim 13, further comprising displaying animations related to steps of the work function plan, and wherein the animations include one or more of a drill position, a diameter of a hole being drilled during the work function, or a progress of the work function.

17. The plan generation process of claim 13, wherein the controller includes a user input circuit structured to receive user inputs, a scanner circuit structured to control the 3-D scanner, a projector circuit structured to control the projector, a robot circuit structured to control movement of the robot, a registration and visualization circuit structured to perform the registration between the scanned target object from the 3-D scanner and the 3-D design model and provide the visualization of the work function plan, a plan generation circuit, and a communications interface,18. The plan generation process of claim 17, wherein the operator provides an input to the user input circuit of the controller to approve or reject the displayed work function plan during the confirmation and verification step.

19. The plan generation process of claim 17, wherein the operator provides a first input to the user input circuit of the controller to restart performing registration between the scanned target object from the 3-D scanner and the 3-D design model of the virtual target object if the projection of the virtual target object is not aligned with the target object.MCC Ref. No 10025-339W0120. The plan generation process of claim 17, wherein the operator provides a second input to the user input circuit of the controller to regenerate the work function plan using the 3-D design model of the virtual target object if a position of the work function is in an incorrect position.

Images