Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

544 results about "Robot controller" patented technology

Method and device for speech-supplemented kinesthetic robot programming

A method of programming an industrial robot (100), composed of a robot manipulator (110) and a robot controller (120), comprises: recording movements of the robot manipulator during a kinesthetic programming session, for thereby obtaining a robot trajectory; capturing speech data while recording the movements of the robot manipulator; decomposing the recorded speech data into a plurality of phases; for each phase of the speech data, parsing the speech data into robot parameter values using a natural-language model (170), identifying a contemporaneous phase of the robot trajectory and annotating it with the robot parameter values; and, on the basis of the annotated robot trajectory, generating a robot program executable by the robot controller.
Owner:ABB (SCHWEIZ) AG

High-performance parallel robot controller based on arm + fpga architecture

The invention belongs to the technical field of parallel robot controllers, and discloses a high-performance parallel robot controller based on an arm + fpga architecture, through deep heterogeneous fusion of an ARM and an FPGA, the control period is shortened to be within 10 microseconds, the trajectory tracking error is controlled to be 0.1 mm or below, and the performance bottleneck of a traditional architecture in a high-speed scene is solved; the multi-core ARM undertakes complex tasks such as global trajectory planning and dynamics solution, and realizes parallel processing by means of an NEON instruction set; the FPGA fully releases the hardware parallel characteristic of the FPGA, real-time tasks such as multi-axis motion control and sensor data fusion are synchronously completed through a distributed logic unit, and a complex decision-real-time execution assembly line cooperation mode is formed. Inertial parameters and load changes of the mechanical arm are estimated in real time through an LSTM neural network, and feedforward compensation is carried out on interference such as mechanical vibration and load abrupt change in combination with an extended state observer achieved through FPGA hardware; the innovatively designed double closed-loop control architecture supports seamless switching between a force control mode and a position control mode.
Owner:SHENZHEN YIYUE INTELLIGENT TECH CO LTD

Intelligent control method and system for industrial robot welding

The invention relates to the technical field of industrial automatic welding and discloses an intelligent control method and system for industrial robot welding, and the method comprises the steps that after a welding task is started, an initial welding process parameter model is called based on the current workpiece specification, and a first welding control instruction sequence is generated; according to the first welding control instruction sequence, a welding gun is controlled to execute welding operation, and multi-source sensing data flow in the welding process is synchronously collected; carrying out fusion feature extraction on the real-time sensing data stream to generate a current welding situation feature vector, when situation deviation exists between the current welding situation feature vector and the reference welding situation feature vector, obtaining a self-adaptive welding process parameter model through online incremental learning, and obtaining a self-adaptive welding process parameter model on the basis of the self-adaptive welding process parameter model. And generating an optimized welding control instruction sequence and issuing the optimized welding control instruction sequence to a robot controller and a welding gun so as to execute self-adaptive welding operation. The welding quality stability and the production efficiency can be improved, and the equipment maintenance cost is reduced.
Owner:山东星辉液压设备有限公司

Space robot finite time anti-interference trajectory tracking control method based on performance constraint and actuator saturation

The invention discloses a space robot finite time anti-interference trajectory tracking control method based on performance constraint and actuator saturation, and the method comprises the steps: firstly solving a state-space equation of a flexible joint free flight space robot, constructing a system error vector in combination with a finite time instruction filter and an anti-saturation auxiliary system, and carrying out the finite time instruction filter and anti-saturation auxiliary system; a novel asymmetric logarithmic obstacle Lyapunov function with an asymmetric fixed time constraint boundary is constructed; and finally, a novel finite time advanced activation anti-saturation auxiliary system and a finite time adaptive controller with a performance constraint effect are constructed, so that finite time anti-interference trajectory tracking control of the space robot is realized. According to the method, the influence of sudden disturbance is considered in the design of the flexible joint free flight space robot controller for the first time, and under the limitation of the saturation performance constraint of the actuator, the trajectory tracking rapidity and accuracy of the space robot are ensured; and the influence of sudden disturbance in a space robot system on the trajectory tracking transient performance can be effectively inhibited.
Owner:NANJING UNIV OF SCI & TECH

Method for planning welding parameters of multiple layers of welding seams according to welding seam feature information

The invention belongs to the technical field of fusion welding, and particularly relates to a method for planning multi-layer and multi-welding-seam welding parameters according to welding seam feature information, which comprises the following steps: 1) laser emitted by a 3D visual sensor is perpendicular to a welding seam and scans along the direction of the welding seam, and the 3D visual sensor scans welding seam data and transmits the data to a robot controller in real time; (2) the robot controller processes the weld joint data, the whole weld joint is subjected to multi-layer and multi-pass planning calculation, and the number of welding layers and the number of welding passes of each layer are planned; (3) the robot controller calculates the layer number and the channel number of a welding channel needing to be welded currently, a welding gun track coordinate set on a welding path and welding parameters corresponding to welding path points; and (4) the robot controller adjusts the welding track and the welding parameters in real time according to the step (1) to the step (3) in the current welding bead welding process, so that the purpose of multi-layer and multi-pass planning self-adaptive welding of the medium-thickness plate is achieved.
Owner:SHENYANG SIASUN ROBOT & AUTOMATION

Method and device for demonstration-based robot programming with adaptive reference frames

A method of programming an industrial robot (100), with a robot manipulator (110) and robot controller (120), comprises: recording movements of the robot manipulator during a demonstration-based robot programming session, for thereby obtaining a robot trajectory; acquiring a video of the robot programming session; and generating, on the basis of the robot trajectory and the video, a robot program (C) executable by the robot controller, wherein the robot program includes at least one motion command which is expressed in a first reference frame (0 1). The method further comprises capturing operator input data indicating a first object (151) in an image in the video. The generated robot program includes a command to identify a position of an object resembling the first object in a work area (150) of a robot which executes the robot program; and the first reference frame is defined with respect to the identified position of the object resembling the first object.
Owner:ABB (SCHWEIZ) AG

Method, device and system for monitoring suction, scrabbling and falling-off of cleaning robot

The invention belongs to the field of computer vision, and particularly relates to a method, a device and a system for monitoring suction-scrabbling falling of a cleaning robot, and the method comprises the following steps: 1) a rear-view camera arranged on the cleaning robot collects suction-scrabbling images, and carries out data labeling; 2) building a deep learning labeling and training environment in a server or a workstation, selecting a YOLOv8 target detection network model, sending labeled suction and scrabbling image data into the model for training, and performing format conversion and quantization on the model after the training is completed; 3) inputting an image acquired in real time into the trained target detection model, and judging whether the suction rake falls off or not; (4) if the suction scrabbling device falls off, the upper computer sends falling abnormal information to a robot controller of the cleaning robot, and the robot controller controls a hub motor to brake emergently; and meanwhile, the upper computer transmits the abnormal information of the suction-scrabbling falling and the position coordinate information of the robot in the global map to the operation and maintenance personnel through the cloud platform and then transmits the abnormal result and the position information of the suction-scrabbling falling to the operation and maintenance personnel.
Owner:SHENYANG XINSONG DIANSHI TECH CO LTD

System and method for robot planning using large language models

A robotic controller for controlling a robot according to a sequence of robotic actions. comprises an input interface configured to receive a plurality of multimodal inputs each specifying instructions for performing a task in a different modality including audio, video, and a text modality. The controller also comprises a multimodal large language model, an action sequence decoder, and a controller. The multimodal LLM includes a multimodal LLM encoder and an LLM decoder. The multimodal LLM encoder is trained with machine learning to transform the multimodal instructions into encodings and the LLM decoder is configured to decode the encodings into a sequence of robotic instructions. The action sequence decoder is trained with machine learning to transform the sequence of robotic instructions into a sequence of actions using a library of robotic skills. The controller is configured to control a robot according to the sequence of actions.
Owner:MITSUBISHI ELECTRIC RESEARCH LABORATORIES INC

Track obstacle real-time detecting and braking system based on multi-sensor redundancy

The invention relates to the field of rail traffic intelligent control, and discloses a rail obstacle real-time detection and braking system based on multi-sensor redundancy, which comprises at least two groups of sensing modules arranged at the front end and the rear end of a train, and each group of sensing module comprises a laser radar used for collecting three-dimensional point cloud data of a forward or backward rail area of the train; the binocular camera is used for acquiring color image data and depth image data in corresponding directions; and the robot controller is in communication connection with the sensing module and is used for receiving sensing data acquired by the laser radar and the binocular camera and executing obstacle recognition, redundant information fusion and brake control logic. By constructing a multi-sensor redundancy sensing structure based on the laser radar and the binocular camera, high-robustness recognition of a dynamic or static obstacle target in a complex orbit environment is realized, and the effect of remarkably improving the obstacle detection accuracy and the false alarm suppression capability is achieved.
Owner:RIZHAO PORT GRP CO LTD +1

Servo press control system and method based on time sensitive network

The invention provides a servo press control system and method based on a time-sensitive network, and relates to the technical field of intelligent manufacturing and industrial automation control, and the system comprises a time-sensitive network TSN which is used for providing global time synchronization and deterministic data transmission; a main controller of the multi-axis servo control unit is used for issuing a motion control instruction and a feeding and discharging control instruction through a TSN. A robot controller in the robot feeding and discharging unit is used for driving an industrial robot to execute plate taking and placing operation under the action of the feeding and discharging control instruction. The visual monitoring unit comprises an industrial camera accessed to the TSN and is used for collecting images and transmitting image data to the edge computing node through the TSN; the edge computing node is used for processing the image data to generate a compensation instruction and feeding back the compensation instruction through the TSN; and the upper monitoring system is used for subscribing real-time data published by the edge computing nodes, so that the control range of the servo pressure control system is expanded, and the convenience of data transmission and interaction is improved.
Owner:NINGBO AOMATE HIGH PRECISION STAMPING MASCH TOOL CO LTD

Input shaping control of a robot arm in different reference spaces

A robot controller for controlling a robot arm includes a first space shaping module configured to provide a shaped first space target motion by convolving a first space target motion with an impulse train, where the first space target motion defines a target motion in a first reference space; a second space shaping module configured to provide a shaped second space target motion by convolving a second target motion with the impulse train; where the second target motion defines the target motion in a second reference space; and a motor controller module to generate motor control signals to the joint motors based on the shaped first space target motion and the shaped second space target motion.
Owner:UNIVERSAL ROBOT

Robot multi-mode interaction control method and related device

The invention discloses a robot multi-mode interaction control method and a related device, and the method comprises the steps that a robot controller obtains user interaction information, and the user interaction information comprises user voice information and / or user action information; analyzing the user interaction information, and determining a shopping guide service type and a target vehicle part; acquiring first component information of the target vehicle component from a preset vehicle knowledge base according to the shopping guide service type; according to the shopping guide service type and the first component information, a target action sequence is determined from a preset action library, the target action sequence comprises joint actions and voice actions, the voice actions are used for outputting voice shopping guide information, and the voice shopping guide information is determined according to the first component information; and sending a first control instruction to the target robot, wherein the first control instruction is used for indicating the target robot to execute the target action sequence. According to the invention, the functionality and intelligence of the humanoid robot as an intelligent shopping guide robot can be improved.
Owner:SHANGHAI FOURIER INTELLIGENCE CO LTD

Distributed robot controller

A distributed robot controller is disclosed, comprising a plurality of local controllers, each associated with a corresponding motor included in a plurality of motors comprising a robot; and a robot level controller coupled communicatively with each of the local controllers and configured to determine a plan to operate the plurality of motors to cause the robot to perform a task and send to each of the local controllers included in the plurality of local controllers at least a set of one or more torques associated with the motor with which that local controller is associated.
Owner:DEXTERITY INC

Robotic inference and control systems

A variety of smart inference systems and robotic control devices are disclosed. A device networking system interconnects mobile devices, robotic devices, and others. A robotic controller secures and manipulates a controlled device user interface. A smart device system interprets data based on a plurality of models operating on a plurality of runtimes based on a plurality7 of capabilities. A smart payment processor system assigns payments accounts automatically as a function of a semantic matching between one or more inferred semantic identities. A robotic emulation device captures and analyzes a video signal from a target device and transmits manipulation signals emulating a peripheral input device. A system of carts are physically coupled in a charging configuration. A conveyor is configured to couple with another conveyor and having sensors to detect weight, size, location or other aspects of conveyed items, and implementing semantic analysis for management and augmentation.
Owner:LUCOMM TECHNOLOGIES INC

Robot system and a method for monitoring a robot system

The invention relates to a method for monitoring a robot system comprising a robot arm and a peripheral device, the method comprising the steps of: providing a communicative peripheral connection between the peripheral device and a robot controller; establishing an operation signal history in a digital storage wherein the operation signal history is based on operation representations; executing a robot operation process on the robot controller; establishing a peripheral signal associated with the peripheral connection; recording the peripheral signal to obtain a peripheral signal representation; updating the operation signal history by providing the peripheral signal representation as an operation representation of the operation representations; and tracking operation of the robot system based on the operation signal history. The invention further relates to a robot system.
Owner:UNIVERSAL ROBOT

Method and device for determining trajectory data, robot controller and storage medium

The embodiment of the invention discloses a method and device for determining trajectory data, a robot controller and a storage medium, and belongs to the technical field of computers. The method comprises the steps that in the process that a robot executes a target action on an object, action data of the robot are obtained, and the action data represent the state of the robot in the process that the robot executes the target action on the object; processing the action data through a trajectory prediction model to obtain trajectory data of the object; according to the method and the device, the trajectory of the object can be immediately predicted after the robot executes the target action on the object, and the condition that the object is predicted after being shot for a period of time through a motion capture sensor is not needed, so that the consumed time is saved, and the efficiency is improved; and the accuracy of the predicted trajectory data is ensured.
Owner:TENCENT TECHNOLOGY (SHENZHEN) CO LTD

A multi-arm mobile robot control system and a control method thereof

The application relates to a multi-arm mobile robot control system, which comprises a robot, each joint position of the mechanical arm of which is configured with a joint controller for driving joint movement; a robot controller for realizing the operation of joint motors according to control quantities; a joint position hard analysis unit for calculating the coordinate positions of each joint or mechanical arm in a space coordinate system to output results; and a joint space point position trajectory constraint model for forming a correction quantity of a joint and superimposing the correction quantity on an original control quantity of the joint to become a final control quantity for controlling the joint, the final control quantity being used for output to the joint controller to realize joint movement. The application provides a multi-arm mobile robot control system, which effectively realizes the cooperative control among multi-arm robots, avoids the mutual collision among mechanical arms, improves the avoidance and dynamic response speed of the mechanical arms, and improves the agility of robot actions.
Owner:YAOSHI ROBOTICS (SHANGHAI) CO LTD +1

Node-edge symbolic consent kernel for real-time ethical computation and verified human intent execution

A node-edge symbolic consent kernel (NESCK) provides a computing architecture in which every instruction is gated by a verifiable human-intent signal and an ethical-predicate chain prior to execution. The system integrates a biometric-sensing front-end (EEG / GSR / facial micro-affect), a symbolic arbitration engine that transforms bio-intent data into consent tokens, and a cryptographically bonded node-edge ledger that records execution lineage, revocation, and audit proofs. Each node represents an executable state bound to a human consent fingerprint, while each edge encodes the ethical transition rules authorizing propagation through the network. At runtime, the kernel evaluates symbolic predicates, verifies zero-knowledge proofs of consent, and allows or halts instruction dispatch. The framework operates across devices, edge nodes, and cloud layers, enabling real-time lawful AI behavior, revocable autonomy, and tamper-proof moral audit trails. Embodiments span neuroadaptive wearables, autonomous vehicles, robotics controllers, and sovereign AI systems requiring continuous consent and transparent accountability.
Owner:ODEH SAMUEL

Method and system for determining elastic properties of industrial robot

A method and system (200) for determining an elastic characteristic of an industrial robot (100) including a plurality of interconnecting shafts including at least one shaft exhibiting elasticity and at least one joint exhibiting static friction. The method comprises: (S12) clamping an end flange (16) of the industrial robot (100) to the flexible clamping device (1); (S15) determining and loading a program into a robot controller of the industrial robot (100), the program defining a clamping deflection motion that produces deflection of one or more links of the industrial robot (100) based on a force interaction between the robot (100) and the flexible clamping device (1); (S16) controlling the industrial robot (100) to perform a clamping deflection motion while sensing a quantity of the industrial robot (100) and a quantity of the flexible clamping device (1); (S17) determining a reduced data set of the sensing measurements of the industrial robot (100) and the flexible clamping device (1) by excluding data of the sensing measurements that may comprise the influence of static friction; (S18) determining the relative attitude and force / torque between the top plate (2) and the base plate (3) of the flexible clamping device (1); and (S19) determining kinetic friction characteristics of the at least one joint exhibiting static friction and elastic characteristics of the at least one shaft exhibiting elasticity, and (S21) updating one or more of the at least one model of the industrial robot (100) with the determined kinetic friction characteristics and elastic characteristics, wherein the updated one or more in the at least one model provides elastic characteristics of the industrial robot (100) and related dynamics forming a scenario for using the parameters to improve control of the industrial robot (100).
Owner:COGNIBOTICS

Robot trajectory planning method and device for grabbing and placing tasks

The invention discloses a grabbing and placing task-oriented robot trajectory planning method and device, and the method comprises the steps: obtaining a natural language task instruction, carrying out the path decomposition based on a large language model, and generating a stage sequence containing pre-grabbing, grabbing, lifting, pre-placing, placing and withdrawing, and each stage target and / or constraint; receiving the optimal grabbing pose and / or placing pose provided by the upstream module, and replacing or correcting stage targets related to grabbing and / or placing; generating a trajectory segment according to the corrected stage target and constraint, and carrying out smoothing processing to obtain a trajectory sequence; performing collision detection on the track sequence, and if collision occurs, outputting collision information and executing local repair or replanning until the collision detection is passed or a preset replanning condition is met; and outputting the executable track sequence of the robot controller. According to the scheme, stage decomposition and key pose landing are achieved, and a collision feedback closed loop is formed to obtain an executable track.
Owner:BEIJING SCI & TECH PATENT OFFICE

Dynamic visual guidance method and system for cooperative assembly of multiple mechanical arms

The invention discloses a dynamic visual guidance method and system for cooperative assembly of multiple mechanical arms, and belongs to the technical field of industrial robot visual guidance, and the method comprises the steps: synchronously collecting an image sequence through a binocular visual system, synchronously recording the pose sequence of each mechanical arm through a robot controller, and obtaining an observation data set; obtaining a two-dimensional trajectory flow of vehicle body assembly area feature points and instrument body feature points through an optical flow tracking algorithm, and obtaining a hand-eye matrix correction through a nonlinear optimization algorithm; obtaining an updated hand-eye matrix and a local pixel compensation field through a spatial interpolation algorithm based on the re-projection error of each feature point; according to the method, self-adaptive compensation of flexible deformation of a vehicle body and dynamic reflection of an instrument is realized by adopting unsupervised clustering and spatial interpolation according to a re-projection error, and the precision of cooperative assembly of multiple mechanical arms and the self-adaptive capability of a system in a complex industrial environment are improved.
Owner:SUZHOU ZHENGTU INTELLIGENT TECH CO LTD

Rare earth metal electrolytic furnace discharging robot and tail end compliance control method and system thereof

The invention provides a rare earth metal electrolytic furnace discharging robot and a tail end compliance control method and system thereof, and the method comprises the steps: building a rare earth discharging simulation environment for simulating a discharging scene, and building a rare earth discharging control model for simulating the motion trail of a discharging tool through reinforcement learning training based on the simulation environment; contact force sense information between a discharging tool and the electrolytic furnace in the discharging process of the rare earth metal electrolytic furnace is obtained in real time and input into the rare earth discharging control model, and a real-time discharging tool movement track is obtained; and the real-time discharging tool motion trail is input into a discharging robot controller so as to achieve flexible control over the tail end of the discharging robot. According to the robot discharging operation method suitable for the rare earth metal electrolytic furnace, the robot discharging operation method suitable for the rare earth metal electrolytic furnace is achieved on the basis of reinforcement learning and force sensing, the force sensor senses the acting force of a tool and the electrolytic furnace, the working tail end of the rare earth metal electrolytic furnace is controlled through the reinforcement learning method, and then anthropomorphic discharging operation is achieved.
Owner:SHANGHAI JIAO TONG UNIVERSITY INNER MONGOLIA RESEARCH INSTITUTE

Method for adaptively correcting track of welding robot and adjusting welding parameters

The invention belongs to the technical field of fusion welding, and particularly relates to a method for adaptively correcting the track of a welding robot and adjusting welding parameters, which comprises the following steps of: scanning a welding seam through a sensor to obtain a current welding seam section image; the robot controller processes the weld joint section image to obtain coordinates of a middle point of a bottom-layer gap of a weld joint butt joint position; in the welding process of the welding robot, the stored welding track coordinates are repeatedly matched, the track coordinate closest to the current position of the welding robot is found, and the coordinate is judged to be the actual position of a welding seam; the welding robot obtains X-axis offset, Y-axis offset and Z-axis offset according to the current coordinates of the movement position of the robot and the matched welding track coordinates, and the movement track of the robot is corrected through the DPM dynamic path correction technology; and the welding parameters of the welding robot in the swing welding process or the welding process are modified or adjusted in real time.
Owner:SHENYANG SIASUN ROBOT & AUTOMATION

Self-adaptive welding device based on machine vision

The self-adaptive welding device comprises a robot, a robot controller, an industrial personal computer, a visual assembly and a welding gun, a visual processing module, a welding seam contour recognition module, a welding seam precision adjustment module and a track generation module are arranged in the industrial personal computer, and the visual assembly is used for conducting multi-angle shooting on a workpiece; the visual processing module is used for identifying geometrical characteristics of a main pipe and a branch pipe of the workpiece; the weld contour recognition module is used for recognizing a weld contour; the welding seam precision adjusting module is used for refining the precise position of the welding seam point at the junction of the main pipe and the branch pipe through a calibration method; the track generation module is used for generating a welding track; and the robot controller receives the welding track and controls the welding gun to move according to the welding track to execute welding operation. According to the method, the reasonable robot welding posture is automatically matched, the track continuity and the welding process stability are improved, through automatic track planning and execution, it is ensured that the quality of each welding seam is stable, and manual operation errors are reduced.
Owner:JIANGSU JINGNING INTELLIGENT MFG CO LTD

Gripper for Handling SMD Component Reel Cassettes, and Associated SMD Component Reel Cassette

A gripper for handling SMD component reel cassettes using a robotic arm automatically controlled by a robot controller wherein the robotic arm carries and manipulates the gripper on a tool flange of the robotic arm. The gripper includes a drive pinion, which is rotatably mounted on the gripper main body and is designed to mesh with an output wheel of the SMD component reel cassette when the SMD component reel cassette is coupled to the gripper. The gripper includes a drive motor, which is disposed on the gripper main body and is designed to automatically drive the drive pinion.
Owner:KUKA DEUT GMBH

Robot grinding and polishing method and system for titanium alloy casing runner welding slag

The invention provides a robot grinding and polishing method and system for titanium alloy casing runner welding slag. The system comprises a robot, a robot controller, a six-dimensional force sensor, a displacement sensor, a three-dimensional scanner, a belt sander and a workbench. The method comprises the following steps: firstly, fixing a casing on a workbench, positioning the casing by using a displacement sensor, scanning a casing flow channel by using a three-dimensional scanner to obtain a flow channel welding slag distribution state, grasping a belt sander by using a robot equipped with a six-dimensional force sensor, carrying out gravity calibration on the belt sander before grinding and polishing, and carrying out welding slag distribution on the belt sander; and the gravity of the belt sander is compensated according to the calibration result. Three-dimensional software is used for dividing a flow channel area, a machining path is calculated through an off-line trajectory planning method, a robot executable machining program is generated through post-processing for grinding and polishing, in the grinding and polishing process, in order to adapt to the complex curved surface of the cartridge receiver flow channel, real-time adjustment is conducted according to the contact force value of a tool and a workpiece, and the contact force value is transmitted to a robot controller; the removal amount can be determined, the roughness is guaranteed, and the surface quality is improved.
Owner:SHENYANG INST OF AUTOMATION - CHINESE ACAD OF SCI

Industrial robot unified control architecture system and universal robot controller

The invention discloses an industrial robot unified control architecture system and a universal robot controller, belongs to the technical field of robot control, and relates to a unified control architecture system based on an industrial robot whole machine performance optimization control algorithm. The problems that an existing industrial robot system is high in brand barrier and high in software and hardware coupling performance are solved. The system comprises the following hierarchical structures: a real-time control kernel layer, a system process and thread structure layer, a communication mechanism layer and a basic function module layer. The industrial robot unified control architecture system and the universal robot controller are suitable for integrating six-axis industrial robots of different brands and models into a unified group control system.
Owner:HARBIN INST OF TECH

System and Method for Interactive Robot Action Replanning Using Large Language Models

A robotic controller for controlling a robot according to a sequence of robotic actions. comprises an input interface to receive multimodal inputs specifying instructions for performing a task in audio, video, and a text modality. The controller transforms the multimodal instructions into encodings using a large language model (LLM) encoder and decodes the encodings into a first sequence of robotic instructions and a robot action description of the actions using an LLM decoder. Human feedback input is received corresponding to at least one action in the first sequence of actions and the controller encodes the feedback input with the robot action description. The controller feeds the encoded data along with multimodal features generated from the encodings into the LLM decoder to generate a corrected sequence of actions. The controller is configured to control a robot according to the corrected sequence of actions.
Owner:MITSUBISHI ELECTRIC RESEARCH LABORATORIES INC