609 results about "Teleoperation" patented technology

Disclosed in the present invention are a multi-modal shared teleoperation system and method for a three-arm space robot. The system at least comprises a master-side teleoperation system, a communication module and a slave-side robot system, wherein the master-side teleoperation system at least comprises two force feedback hand controllers, a microphone array and upper computer software, and the slave-side robot system comprises two operating arms each equipped with a gripper at the end, an observation arm having a binocular camera mounted at the end, a vision unit, a force sensor and lower computer software. The method comprises: an operator controlling two operating arms of an extravehicular robot to execute a task, and controlling an observation arm to acquire a better local field of view. In the method, a multi-modal teleoperation method comprising pose control, voice control and force control is fused with autonomous control of a robot by means of a shared control algorithm, and thus, human-robot collaborative control over the position, orientation and contact force of a robotic arm can be realized on the basis of the requirements of the operator, and the robot autonomously executes other relatively simple tasks, thereby reducing the operation burden of operators, and improving the control efficiency.

The invention belongs to the technical field of robot control, particularly provides a force feedback teleoperation control system and method for the tail end of a mechanical arm of a quadruped robot, and aims at the key challenges that control errors are caused by communication time delay and soft obstacles are difficult to recognize in a dynamic environment. Modeling and judgment are conducted on the contact state of the tail end of the mechanical arm in advance, and feedforward control and buffer adjustment oriented to communication time delay are achieved. And meanwhile, a dynamic semantic map is constructed in combination with multi-source sensing information, and soft obstacle reasoning and path optimization are performed by fusing a tail end force sense change trend, so that the recognition and avoidance capabilities of the system in a complex and invisible obstacle environment are remarkably improved. The system has good perspectiveness, self-adaptability and high redundancy safety characteristics, is suitable for multi-task inspection operation of industrial sites such as a thermal power plant, and is especially suitable for a remote man-machine cooperative operation scene in a narrow space.

A dual-arm cooperative potential field-based force-guided teleoperation system and control method. By means of real-time poses of tool center points of two robotic arms, whether the shortest distances dp-s,i between a target object and workspace boundaries of the two arms are lower than a threshold Ds is iteratively determined; if the shortest distances are both greater than the threshold, a dual-arm symmetric cooperation strategy is adopted, and if the shortest distances are lower than the threshold, a single-arm prioritized cooperation strategy is adopted, and the arm is used as a priority arm; cooperation factors δi of the two arms are respectively determined on the basis of the cooperation strategies; a dual-arm cooperative potential field is constructed on the basis of the cooperation factors, the distances between the target object and the tool center points of the two arms, and the position of an obstacle; and corresponding dynamic virtual constraint forces are generated on the basis of the dual-arm cooperative potential field, and two force feedback hand controllers are used to guide and assist an operator in controlling the two robotic arms to complete a dual-arm cooperative task.

Systems and methods that allow robots to perform tasks for users are provided. A robot may comprise one or more robotic arms and / or a mobile base. The arms may be controlled by electric actuators and may have six or more degrees of freedom. The robot may have sensors which can be accessed remotely. Robots may have varying levels of autonomy, including, for example, full teleoperation (in which a human can have detailed control over the robot) or full autonomy (in which the robot can complete a task without any human intervention). Various entities can interact with individual robots or groups of robots over a network, locally, directly or in person, or any combination thereof. A management system can allow entities to control and / or monitor the robots.

A teleoperations system may be used to generate virtual paths of travel for an autonomous vehicle based upon teleoperations system-provided suggestions, and virtual path suggestion constraints, e.g., based upon physical constraints associated with an autonomous vehicle such as minimum turning radius, trailer sweep, and the like, may be used assist in guiding a teleoperations system operator during generation of virtual path suggestions.

The invention discloses a dexterous hand teleoperation method based on vision-touch fusion. The method comprises the following steps: 1) vision detection; the method comprises the following steps of (1) carrying out remote control, (2) carrying out redirection, (3) carrying out tactile grabbing, (4) carrying out vision-tactile interaction, (5) carrying out a communication process and (6) carrying out dexterous hand.The method can stably map human hand actions under the complex vision condition, safe and reliable grabbing control is achieved through tactile feedback, meanwhile, multiple dexterous hand configurations and cross-platform deployment are supported, and the precision, robustness and practicability of teleoperation can be improved.

The present invention relates to the field of robotic arm teleoperation technology, and specifically to a construction method and system for real-time teleoperation of a dual arm and hand robot based on vision guidance, which includes setting up a first experimental platform, acquiring pose information of the dual arm and hand robot and a homogeneous transformation matrix of an end joint and transforming a first coordinate system; setting up a second experimental platform; performing calibration by a plurality of cameras and transforming a second coordinate system; obtaining spatial transformation pose information of the end joint of the robotic arm under a tracking state, reading finger pose data, and performing real-time following. The real-time teleoperation system of the present disclosure includes a control system host, an operation glove, an optical locator, a positioning coordinate plate, a marker ball tool, and a dual arm and hand robot. The present disclosure introduces a method of identifying the marker ball tool by the optical locator into the teleoperation control of the dual arm and hand robot, which is more accurate and faster. Meanwhile, the method of remotely controlling the robotic arm by an operator wearing the operation gloves can provide better human-machine interaction function.

The invention discloses a real-time whole-body remote operation control system for a humanoid robot, and the system comprises a motion instruction generation subsystem which is located at an operation end and is used for receiving and analyzing input instructions of various modes, and carrying out the unified conversion and complementation of the input instructions into a whole-body joint motion instruction which is suitable for a skeleton structure of a target humanoid robot; the body motion control subsystem is located at the robot end and used for receiving the whole-body motion instruction, adaptively generating a whole-body motion coordination control signal capable of ensuring the stability of the robot in combination with the state of the robot and environment information, and driving and executing the whole-body motion coordination control signal; through cooperation of the two subsystems, the system provides a dual guarantee path for realizing whole-body teleoperation: a complete whole-body instruction can be generated based on the instruction generation subsystem, and whole-body motion can be adaptively generated by the body motion control system according to a received local instruction; and the robustness, the flexibility and the reliability of the system under different conditions are ensured.

The invention provides a robot master-slave arm teleoperation software and hardware system. The system comprises a host, a master arm, a slave arm and a scene acquisition module, the master arm is used for collecting task action data and transmitting the task action data to the host; the host processes the task action data to generate an action instruction conforming to the driven arm, and the driven arm executes a corresponding action according to the action instruction; the scene acquisition module is used for acquiring operation scene data of the driven arm and transmitting the operation scene data to the host; and the host is also used for associatively storing the task action data, the execution action data of the driven arm and the operation scene data as training data of the intelligent model with the body. The host converts the task action data collected by the driving arm into the control instruction suitable for the driven arm, teleoperation of different types of driven arms can be supported, a large amount of training data of the body-equipped intelligent model suitable for different types of driven arms with different arm lengths is conveniently obtained by collecting and associating related data, and the training efficiency is improved. The device has the function of integrating intelligent data acquisition and model verification.

The invention provides a binocular vision teleoperation robot system, a robot and a teleoperation method and device, and relates to the technical field of computers. The method comprises the following steps: receiving a binocular video stream obtained by carrying out image acquisition on an actual scene by a binocular vision robot, and rendering the binocular video stream to obtain a three-dimensional scene; displaying the three-dimensional scene to an operator of the binocular vision robot to capture motion data of the operator observing the three-dimensional scene; determining a target area from the actual scene according to the action data, and obtaining a visual control instruction based on the target area; the vision control instruction is transmitted to the binocular vision robot for teleoperation; wherein the visual control instruction is used for adjusting imaging parameters of the binocular vision robot for image acquisition of an actual scene, and the visual direction of the binocular vision robot can be rapidly and naturally kept consistent with the attention point of the operator.

The invention relates to the technical field of humanoid robot control. The invention provides a humanoid robot bilateral force feedback teleoperation method of a force compensation mechanism. According to the method, a 5G private network is used for deployment, the characteristics of low delay, high reliability and large bandwidth of the 5G private network are utilized, real-time interaction of high-dimensional haptic data in a remote and cross-domain environment is effectively supported, cross-domain teleoperation can be achieved, high-fidelity feedback transmission of environment contact force is effectively achieved, and the real-time interaction of the high-dimensional haptic data in the remote and cross-domain environment is achieved. The precision, the operation efficiency and the overall safety of remote operation of the humanoid robot in complex industrial scenes such as nuclear power maintenance, precise assembly and high-risk environment operation are remarkably improved, the gap between traditional automation and low-efficiency remote operation is filled up, and reliable technical support is provided for advanced industrial application needing man-machine cooperation fine force control.

Provided herein is a control system and methods thereof for controlling operation of a robot. The control system comprises: a robot, including a plurality of sensors wherein each generating a stream of raw sensor data having a data type, size, and frequency, and a plurality of actuators cause movement of the robot; an autonomous control subsystem configured to receive the sensor data and output autonomous actuator data; and a teleoperation control subsystem configured to receive the sensor data, transmit the sensor data to a human operator, and output teleoperation actuator control signals, wherein the autonomous control subsystem and the teleoperation control subsystem receive, from the robot, sensor data having the same data type, size, and frequency, and wherein the autonomous actuator data and the teleoperation actuator data have the same data type, size, and frequency; and a control-determining subsystem configured to switch between autonomous and teleoperation control.

The invention discloses a robot dexterous arm teleoperation sharing control method and a related device, and relates to the technical field of robotics.The method comprises the steps that smooth expected pose information and expected angle state information are obtained, and pose information of a mechanical arm of a slave end at the current stage and angle state information and contact force information of a dexterous hand are obtained; based on the expected pose information and the joint angle information of the mechanical arm, the inverse kinematics of the mechanical arm is solved through a nonlinear optimization method, and a joint angle instruction of the mechanical arm is obtained; based on the expected angle state information, the angle state information of the dexterous hand and the contact force information, utilizing a fuzzy logic theory and a dynamic compliance primitive model to carry out adaptive variable impedance control on the dexterous hand to obtain a joint torque instruction of the dexterous hand; and feeding back the contact force information of the dexterous hand in the next stage to a main end operator. The success rate and the working efficiency of the teleoperation robot can be improved.

The invention discloses a dexterous hand general grabbing method and device based on single-visual-angle vision and demonstration optimization, and relates to the field of dexterous hand grabbing and artificial intelligence. The method comprises the following steps: firstly, building a real field scene and teleoperation equipment, and collecting expert data of objects with multiple sizes and shapes; and in combination with the collected expert data, on the basis of an algorithm obtained by improving a DAPG algorithm, a general grabbing strategy of cross-object shape primitives is learned in combination with domain randomization operation in a simulation environment. Geometric features of an unknown object are analyzed in real time through the first visual angle visual information, an initial grabbing track is generated, the grabbing force and posture are adjusted by fusing dynamic feedback of a force touch sensor, and high-generalization and high-reliability grabbing is carried out. The problems that a traditional method depends on a large amount of real expert data, a large amount of object model training is needed, strategy generalization is poor, and simulation migration stability is insufficient are solved.

The invention provides a digital twinning-combined dexterous hand teleoperation hand action migration method. The method comprises the following steps: acquiring an image sequence of a hand through a binocular camera; for each group of images in the image sequence, determining a three-dimensional coordinate sequence of the hand feature points; based on a pre-established digital twin skeleton model of the dexterous hand, establishing a mapping relation between the hand feature points and corresponding points in the digital twin skeleton model; converting the three-dimensional coordinate sequence of the hand feature point into a three-dimensional coordinate sequence of a corresponding mapping point in the digital twin skeleton model based on the mapping relation, taking the three-dimensional coordinate sequence as the position of the mapped hand feature point, and determining an optimal buckling angle and an optimal abduction angle; and based on the optimal buckling angle and the optimal abduction angle, a joint angle sequence is obtained, and corresponding joint actions in the dexterous hand are controlled to enable the dexterous hand to perform action migration according to hand actions in the image sequence, so that the smoothness and precision of the actions of the dexterous hand in the action migration process are improved.

A computer-implemented method executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the location, a direction for pointing the sensor toward the location, and an alignment pose of the robot to cause the sensor to point in the direction toward the location. The operations include commanding the robot to move from a current pose to the alignment pose. After the robot moves to the alignment pose and the sensor is pointing in the direction toward the location, the operations include commanding the sensor to capture the sensor data of the location in the environment.

The invention discloses a smart type library-based teleoperation method, system and device, and a storage medium. The method comprises the following steps of: constructing a smart operation type library; analyzing the task instruction through a preset multi-modal large language model MLLM to obtain a corresponding operation step, and matching a dexterous operation type corresponding to the current step from a dexterous operation type library; intuitive accurate control between the hand and the manipulator is realized through an interpolation mapping strategy. The interpolation mapping strategy specifically comprises the following steps: capturing a fingertip position of a human hand in real time; calculating the normalized projection proportion of the current posture of the human hand relative to the stretching state and the contraction state; and according to the projection proportion, linear interpolation is carried out on the extension and contraction joint angles of the manipulator, and a manipulator joint angle instruction is generated. According to the method, the adaptability and stability of teleoperation are provided by establishing a systematized smart type library and combining a type automatic retrieval module assisted by a multi-modal large language model and an interpolation mapping control strategy.

The invention discloses a mechanical arm teleoperation control method and system based on incremental mapping, and the method comprises the steps: carrying out the noise reduction processing of the position and posture of a handle through adaptive low-pass filtering, and achieving the position and posture consistency mapping from a handle space to a mechanical arm working space after coordinate system axis rearrangement and similar transformation mapping; the system provides two kinds of control output of a pose tracking mode and a speed tracking mode, real-time inverse kinematics solution is carried out through a Jacobian matrix and a damping least square method in the speed mode, and resolution scaling and exponential moving average / spherical linear interpolation smooth filtering are carried out on a target pose in the pose mode. And then the handle motion instruction is mapped into a Cartesian instruction or a joint space instruction at the tail end of the mechanical arm, closed-loop execution is conducted through a real-time state feedback link, and the method can be used for scenes such as robot service, teaching, remote operation and data collection.

The invention provides a pipeline welding method and system fusing teleoperation teaching and visual recognition. The method comprises the steps that discrete key point positions in a pipeline welding seam path picked up by teleoperation equipment teaching are obtained; generating an initial welding track by using a track interpolation method based on the key point location; a guide point is generated based on three-dimensional contour data, collected when a line laser contour sensor scans along the initial welding track, of an actual pipeline welding seam; the deviation value between the guide point and a corresponding track point on the initial welding track is obtained, and the initial welding track is compensated and corrected according to the deviation value to generate an actual welding track; and the welding robot is controlled to execute multi-layer welding operation along the actual welding track. According to the method, the experience of people and the precision of laser vision are combined, the planning efficiency and the track precision of the welding path are remarkably improved, and the welding quality and the operation safety of the pipeline under the severe working condition are guaranteed.

Provided herein is a control system and methods thereof for controlling operation of a robot. The control system comprises: a robot, including a plurality of sensors wherein each generating a stream of raw sensor data having a data type, size, and frequency, and a plurality of actuators cause movement of the robot; an autonomous control subsystem configured to receive the sensor data and output autonomous actuator data; and a teleoperation control subsystem configured to receive the sensor data, transmit the sensor data to a human operator, and output teleoperation actuator control signals, wherein the autonomous control subsystem and the teleoperation control subsystem receive, from the robot, sensor data having the same data type, size, and frequency, and wherein the autonomous actuator data and the teleoperation actuator data have the same data type, size, and frequency; and a control-determining subsystem configured to switch between autonomous and teleoperation control.

The invention provides an acquisition-training-evaluation integrated method and system for a humanoid robot. The method comprises the following steps: acquiring operation data of an operator by using master-slave control teleoperation equipment, and optimizing the operation data; the mechanical arm is controlled to respond, and data collection is conducted in the movement process of the mechanical arm; processing and storing the acquired data; applying the processed required information to a reasoning training process of the robot to form a reasoning model; a reasoning training result is deployed to the robot, so that the robot can start to execute a test task; real data of the robot in the execution process are collected, and the motion effect of robot training is evaluated and judged; a robot training reasoning model which does not conform to an evaluation standard is optimized, a data acquisition process and a training reasoning process are purposefully optimized from an evaluation result, and a'training-adjusting-correcting 'whole-process data active discovery and utilization mechanism is formed, so that the construction of a high-quality and multi-modal universal intelligent data set with a body is accelerated, and the accuracy and the reliability of the system are improved. Dependence on real data acquisition is reduced, and robot perception, understanding, reasoning and general generalization capabilities are enhanced. The problems that in the robot training process, training data and evaluation indexes are separated and separated, so that an integrated closed loop cannot be achieved in the collection-training-evaluation process, the evaluation result cannot act on the robot training improvement process, and the virtuous circle that the evaluation result counteracts on robot training cannot be achieved are solved.

The invention discloses a humanoid robot teleoperation system and a control method thereof. The system comprises the steps that an exoskeleton unit collects joint angle information and arm length information of arms of an operator; the data glove collects finger joint angle information of an operator; the inertial measurement unit collects space attitude information of the head and the chest of an operator; the double-shaft control pedal is used for generating integral moving, lifting and rotating control signals of the humanoid robot on the basis of foot actions of an operator; and the central controller receives the arm joint angle information, the arm length information, the finger joint angle information, the head and chest space attitude information and moving, lifting and rotating control signals of the humanoid robot, obtains a whole-body cooperative motion control instruction of the humanoid robot through kinematics calculation, and sends the whole-body cooperative motion control instruction to a control end of the humanoid robot. The high-fidelity teleoperation of the whole-body motion of the humanoid robot is realized by fusing the collected whole-body motion data of the operator and combining the self-adaptive kinematics of the central controller for resolving.

The invention belongs to the technical field of medical training, and discloses an operation training system and method based on multi-dimensional data perception fusion, and the system comprises an image collection module, a six-degree-of-freedom operation mechanism, a somatosensory and tactile feedback glove, an operation training robot, a simulation heart model, and a control system. The image acquisition module is in signal connection with the control system and is used for acquiring a 3D high-definition image and instrument space coordinates of an operation area in real time; and the six-degree-of-freedom operating mechanism is in signal connection with the control system and is used for converting an operation instruction of a doctor into a control signal of the operation training robot and feeding back force sense information. According to the scheme, a high-simulation operation training environment is provided through the six-degree-of-freedom teleoperation mechanism, the somatosensory and tactile sensing gloves, the operation training robot and the simulation heart model made of the multi-dimensional sensing electronic skin, multi-dimensional recording and feedback of operation operation are achieved, and the operation training efficiency is improved by combining a transfer learning method. And accurate recording and reproduction of the operation process of the qualified doctor are realized.

A vehicle teleoperation system may comprise a single, integrated unit that is installed within a vehicle to enable remote driving operations. The vehicle teleoperation system may include a processing island, a connectivity island, a safety island, and various interfaces by which to connect the vehicle teleoperation system to various systems onboard a vehicle. For example, the vehicle teleoperation system may receive and process various sensor data, transmit the sensor data for presentation to a teleoperator at a teleoperator station, receive and process user commands from the teleoperator at the teleoperator station, and instruct remote execution of the user commands by the vehicle.

The invention discloses a master-slave teleoperation-oriented force sense information master control system and a resolving method thereof, which are applied to a master-slave teleoperation scene, the master-slave teleoperation scene comprises a master-end teleoperation system, a slave-end mechanical arm system and a communication module, and the master-end teleoperation system comprises a master-end hand controller and a master-end upper computer which are directly connected; the master end upper computer is further internally provided with a force sense information master control unit, and the force sense information master control unit obtains interaction force of the slave end mechanical arm and the external environment under the condition that a sensor is arranged at the tail end of the slave end mechanical arm or no sensor exists, and calculates the interaction force into the master end hand controller. A combined filter is provided to cope with force information interference factors, a user interface is further provided to feed back interaction force between a slave end mechanical arm and an external environment to a master end visual interface, master-slave end overall exchange and comprehensive control of force sense information are achieved, a visual reference is provided for a master end operator to make a decision, and a force sense immediacy sense can be established for master end hand controller equipment.

The invention discloses an aircraft remote control delay compensation response method and system, and the method comprises the steps: obtaining airborne data, and constructing a multi-source data set; calculating bidirectional transmission delay based on a bidirectional delay estimation algorithm, and calculating initial total delay; the initial total delay is dynamically calibrated by using a self-adaptive delay calibration algorithm, the confidence coefficient is evaluated, and the delay precision is improved; deducing a flight state in a delay time period by adopting a delay deduction hybrid model, and dynamically adjusting parameters along with load; according to the deduction result and the original instruction, generating a compensation instruction containing constraints by using a model prediction control algorithm; and executing the instruction after onboard verification, comparing the actual state with the expected state, and feeding back a deviation value to optimize the weight of the next-round model so as to form a closed-loop lifting mechanism. According to the method, the problems of incomplete delay composition analysis, poor environmental adaptability, lack of constraint collaboration of compensation instructions, no closed-loop optimization mechanism and the like in the existing aircraft remote control delay compensation are solved.

The invention discloses a quick positioning and clamping mechanism and method for a sealing surface teleoperation repairing device in a high-level environment. The quick positioning and clamping mechanism comprises a target body base, a positioning pin, a repairing device base, a coarse positioning mechanism, a supporting plate assembly, a cam self-locking and clamping mechanism and a guide shaft linkage rotating assembly. The coarse positioning mechanisms are fixed to the two sides of a bottom plate of the repairing device base, the two sides of each coarse positioning mechanism are matched with positioning grooves in side plates on the two sides of the target body base, the positioning pins are inserted into fine positioning holes in the bottom plate of the repairing device base after coarse positioning, and the supporting plate assembly is fixed to the bottom plate of the repairing device base. The cam self-locking clamping mechanism is fixed to the supporting plate assembly, clamping and fixing between the target body base and the repairing device base are completed by operating the cam self-locking clamping mechanism, and the guide shaft linkage rotating assembly is fixed to the cam self-locking clamping mechanism. And clamping jaws on the two sides of the cam self-locking clamping mechanism are synchronously rotated under remote operation through linkage of the guide shaft and the rotating assembly. The positioning reliability and efficiency of the sealing surface teleoperation repairing device in the high-level radioactive environment are improved.

The invention discloses a two-arm smart teleoperation method and system based on motion redirection and self-adaption, and relates to the technical field of robots, and the method comprises the steps: determining the posture of a human hand and the posture of a human arm according to a human body image; the robot is redirected according to the human hand posture and the human arm posture through the space attention gating map convolutional network, and the joint angle of a mechanical hand of the robot and the joint angle of a mechanical arm of the robot are correspondingly obtained; the mechanical arm is driven according to the joint angle of the mechanical arm so that the mechanical arm can grab the target object; and target force needed for grabbing the target object is determined, and the mechanical arm is driven to grab the target object according to the target force and the joint angle of the mechanical arm. According to the method, topology modeling and cross-platform action redirection between the human body and the robot are achieved through the space attention gating map convolutional network, pairing data dependence is eliminated, and therefore the high precision of action reproduction and the safety and flexibility of the grabbing process are both considered in the complex dynamic environment.

The arrangements disclosed herein relate to receiving a data stream of a medical procedure performed with a robotic medical system; identifying, using the data stream and with one or more models trained with machine learning, a type of an anatomical structure on which the medical procedure is performed, and a type of an interaction with the anatomical structure; and determining an amount of force to be applied to the anatomical structure based at least in part on the type of the anatomical structure and the type of the interaction with the anatomical structure. Arrangements also relate to providing an indication of the amount of force to control performance of the medical procedure with the robotic medical system.

The invention provides a mechanical arm teleoperation system adaptive to a quadruped robot and a control method, and relates to the technical field of robot control. According to the method, firstly, a far-end main mechanical arm is introduced on the basis of a mechanical arm of the quadruped robot to form a main-end mechanical arm system and a slave-end mechanical arm system, and a remote operator remotely maps an action instruction to the slave-end mechanical arm system by controlling the main-end mechanical arm; then, a clamping jaw module with an image and tail end contact force sensor is installed at the tail end of the slave end mechanical arm, and image data collection and tail end contact force collection on a working site are achieved; secondly, the image information and the tail end contact force are remotely transmitted to a remote operator in real time through a wireless transmission module and a remote man-machine interaction module; and finally, a remote operator controls the main end mechanical arm in real time according to the real-time image information and the tail end contact force, remote operation of the quadruped robot mounting mechanical arm in a complex scene is achieved, and the operation efficiency and the operation precision are improved.