689 results about "Teleoperation" patented technology

Disclosed in the present invention are a multi-modal shared teleoperation system and method for a three-arm space robot. The system at least comprises a master-side teleoperation system, a communication module and a slave-side robot system, wherein the master-side teleoperation system at least comprises two force feedback hand controllers, a microphone array and upper computer software, and the slave-side robot system comprises two operating arms each equipped with a gripper at the end, an observation arm having a binocular camera mounted at the end, a vision unit, a force sensor and lower computer software. The method comprises: an operator controlling two operating arms of an extravehicular robot to execute a task, and controlling an observation arm to acquire a better local field of view. In the method, a multi-modal teleoperation method comprising pose control, voice control and force control is fused with autonomous control of a robot by means of a shared control algorithm, and thus, human-robot collaborative control over the position, orientation and contact force of a robotic arm can be realized on the basis of the requirements of the operator, and the robot autonomously executes other relatively simple tasks, thereby reducing the operation burden of operators, and improving the control efficiency.

The invention belongs to the technical field of robot control, particularly provides a force feedback teleoperation control system and method for the tail end of a mechanical arm of a quadruped robot, and aims at the key challenges that control errors are caused by communication time delay and soft obstacles are difficult to recognize in a dynamic environment. Modeling and judgment are conducted on the contact state of the tail end of the mechanical arm in advance, and feedforward control and buffer adjustment oriented to communication time delay are achieved. And meanwhile, a dynamic semantic map is constructed in combination with multi-source sensing information, and soft obstacle reasoning and path optimization are performed by fusing a tail end force sense change trend, so that the recognition and avoidance capabilities of the system in a complex and invisible obstacle environment are remarkably improved. The system has good perspectiveness, self-adaptability and high redundancy safety characteristics, is suitable for multi-task inspection operation of industrial sites such as a thermal power plant, and is especially suitable for a remote man-machine cooperative operation scene in a narrow space.

A dual-arm cooperative potential field-based force-guided teleoperation system and control method. By means of real-time poses of tool center points of two robotic arms, whether the shortest distances dp-s,i between a target object and workspace boundaries of the two arms are lower than a threshold Ds is iteratively determined; if the shortest distances are both greater than the threshold, a dual-arm symmetric cooperation strategy is adopted, and if the shortest distances are lower than the threshold, a single-arm prioritized cooperation strategy is adopted, and the arm is used as a priority arm; cooperation factors δi of the two arms are respectively determined on the basis of the cooperation strategies; a dual-arm cooperative potential field is constructed on the basis of the cooperation factors, the distances between the target object and the tool center points of the two arms, and the position of an obstacle; and corresponding dynamic virtual constraint forces are generated on the basis of the dual-arm cooperative potential field, and two force feedback hand controllers are used to guide and assist an operator in controlling the two robotic arms to complete a dual-arm cooperative task.

Systems and methods that allow robots to perform tasks for users are provided. A robot may comprise one or more robotic arms and / or a mobile base. The arms may be controlled by electric actuators and may have six or more degrees of freedom. The robot may have sensors which can be accessed remotely. Robots may have varying levels of autonomy, including, for example, full teleoperation (in which a human can have detailed control over the robot) or full autonomy (in which the robot can complete a task without any human intervention). Various entities can interact with individual robots or groups of robots over a network, locally, directly or in person, or any combination thereof. A management system can allow entities to control and / or monitor the robots.

According to the double-arm mechanical arm teleoperation system based on the VR head-mounted display and the control method, high-precision and flexible teleoperation is achieved by accurately capturing movement tracks of the two hands and mapping the movement tracks to the double-arm mechanical arm, meanwhile, the content shot by the scene camera is transmitted to the VR head-mounted display in a streaming mode, and the real-time performance and immersion of operation are enhanced. In addition, the intelligent obstacle avoidance strategy designed by the invention effectively prevents the movement of the mechanical arm from being out of control in the operation process, and ensures the safety and reliability of the operation. The system has the remarkable advantages of high-precision operation, real-time performance, immersion, multi-scene adaptability, safety, reliability and the like, is suitable for interaction and task execution of a universal robot in industrial, household and other scenes, replaces human beings to execute precise operation in a high-risk environment, and has wide application prospects and important practical significance.

A teleoperations system may be used to generate virtual paths of travel for an autonomous vehicle based upon teleoperations system-provided suggestions, and virtual path suggestion constraints, e.g., based upon physical constraints associated with an autonomous vehicle such as minimum turning radius, trailer sweep, and the like, may be used assist in guiding a teleoperations system operator during generation of virtual path suggestions.

The invention discloses a dexterous hand teleoperation method based on vision-touch fusion. The method comprises the following steps: 1) vision detection; the method comprises the following steps of (1) carrying out remote control, (2) carrying out redirection, (3) carrying out tactile grabbing, (4) carrying out vision-tactile interaction, (5) carrying out a communication process and (6) carrying out dexterous hand.The method can stably map human hand actions under the complex vision condition, safe and reliable grabbing control is achieved through tactile feedback, meanwhile, multiple dexterous hand configurations and cross-platform deployment are supported, and the precision, robustness and practicability of teleoperation can be improved.

The present invention relates to the field of robotic arm teleoperation technology, and specifically to a construction method and system for real-time teleoperation of a dual arm and hand robot based on vision guidance, which includes setting up a first experimental platform, acquiring pose information of the dual arm and hand robot and a homogeneous transformation matrix of an end joint and transforming a first coordinate system; setting up a second experimental platform; performing calibration by a plurality of cameras and transforming a second coordinate system; obtaining spatial transformation pose information of the end joint of the robotic arm under a tracking state, reading finger pose data, and performing real-time following. The real-time teleoperation system of the present disclosure includes a control system host, an operation glove, an optical locator, a positioning coordinate plate, a marker ball tool, and a dual arm and hand robot. The present disclosure introduces a method of identifying the marker ball tool by the optical locator into the teleoperation control of the dual arm and hand robot, which is more accurate and faster. Meanwhile, the method of remotely controlling the robotic arm by an operator wearing the operation gloves can provide better human-machine interaction function.

The invention discloses a real-time whole-body remote operation control system for a humanoid robot, and the system comprises a motion instruction generation subsystem which is located at an operation end and is used for receiving and analyzing input instructions of various modes, and carrying out the unified conversion and complementation of the input instructions into a whole-body joint motion instruction which is suitable for a skeleton structure of a target humanoid robot; the body motion control subsystem is located at the robot end and used for receiving the whole-body motion instruction, adaptively generating a whole-body motion coordination control signal capable of ensuring the stability of the robot in combination with the state of the robot and environment information, and driving and executing the whole-body motion coordination control signal; through cooperation of the two subsystems, the system provides a dual guarantee path for realizing whole-body teleoperation: a complete whole-body instruction can be generated based on the instruction generation subsystem, and whole-body motion can be adaptively generated by the body motion control system according to a received local instruction; and the robustness, the flexibility and the reliability of the system under different conditions are ensured.

The invention provides a robot master-slave arm teleoperation software and hardware system. The system comprises a host, a master arm, a slave arm and a scene acquisition module, the master arm is used for collecting task action data and transmitting the task action data to the host; the host processes the task action data to generate an action instruction conforming to the driven arm, and the driven arm executes a corresponding action according to the action instruction; the scene acquisition module is used for acquiring operation scene data of the driven arm and transmitting the operation scene data to the host; and the host is also used for associatively storing the task action data, the execution action data of the driven arm and the operation scene data as training data of the intelligent model with the body. The host converts the task action data collected by the driving arm into the control instruction suitable for the driven arm, teleoperation of different types of driven arms can be supported, a large amount of training data of the body-equipped intelligent model suitable for different types of driven arms with different arm lengths is conveniently obtained by collecting and associating related data, and the training efficiency is improved. The device has the function of integrating intelligent data acquisition and model verification.

The invention provides a binocular vision teleoperation robot system, a robot and a teleoperation method and device, and relates to the technical field of computers. The method comprises the following steps: receiving a binocular video stream obtained by carrying out image acquisition on an actual scene by a binocular vision robot, and rendering the binocular video stream to obtain a three-dimensional scene; displaying the three-dimensional scene to an operator of the binocular vision robot to capture motion data of the operator observing the three-dimensional scene; determining a target area from the actual scene according to the action data, and obtaining a visual control instruction based on the target area; the vision control instruction is transmitted to the binocular vision robot for teleoperation; wherein the visual control instruction is used for adjusting imaging parameters of the binocular vision robot for image acquisition of an actual scene, and the visual direction of the binocular vision robot can be rapidly and naturally kept consistent with the attention point of the operator.

The invention relates to the technical field of humanoid robot control. The invention provides a humanoid robot bilateral force feedback teleoperation method of a force compensation mechanism. According to the method, a 5G private network is used for deployment, the characteristics of low delay, high reliability and large bandwidth of the 5G private network are utilized, real-time interaction of high-dimensional haptic data in a remote and cross-domain environment is effectively supported, cross-domain teleoperation can be achieved, high-fidelity feedback transmission of environment contact force is effectively achieved, and the real-time interaction of the high-dimensional haptic data in the remote and cross-domain environment is achieved. The precision, the operation efficiency and the overall safety of remote operation of the humanoid robot in complex industrial scenes such as nuclear power maintenance, precise assembly and high-risk environment operation are remarkably improved, the gap between traditional automation and low-efficiency remote operation is filled up, and reliable technical support is provided for advanced industrial application needing man-machine cooperation fine force control.

The invention discloses a master-slave robot force feedback teleoperation control system and method for cleaning the surface of an object, force feedback teleoperation control of a master-slave robot is realized through a double admittance control loop, stable contact with objects with different rigidities is realized by using a variable stiffness algorithm, and the engineering task requirement of cleaning attachments on the surface of the object is realized. Admittance algorithm parameters are modified in real time through a variable stiffness algorithm, uncertain motion such as buffeting caused by contact between a robot and a high-stiffness object in a conventional admittance algorithm is avoided, the robot can be in contact with objects with different rigidities and move stably, meanwhile, the force feedback effect of the admittance algorithm is achieved, and teleoperation control has a sense of immediacy.

The invention discloses a tea garden ditching and fertilizing all-in-one machine which adopts an electric crawler chassis which is independently driven on two sides, a fertilizing mechanism is located at the top of a whole machine frame, a ditching and earthing mechanism is located at the rear end of the whole machine frame, and the ditching depth can be adjusted by adjusting the length of a hydraulic telescopic rod; the control console is located at the front end of the whole machine frame and used for a driver to control the movement of the whole machine and the movement of all the mechanisms during driving, and meanwhile the ditching and fertilizing all-in-one machine for the tea garden can be controlled by selecting a wireless control mode under the condition that the road condition is simple. The ditching and fertilizing operation of the tea garden is more flexible; a wired and wireless combined remote control operation mode of ditching, fertilizing and earthing in the tea garden is realized; organic combination of the adjustable soil covering plate and the soil-splitting type fertilizing straight pipe is realized, and the soil crushing and covering link is more efficient; the ditching and fertilizing operation of the tea garden can be carried by a driver, and the complicated tea garden environment can be easily and timely adjusted in the operation process.

Provided herein is a control system and methods thereof for controlling operation of a robot. The control system comprises: a robot, including a plurality of sensors wherein each generating a stream of raw sensor data having a data type, size, and frequency, and a plurality of actuators cause movement of the robot; an autonomous control subsystem configured to receive the sensor data and output autonomous actuator data; and a teleoperation control subsystem configured to receive the sensor data, transmit the sensor data to a human operator, and output teleoperation actuator control signals, wherein the autonomous control subsystem and the teleoperation control subsystem receive, from the robot, sensor data having the same data type, size, and frequency, and wherein the autonomous actuator data and the teleoperation actuator data have the same data type, size, and frequency; and a control-determining subsystem configured to switch between autonomous and teleoperation control.

The invention discloses a robot dexterous arm teleoperation sharing control method and a related device, and relates to the technical field of robotics.The method comprises the steps that smooth expected pose information and expected angle state information are obtained, and pose information of a mechanical arm of a slave end at the current stage and angle state information and contact force information of a dexterous hand are obtained; based on the expected pose information and the joint angle information of the mechanical arm, the inverse kinematics of the mechanical arm is solved through a nonlinear optimization method, and a joint angle instruction of the mechanical arm is obtained; based on the expected angle state information, the angle state information of the dexterous hand and the contact force information, utilizing a fuzzy logic theory and a dynamic compliance primitive model to carry out adaptive variable impedance control on the dexterous hand to obtain a joint torque instruction of the dexterous hand; and feeding back the contact force information of the dexterous hand in the next stage to a main end operator. The success rate and the working efficiency of the teleoperation robot can be improved.

The invention discloses a dexterous hand general grabbing method and device based on single-visual-angle vision and demonstration optimization, and relates to the field of dexterous hand grabbing and artificial intelligence. The method comprises the following steps: firstly, building a real field scene and teleoperation equipment, and collecting expert data of objects with multiple sizes and shapes; and in combination with the collected expert data, on the basis of an algorithm obtained by improving a DAPG algorithm, a general grabbing strategy of cross-object shape primitives is learned in combination with domain randomization operation in a simulation environment. Geometric features of an unknown object are analyzed in real time through the first visual angle visual information, an initial grabbing track is generated, the grabbing force and posture are adjusted by fusing dynamic feedback of a force touch sensor, and high-generalization and high-reliability grabbing is carried out. The problems that a traditional method depends on a large amount of real expert data, a large amount of object model training is needed, strategy generalization is poor, and simulation migration stability is insufficient are solved.

The invention provides a digital twinning-combined dexterous hand teleoperation hand action migration method. The method comprises the following steps: acquiring an image sequence of a hand through a binocular camera; for each group of images in the image sequence, determining a three-dimensional coordinate sequence of the hand feature points; based on a pre-established digital twin skeleton model of the dexterous hand, establishing a mapping relation between the hand feature points and corresponding points in the digital twin skeleton model; converting the three-dimensional coordinate sequence of the hand feature point into a three-dimensional coordinate sequence of a corresponding mapping point in the digital twin skeleton model based on the mapping relation, taking the three-dimensional coordinate sequence as the position of the mapped hand feature point, and determining an optimal buckling angle and an optimal abduction angle; and based on the optimal buckling angle and the optimal abduction angle, a joint angle sequence is obtained, and corresponding joint actions in the dexterous hand are controlled to enable the dexterous hand to perform action migration according to hand actions in the image sequence, so that the smoothness and precision of the actions of the dexterous hand in the action migration process are improved.

A computer-implemented method executed by data processing hardware of a robot causes the data processing hardware to perform operations. The operations include receiving a sensor pointing command that commands the robot to use a sensor to capture sensor data of a location in an environment of the robot. The sensor is disposed on the robot. The operations include determining, based on an orientation of the sensor relative to the location, a direction for pointing the sensor toward the location, and an alignment pose of the robot to cause the sensor to point in the direction toward the location. The operations include commanding the robot to move from a current pose to the alignment pose. After the robot moves to the alignment pose and the sensor is pointing in the direction toward the location, the operations include commanding the sensor to capture the sensor data of the location in the environment.

The invention discloses a smart type library-based teleoperation method, system and device, and a storage medium. The method comprises the following steps of: constructing a smart operation type library; analyzing the task instruction through a preset multi-modal large language model MLLM to obtain a corresponding operation step, and matching a dexterous operation type corresponding to the current step from a dexterous operation type library; intuitive accurate control between the hand and the manipulator is realized through an interpolation mapping strategy. The interpolation mapping strategy specifically comprises the following steps: capturing a fingertip position of a human hand in real time; calculating the normalized projection proportion of the current posture of the human hand relative to the stretching state and the contraction state; and according to the projection proportion, linear interpolation is carried out on the extension and contraction joint angles of the manipulator, and a manipulator joint angle instruction is generated. According to the method, the adaptability and stability of teleoperation are provided by establishing a systematized smart type library and combining a type automatic retrieval module assisted by a multi-modal large language model and an interpolation mapping control strategy.

The invention discloses a mechanical arm teleoperation control method and system based on incremental mapping, and the method comprises the steps: carrying out the noise reduction processing of the position and posture of a handle through adaptive low-pass filtering, and achieving the position and posture consistency mapping from a handle space to a mechanical arm working space after coordinate system axis rearrangement and similar transformation mapping; the system provides two kinds of control output of a pose tracking mode and a speed tracking mode, real-time inverse kinematics solution is carried out through a Jacobian matrix and a damping least square method in the speed mode, and resolution scaling and exponential moving average / spherical linear interpolation smooth filtering are carried out on a target pose in the pose mode. And then the handle motion instruction is mapped into a Cartesian instruction or a joint space instruction at the tail end of the mechanical arm, closed-loop execution is conducted through a real-time state feedback link, and the method can be used for scenes such as robot service, teaching, remote operation and data collection.

The invention provides a multi-modal data feature extraction and optimization method based on teleoperation robot tasks, and relates to the technical field of robot imitation learning, and the method comprises the steps: firstly carrying out the data optimization of multi-modal data obtained by a robot through setting the truncation proportion of time steps, and removing meaningless operation data; then, depth information is introduced into the RGB color image in the multi-modal data needing to be processed, and the environment perception capability is enhanced; and finally, performing multi-modal feature mining by applying a DINO technology, wherein the multi-modal feature mining is specifically realized through a pre-training DINOv2 model of a ViT architecture. Through an innovative data optimization and multi-modal feature extraction method, the problems of data redundancy, insufficient feature extraction and low multi-modal information fusion efficiency in imitation learning are solved, and a remarkable actual value is brought to a user.

The invention belongs to the technical field of robots, in particular to the technical field of teleoperation robots, and particularly relates to a teleoperation master manipulator based on a power-off brake, which comprises a pose acquisition part and a tail end operation part, in an actual use scene, an operator holds a handle at the tail end and pulls a trigger, a power-off brake is powered on for contact braking, and the operator pulls a main hand to move; in the movement process, the absolute encoders of all the shafts collect the current angle and directly transmit the current angle to the isomorphic mechanical arm, or the tail end pose of the main hand is settled through forward kinematics and then mapped to the heterogeneous mechanical arm through inverse kinematics; and the trigger is released, the power-off brake is immediately powered off, all joints of the master manipulator are locked, then the posture is locked, and the master manipulator enters a posture keeping mode. A joint motor in the main hand is replaced by a brake, so that a motor servo control system is also removed, and the cost and the out-of-control risk are reduced.

The invention provides a pipeline welding method and system fusing teleoperation teaching and visual recognition. The method comprises the steps that discrete key point positions in a pipeline welding seam path picked up by teleoperation equipment teaching are obtained; generating an initial welding track by using a track interpolation method based on the key point location; a guide point is generated based on three-dimensional contour data, collected when a line laser contour sensor scans along the initial welding track, of an actual pipeline welding seam; the deviation value between the guide point and a corresponding track point on the initial welding track is obtained, and the initial welding track is compensated and corrected according to the deviation value to generate an actual welding track; and the welding robot is controlled to execute multi-layer welding operation along the actual welding track. According to the method, the experience of people and the precision of laser vision are combined, the planning efficiency and the track precision of the welding path are remarkably improved, and the welding quality and the operation safety of the pipeline under the severe working condition are guaranteed.

Provided herein is a control system and methods thereof for controlling operation of a robot. The control system comprises: a robot, including a plurality of sensors wherein each generating a stream of raw sensor data having a data type, size, and frequency, and a plurality of actuators cause movement of the robot; an autonomous control subsystem configured to receive the sensor data and output autonomous actuator data; and a teleoperation control subsystem configured to receive the sensor data, transmit the sensor data to a human operator, and output teleoperation actuator control signals, wherein the autonomous control subsystem and the teleoperation control subsystem receive, from the robot, sensor data having the same data type, size, and frequency, and wherein the autonomous actuator data and the teleoperation actuator data have the same data type, size, and frequency; and a control-determining subsystem configured to switch between autonomous and teleoperation control.

The invention provides an acquisition-training-evaluation integrated method and system for a humanoid robot. The method comprises the following steps: acquiring operation data of an operator by using master-slave control teleoperation equipment, and optimizing the operation data; the mechanical arm is controlled to respond, and data collection is conducted in the movement process of the mechanical arm; processing and storing the acquired data; applying the processed required information to a reasoning training process of the robot to form a reasoning model; a reasoning training result is deployed to the robot, so that the robot can start to execute a test task; real data of the robot in the execution process are collected, and the motion effect of robot training is evaluated and judged; a robot training reasoning model which does not conform to an evaluation standard is optimized, a data acquisition process and a training reasoning process are purposefully optimized from an evaluation result, and a'training-adjusting-correcting 'whole-process data active discovery and utilization mechanism is formed, so that the construction of a high-quality and multi-modal universal intelligent data set with a body is accelerated, and the accuracy and the reliability of the system are improved. Dependence on real data acquisition is reduced, and robot perception, understanding, reasoning and general generalization capabilities are enhanced. The problems that in the robot training process, training data and evaluation indexes are separated and separated, so that an integrated closed loop cannot be achieved in the collection-training-evaluation process, the evaluation result cannot act on the robot training improvement process, and the virtuous circle that the evaluation result counteracts on robot training cannot be achieved are solved.

The invention discloses a humanoid robot teleoperation system and a control method thereof. The system comprises the steps that an exoskeleton unit collects joint angle information and arm length information of arms of an operator; the data glove collects finger joint angle information of an operator; the inertial measurement unit collects space attitude information of the head and the chest of an operator; the double-shaft control pedal is used for generating integral moving, lifting and rotating control signals of the humanoid robot on the basis of foot actions of an operator; and the central controller receives the arm joint angle information, the arm length information, the finger joint angle information, the head and chest space attitude information and moving, lifting and rotating control signals of the humanoid robot, obtains a whole-body cooperative motion control instruction of the humanoid robot through kinematics calculation, and sends the whole-body cooperative motion control instruction to a control end of the humanoid robot. The high-fidelity teleoperation of the whole-body motion of the humanoid robot is realized by fusing the collected whole-body motion data of the operator and combining the self-adaptive kinematics of the central controller for resolving.

The invention belongs to the technical field of medical training, and discloses an operation training system and method based on multi-dimensional data perception fusion, and the system comprises an image collection module, a six-degree-of-freedom operation mechanism, a somatosensory and tactile feedback glove, an operation training robot, a simulation heart model, and a control system. The image acquisition module is in signal connection with the control system and is used for acquiring a 3D high-definition image and instrument space coordinates of an operation area in real time; and the six-degree-of-freedom operating mechanism is in signal connection with the control system and is used for converting an operation instruction of a doctor into a control signal of the operation training robot and feeding back force sense information. According to the scheme, a high-simulation operation training environment is provided through the six-degree-of-freedom teleoperation mechanism, the somatosensory and tactile sensing gloves, the operation training robot and the simulation heart model made of the multi-dimensional sensing electronic skin, multi-dimensional recording and feedback of operation operation are achieved, and the operation training efficiency is improved by combining a transfer learning method. And accurate recording and reproduction of the operation process of the qualified doctor are realized.

The invention discloses a mechanical arm control method and system based on imitation learning, and belongs to the field of intelligent robots and automatic control. The data collection layer designs a data collection method based on a VR platform, and is responsible for collecting teaching data to make a complex scene data set, performing coordinate conversion to solve angle information of each joint of the mechanical arm, realizing VR teleoperation teaching and teaching data collection, and transmitting the teaching data to the ACT algorithm layer for multi-modal image fusion; the multi-modal image fusion ACT algorithm layer carries out feature extraction and fusion denoising on complex illumination scene data, then infrared and visible light information fusion complementary features are combined with an ACT algorithm to obtain predicted action parameters, the predicted action parameters are converted into corresponding action plans through a motion control strategy, and a mechanical arm is driven to autonomously complete tasks. The method solves the problems that the performance of a mainstream imitation learning algorithm is limited in a complex illumination environment, the motion of the mechanical arm is prone to deflection and the like, and the perceptual performance and the capacity of adapting to complex scenes of the mechanical arm under the complex illumination condition are remarkably improved.

The present invention discloses a master-slave control method and system for a master-slave heterogeneous teleoperation system, including: an operator inputs the link parameters of the slave robot for forward kinematics and inverse kinematics analysis to obtain forward kinematics and inverse kinematics models; the operator modifies the master-slave mapping scale coefficient to achieve the movement of the slave robot with different efficiencies and accuracies; a variable-scale incremental master-slave mapping method is designed to realize the mapping from the position increment of the master robot to the position increment of the slave robot, and a BP neural network is designed to predict and compensate the master-slave following error, further reducing the master-slave following error. The present invention can not only realize the large-range movement of the slave robot in free space, but also ensure the movement positioning accuracy in the fine operation space, with strong versatility and convenient implementation.