174 results about "Robot action" patented technology

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a task processing method, device, equipment and medium based on an expert sub-path, comprising: setting a shared attention layer, an understanding expert sub-path, a control expert sub-path and a router module in a neural network, the method comprises the following steps: constructing a robot action trajectory data set, a visual text pair data set and a structured language template set, training based on the robot action trajectory data set and the structured language template set to obtain an intermediate training model, and training based on the intermediate training model and the visual text pair data set to generate a unified training model; and when target task input is received, the router module selects a corresponding expert sub-path, and a result is output through the unified training model. According to the method, staged training is combined with an expert mixing mechanism, catastrophic forgetting is avoided, semantic understanding and robot control keep stable coexistence in a unified model, and the accuracy of task execution is improved.

The present disclosure provides a method for generating annotation data for robotic training using a hierarchical transformer-based model with multiple layers. The transformer-based model includes Alpha models generating low-level control outputs and Beta models generating high-level control outputs. The method receives multimodal input data comprising visual sensor data and natural language instructions, processes this data through the hierarchical transformer-based model to generate annotations at different abstraction levels, wherein Beta models create semantic annotations describing task objectives and Alpha models generate motor command annotations specifying robotic actions, and stores these annotations with the input data to create annotated training data for robotic control systems.

The invention discloses an intelligent robot with a body and a robot motion control system and method.The intelligent robot with the body is provided with an execution component, a moving module and the robot motion control system, and the robot motion control system comprises a multi-modal input encoder, a multi-modal output encoder and a multi-modal output encoder, the multi-modal feature fusion module is used for collecting and processing multi-modal data to obtain multi-modal features and fusing the multi-modal features to obtain a multi-modal feature token sequence; the action expert network is used for mapping the fused multi-modal feature token sequence into an abstract robot action sequence block; and the execution controller is used for converting the abstract robot action sequence block into a control instruction which can be executed by bottom hardware. According to the method, direct mapping from environment perception and semantic understanding to movement control and execution of specified task actions can be realized, so that the task understanding, response speed and execution completion degree of the robot in a complex environment are improved.

A robotic controller for controlling a robot according to a sequence of robotic actions. comprises an input interface configured to receive a plurality of multimodal inputs each specifying instructions for performing a task in a different modality including audio, video, and a text modality. The controller also comprises a multimodal large language model, an action sequence decoder, and a controller. The multimodal LLM includes a multimodal LLM encoder and an LLM decoder. The multimodal LLM encoder is trained with machine learning to transform the multimodal instructions into encodings and the LLM decoder is configured to decode the encodings into a sequence of robotic instructions. The action sequence decoder is trained with machine learning to transform the sequence of robotic instructions into a sequence of actions using a library of robotic skills. The controller is configured to control a robot according to the sequence of actions.

The invention provides a robot action generation method and device, and the method comprises the steps: responding to a target task received by a robot, and obtaining image data and text description information associated with the target task; on the basis of the image data and the text description information, potential action information and fused visual representation information are determined by utilizing a large language model obtained by supervised training provided based on a potential action model; compressing and splicing the image data, the potential action information and the fused visual representation information to obtain control sequence information; and de-noising and splicing the state information and the control sequence information corresponding to the robot to generate action sequence information, so that the robot executes a corresponding action based on the action sequence information. By means of the method, the smoothness and coherence of actions generated by the robot are improved.

The invention provides a robot control method based on a visual language action model and related equipment thereof, and relates to the technical field of intelligent robots, and the method comprises the steps: obtaining visual observation information, a target language instruction and robot body state information of a target robot; reasoning the visual observation information, the target language instruction and the robot body state information through a pre-trained visual language action model to obtain a predicted low-dimensional potential vector; reconstructing the predicted low-dimensional potential vector into a robot action instruction through a pre-trained high-dimensional action reconstruction module; wherein the dimension of the robot action instruction is matched with the action space dimension of the target robot; and the target robot is driven to execute the high-dimensional robot action instruction. According to the method, the limitation of the existing visual language action model on the action output dimension can be solved, so that the high-degree-of-freedom robot is effectively controlled.

The embodiment of the invention relates to the technical field of information, and discloses an intelligent multi-source data quality evaluation and verification method and device, a medium and a product, and the method comprises the steps: converting intelligent multi-source data into standardized format data; performing quality evaluation on the standardized format data, wherein the quality evaluation comprises at least one of integrity check, availability check and consistency check; loading a robot URDF model corresponding to the standardized format data, driving the URDF model to move according to joint data, and synchronously playing corresponding visual data for realizing linkage playback of model actions and visual pictures; analyzing linkage playback through the first large model and / or the second large model; wherein the first large model is used for judging the physical safety of the robot action, and the second large model is used for judging the high-level semantic correctness of the robot action and task annotation, so that a high-quality and reliable self-service intelligent data set is constructed.

The invention relates to a robot motion reasoning method and system based on a Gaussian motion field, and the method comprises the steps: inputting a sparse and uncalibrated multi-view RGB image, extracting mixed scene features through a visual Transform backbone network, constructing a dynamic Gaussian motion field, endowing each Gaussian unit with a learnable motion attribute, and carrying out the reasoning of the motion of a robot. And realizing synchronous modeling of scene geometry and motion evolution. The system further comprises a multi-modal query module, a point cloud registration module, a diffusion model optimization module and a closed-loop control module which are respectively used for current and future scene reconstruction, mechanical arm tail end clamping jaw action estimation, action sequence optimization and real-time feedback and model updating in the execution process. According to the method, through a unified space-time modeling and closed-loop control mechanism, the robot operation problem in dynamic, shielding and uncalibrated environments is effectively solved, and the method is suitable for various application scenes such as industrial automation and service robots.

The invention discloses a health monitoring system of an old-age care robot, which relates to the technical field of health monitoring, and comprises the steps of constructing an indoor multi-modal data pool, generating an encrypted feature vector, establishing a steady-state physiological behavior baseline according to a historical encrypted feature vector, and outputting a health risk level and an intervention instruction set. According to the invention, all-time and all-scene health monitoring is realized by integrating a multi-mode sensor, old people do not need to wear equipment, feature compression and homomorphic encryption technologies are adopted, data security is ensured, a threshold value can be updated in real time, and the method is suitable for being applied to the health monitoring of the old people. The health state of the old people is accurately monitored, health risk prediction and personalized intervention are carried out in combination with analysis of behaviors such as falling and static abnormity, the prompt mode is dynamically adjusted according to the ability of the old people, the data processing efficiency is optimized through the edge cloud collaborative architecture, the data privacy and stability are guaranteed, and the life quality and safety are improved.

The invention relates to the technical field of robots, and discloses a robot action prediction method and device, computer equipment and a storage medium, and the method comprises the steps: generating an input observation sequence according to a position code and a current feature vector, and predicting an action sequence through Transform and the input observation sequence; and exponential decay weighted average processing is carried out on the predicted action sequence, and a target action is determined. Through the above mode, the multi-modal observation data is converted into the unified feature vector through the feature extraction network, key information in the observation data is reserved, the time sequence dependency relationship and context information in the observation data are fully captured by using the Transform decoder, and the robot action sequence is accurately predicted. And exponential decay weighted average processing is performed on the predicted action sequence, so that the action sequence is further smoothed, the prediction instability is reduced, the finally determined target action is more accurate, and the performance and success rate of the robot during task execution are improved.

The embodiment of the invention provides a robot action generation method and system, a medium, equipment and a program product. The method comprises the following steps: acquiring visual input data of a robot and corresponding language instruction data; obtaining an intermediate action representation by using a pre-constructed cross-modal shared semantic space, a corresponding relationship between the shared semantic representation and the intermediate action representation, the visual input data and the language instruction data; wherein the shared semantic representation is obtained on the basis of visual input data and / or language instruction data by using a cross-modal shared semantic space; and action parameters are generated according to the intermediate action representation so as to drive the robot to execute corresponding actions. Due to the fact that the cross-modal shared semantic space can enable semantic information from different modalities to be measured under the unified scale, and the corresponding relation can serve as middle bridging representation from semantics to middle action representation, the ability of a robot to understand and execute common language instructions in diversified real scenes is improved.

The invention provides a robot action redirection method, a storage medium, electronic equipment and a product, and relates to the field of robot control. The method comprises the steps that motion data of a to-be-simulated object and pre-configured joint offset parameters are obtained, wherein the joint offset parameters are used for describing rotation transformation data from a joint coordinate system of the to-be-simulated object to a joint coordinate system of a robot; based on the joint offset parameters, joint offset mapping from the to-be-simulated object to the robot is established; and based on the joint offset mapping, converting the motion data of the to-be-simulated object into motion data of the robot so as to drive the robot to execute corresponding actions. According to the method, the difference between the to-be-simulated object and the robot in joint coordinate system rotation transformation can be effectively eliminated, the physical feasibility and motion naturalness of robot actions are ensured, and the interaction requirements in a complex interaction scene are met.

The invention relates to a robot autonomous operation optimization method and system based on virtual simulation and multi-hypothesis planning. The method comprises the steps that a virtual simulation platform scene with the same configuration as a real experiment scene is built; robot training is carried out in the virtual simulation platform scene, and an autonomous operation strategy is generated; constructing a plurality of hypothetical states based on the current state of the robot, and obtaining a virtual track action sequence of each hypothetical state based on an autonomous operation strategy; sequentially verifying whether the virtual track action sequence corresponding to each virtual state is feasible or not according to a time sequence, and if the virtual track action sequence is feasible, taking the virtual track action sequence as an optimal virtual track action sequence; and aligning the real robot state with the assumed state corresponding to the optimal virtual track action sequence, and generating a real robot action sequence based on the current state and the optimal virtual track action sequence by adopting a cooperative action integration mechanism. Compared with the prior art, the method has the advantages that the difference between virtuality and reality of the model is reduced, and the execution success rate is increased.

A robotic controller for controlling a robot according to a sequence of robotic actions. comprises an input interface to receive multimodal inputs specifying instructions for performing a task in audio, video, and a text modality. The controller transforms the multimodal instructions into encodings using a large language model (LLM) encoder and decodes the encodings into a first sequence of robotic instructions and a robot action description of the actions using an LLM decoder. Human feedback input is received corresponding to at least one action in the first sequence of actions and the controller encodes the feedback input with the robot action description. The controller feeds the encoded data along with multimodal features generated from the encodings into the LLM decoder to generate a corrected sequence of actions. The controller is configured to control a robot according to the corrected sequence of actions.

A multi-module collaborative real-time decision planning system for a surgical robot comprises eight modules including a data acquisition and processing module, a system modeling module, a preoperative planning module, a neural network module, a decision planning module, an execution control module, an autonomous obstacle avoidance module and an autonomous evaluation module. The system obtains robot state and operation environment information in real time through multiple sensors, and constructs an operation environment and motion model; the inverse kinematics is solved by using a neural network, and the multi-solution problem of the inverse kinematics is solved; the preoperative planning module identifies key points and generates an initial path, the decision planning module dynamically adjusts the path in combination with joint constraints, and the execution control module controls the robot to act in a closed-loop mode according to instructions and corrects the relation between force and current in real time; the autonomous obstacle avoidance module monitors the current of a driving motor to realize collision detection and path adjustment; and the autonomous evaluation module continuously evaluates the operation quality through a reward function and feeds back optimization. The system can significantly improve the real-time performance, precision and safety of the surgical robot in a complex environment, and is suitable for the fields of medical treatment, industry, space exploration and the like.

The invention provides a robot action planning method and device, a terminal and a medium. The method comprises the steps that a user instruction, a scene perception image set, sensor information and a safety threshold table are acquired; generating an action draft according to the user instruction and the scene perception image set; performing virtual simulation on the scene perception image set and the sensor information according to a dynamic virtual twinning method to obtain virtual twinning state information; performing processing through the action draft and the virtual twinning state information, determining a risk vector of current iteration, generating an iterative action draft based on the risk vector of the current iteration and a safety threshold table, and obtaining iterative virtual twinning state information; and determining an iterative risk vector according to the iterative action draft and the iterative virtual twin state information, and obtaining a robot action instruction. On the premise of not depending on a pre-established and perfect global physical model, accurate physical consequence prediction is carried out on locally and dynamically changing interaction scenes.

The invention discloses a hierarchical robot operation strategy generation method, device and equipment, and the generation method comprises the steps: collecting a hand-eye camera image and an external camera image of a robot, and obtaining a natural language instruction; taking a hand-eye camera image and an external camera image as image observation, and inputting the images into a cyclic consistency variational auto-encoder to obtain semantic features and spatial features; generating a new-view-angle hand-eye camera image, performing multi-view-angle fusion with image observation, and updating semantic features; inputting the semantic features and the natural language instruction into a basic skill discriminator to generate a robot skill category; and inputting the spatial features and the robot skill categories into an action generator to generate robot actions. The invention relates to the technical field of robot intelligent control and artificial intelligence, can significantly improve the success rate and reliability of the robot executing long-time-sequence and multi-task operation in a complex and changing environment, and has a wide application prospect.

The invention provides a data generation method, device and equipment for dexterous operation and a storage medium, and the method comprises the steps: obtaining at least one piece of demonstration data generated when a robot executes a dexterous demonstration task, the dexterous demonstration task comprises the initial pose of at least one target object, the demonstration data comprises a first robot action sequence and first observation data corresponding to the first robot action sequence; according to the first observation data, the corresponding first robot action sequence is divided into a motion section and a skill section corresponding to each target object; responding to multiple times of pose adjustment of the target object, performing pose adaptation on the motion section and the skill section corresponding to the target object, and generating a plurality of second robot action sequences; and executing the second robot action sequence in the simulation environment, collecting corresponding second observation data, and pairing the second observation data with the second robot action sequence to generate training data.

The invention provides a robot action sequence generation method and device, equipment and a medium, and the method comprises the steps: inputting real-time multi-modal observation data and a current joint state into a pre-trained action prediction model, and generating an initial action prediction sequence containing a plurality of future time steps; constructing a space-time heterogeneous guide mask matrix by utilizing the current tail end linear speed and the execution state of a tail end operation part in the target robot; and correcting the initial action prediction sequence by using the space-time heterogeneous guide mask matrix and the historical action sequence queue to obtain a target action sequence, and sending the target action sequence to a controller of the target robot. By means of the method and device, the problems that in the track splicing process of a traditional method, a mechanical arm shakes, and an end effector is switched in an uncertain state are effectively solved, and the stability and response precision of target robot control are improved.

The invention discloses a robot control method and device, a controller, a storage medium and a computer program product. The robot control method and device are applied to a scene in which a robot is jointly controlled based on safety equipment and N control ends. The control method of the robot comprises the following steps: establishing communication connection with at least one control end which needs to control the robot in the N control ends; recording one control end which currently needs to control the robot in the at least one control end which establishes the communication connection as a current control end; acquiring an operation command sent by the current control end, and acquiring security information sent by the security equipment; and based on the operation command sent by the current control end and the safety information sent by the safety equipment, cooperatively controlling the robot to act. According to the scheme, the multi-terminal equipment, the safety equipment and the robot body are cooperatively controlled, so that the multi-terminal equipment simultaneously controls the cooperative work of the robot, and the programming efficiency and accuracy are improved.

The invention discloses a robot action generation method integrating multi-layer feature bridging and world knowledge prediction. The method comprises the following steps: extracting multi-layer middle layer visual features and action query hidden variables by utilizing a pre-trained visual language model; an initialization strategy based on robot body sensing state guidance is adopted, real-time pose priori is injected for action query, and traditional all-zero initialization is replaced; a spatial perception vector gating mechanism is introduced into the bridging attention module, and fine-grained selection of specific image region features is achieved; meanwhile, a world knowledge prediction task is integrated, and physical common knowledge is enhanced through explicit modeling environment depth, semantics and a dynamic region; and finally, replacing a traditional L1 regression action head with a diffusion model architecture, and generating an optimal action sequence under multi-modal distribution through an iterative denoising process. According to the method, the operation precision of the robot in a long-range complex task is improved, and the problem of control failure caused by an action'averaging effect 'is solved while the light weight of the model is kept.

The invention relates to the technical field of artificial intelligence, provides a double-arm robot control method, device and equipment and a medium, can be applied to financial and medical health care service scenes, and can utilize multiple types of sensors to collect current environment data so as to train a visual language action model, so that the model preliminarily has basic generalization ability; a feature extraction module of a double-arm robot action decision model is utilized to dynamically extract multi-modal features of real-time environment information based on an attention mechanism, so that multi-modal feature representation adapting to a dynamic environment is quickly captured, the perceptibility of the model to a new environment is improved, and the generalization ability of the model is improved; and inputting the task target, the constraint condition and the multi-modal features obtained through analysis into a strategy network of the double-arm robot action decision model to obtain an action sequence of the double-arm robot, and controlling the double-arm robot to execute the action sequence so as to realize accurate understanding and decision making of tasks in a new environment, and improve the adaptability of the model to the new environment.

The invention relates to the field of reinforcement learning, in particular to a reinforcement learning-based large language model antagonism fine-tuning enhancement system. The state prediction module is used for acquiring system state data and robot action data; based on the robot action data, generating a prediction state vector through a neural network prediction model, and comparing the prediction state vector with the system state data to obtain a residual vector; the action decoding module inputs the residual vector into a feedforward neural network, codes the residual vector to generate a natural language state report, and performs autoregression decoding through a large language model to obtain a loss value; the strategy optimization module converts a penalty function value into a reinforcement learning reward signal through a nonlinear function, and the PPO algorithm calculates the total loss according to the reinforcement learning reward signal; and calculating, optimizing and updating the gradient of the low-rank adaptive parameter in the large language model based on the total loss. According to the invention, through the feedforward neural network and the PPO algorithm, the tiny dynamic deviation is captured, and the predictability and safety of the system are improved.

The invention provides a training method and a control method of a surgical robot action prediction model, and the training method comprises the steps: obtaining a plurality of groups of sample data and an evaluation index of each group of sample data, each group of sample data comprising a user instruction, a robot state and a corresponding robot action; utilizing a preset reward function to calculate the reward of each group of sample data according to the evaluation index of each group of sample data; based on the multiple groups of sample data and the corresponding rewards, taking the maximization of the rewards as a target, and carrying out preliminary training on the surgical robot action prediction model; acquiring a real-time operation image, and constructing a virtual operation environment simulating a real operation scene based on the real-time operation image; in the virtual operation environment, the simulation operation robot executes candidate actions of different user instructions in different states, rewards of the candidate actions are calculated according to the reward function, the preliminarily trained model is adjusted through the rewards of the candidate actions, and a trained operation robot action prediction model is obtained.

The invention discloses a motion arrangement method and system for an intelligent robot with a body based on a large model and a knowledge base, and belongs to the technical field of artificial intelligence, and the method comprises the steps: constructing a meta-motion base and a meta-motion constraint RAG knowledge base of the robot; receiving a task instruction and generating an environment and state description text; retrieving in a meta-action constraint RAG knowledge base according to the task instruction and the generated environment and state description text, and generating an action constraint behavior list according to a retrieval result; generating a structured cue word based on the action constraint behavior list; analyzing the generated structured cue word by using a large language model (LLM) to obtain a composite action sequence draft; and checking the composite action sequence draft by using a preset empirical rule, and taking the qualified composite action sequence draft as a final robot action arrangement result. According to the invention, the success rate and compliance of LLM planning robot motion arrangement are improved.

The present application belongs to the technical field of embodied intelligent robots, and discloses a kind of embodied intelligent humanoid robot remote control data acquisition and data closed loop system, and the instruction, action, acquisition ternary time sequence synchronization mechanism is built by time sequence cooperation module, the dynamic cooperation of three is realized using time sensitive network technology, effectively reduce time sequence deviation, while combining the space-time alignment algorithm of data processing module, guarantee the consistency of multi-modal data in time and space dimension;Make the collected data can accurately fit human operation logic and robot action state, improve the optimization efficiency of data closed loop, successfully adapt to the demand of complex assembly, flexible material processing and other high-precision operation scene;Rely on evaluation module to establish the quantitative correlation system of data quality and control effect, can real-time filter invalid data such as sensor failure, action fuzzy, reduce the occupation of redundant storage and computing resources.

The invention relates to a training method of a perception and control model, and a robot control method and device. The method comprises the following steps: acquiring a plurality of sample image sets; wherein each sample image set comprises a depth image and an elevation image which represent the same terrain; performing multi-stage training on the initial model according to the plurality of sample image sets to obtain a sensing and control model for controlling the action of the robot; wherein the multi-stage training comprises the following steps of: training an elevation image codec by utilizing an elevation image to obtain a trained elevation image codec; training a depth image codec by using the depth image and the trained elevation image codec to obtain a trained depth image codec; and performing reinforcement learning training on the action strategy neural network by using the trained depth image codec, thereby facilitating the improvement of the model training efficiency through the interaction of perception and control, helping the robot to accurately perceive the working environment, and realizing accurate action control.

The application discloses a kind of based on behavior control's bionic robot action control method, it is related to technical bionic robot control field, including the following steps: establishing global terrain coordinate system, completing multi-sensor calibration, robot starts autonomous navigation task, vision sensor starts to collect surrounding environment image;Image is handled, and the color, texture feature in image is extracted;Interference object identification module identifies the interference object therein, and extracts its feature information;Interference index analysis module calculates interference index according to the feature information of interference object;Terrain environment complexity evaluation module real-time collection terrain environment information, evaluates the complexity of terrain environment;Action control decision module comprehensively interference index and terrain environment complexity evaluation result, makes action control decision, and sends control instruction to motion execution mechanism, the application solves the problem of bionic robot in complex environment motion intelligence and safety.

The application provides a robot action prediction method and device, equipment and a storage medium, wherein the method comprises: extracting global features and local features of an action sequence based on a reinforcement learning action prediction model, and predicting a predicted action sequence based on the extracted global features and local features. By extracting the global features and local features of the action sequence respectively, the continuity of the historical action sequence in time can be concerned while focusing on the local correlation of the action sequence. By extracting the features through the Mamba module, the selectivity and context awareness of the model can be further enhanced, thereby improving the performance of the model. In addition, since the Mamba module has a significant lightweight advantage, the action prediction model improved based on the Mamba module also has the advantages of being lightweight and easy to deploy.