277 results about "Action model" patented technology

The invention relates to the technical field of robot control, and discloses a robot control method, system, equipment and medium based on a multi-modal large model, and the method comprises the steps: collecting the multi-source modal data of a scene where an operation task is located, and carrying out the processing through a machine learning model, obtaining a multi-modal feature, and carrying out the position coding and Transform fusion processing, multi-modal fusion features are obtained, the multi-modal fusion features and the constructed job task knowledge base are input into a large language model to decompose a target job task, a human-in-the-loop mechanism is introduced to optimize a decomposition result, and a sub-task sequence is obtained; according to a subtask type in the subtask sequence, processing the subtask sequence through a visual language action model or a reinforcement learning model, and generating a motion instruction to enable the robot to start an execution process of the target operation task; live-line work tasks are processed through the multi-modal large models LLM, VLA and the like, and the work efficiency of the autonomous distribution network live-line work robot is improved.

The present disclosure provides a system for generating motor control commands for a humanoid robot, comprising an alpha model with over 1 billion parameters that processes visual observations and language instructions at a first frequency to generate contextual embeddings, and a beta model operating at a higher second frequency. The beta model includes an embodiment-specific state encoder projecting robot state information into a shared embedding space, a diffusion transformer module generating denoised action sequences through iterative flow-matching that cross-attends to the alpha model's contextual embeddings, and an embodiment-specific action decoder converting denoised sequences into motor control commands. The beta model generates action chunks comprising future action sequences over a predetermined time horizon in a single inference step, with the complete system having less than 5 billion parameters.

The present disclosure provides a humanoid robot comprising a torso having an alpha model deployed on a first GPU, and wherein said alpha model includes a first number of parameters and is configured to receive a natural language command from a human and generate processed data, a beta model deployed on a second GPU, and wherein said beta model includes a second number of parameters and is configured to receive the processed data from the alpha model and provide output data used to control an extent of the left wrist, and wherein the first number of parameters is larger than the second number of parameters, and a unified training framework is used to jointly train the alpha model and the beta model.

The invention relates to a three-dimensional live-action model generation method for city updating. The method comprises the following steps: collecting multi-source city image information, and obtaining semantic data according to the multi-source city image information; performing city modeling visualization according to the multi-source city image information to generate a city model; and matching the semantic data with the city model to obtain a three-dimensional real scene model containing the semantic data. According to the method, city information can be obtained more comprehensively by performing total factor accurate collection through a multi-source three-dimensional monitoring technology, the accuracy of a real scene model is improved, semantic data is matched with a city model, a three-dimensional real scene model containing the semantic data is obtained, multiple materials and textures of the model are combined, and the accuracy of the real scene model is improved. And the model is lightened, so that the purpose of reducing the resource loading time is achieved.

The invention discloses a flexible robot motion control system and method based on a visual language action model, and the system comprises visual sensors which are disposed at a plurality of joints of a flexible robot, and are used for observing the multi-joint sensing information of the flexible robot; the remote voice input unit is installed on the flexible robot body and used for inputting a voice instruction of remote operation to the flexible robot; the motion control module based on the VLA framework is mounted in the flexible robot and used for controlling the flexible robot to act based on a visual language action model according to the multi-joint sensing information and the voice instruction; through the design of the intelligent control system, the complexity of the system is remarkably reduced, the control precision and flexibility are optimized, the coordination control problem of complex body segment units is solved, high-performance autonomous motion control is achieved, the system cost is reduced, the adaptability of the robot in diversified environments is enhanced, and the robot can be widely deployed in complex application scenes.

The invention discloses a human-computer interaction method and system based on a vision-language-action model, and belongs to the field of human-computer interaction. According to the method, an anchoring ring strategy is adopted to collect human teaching data to finely adjust the VLA model, and then the trained model is applied to an actual human-computer interaction scene. In the data acquisition stage, a first operator guides a master robot to execute task actions, and a slave robot synchronously moves and interacts with a second operator, and returns to a predefined initial position after each interaction; a teaching sample is formed by recording a robot state, an environment image and an instruction text, and a high-quality data set is generated through data enhancement. In the application stage, the real-time robot state, the environment image and the instruction text serve as input, an action instruction is generated through the VLA model, and the robot is driven to complete a cooperation task. According to the method, the data utilization efficiency and the model generalization ability are remarkably improved, the difference between simulation and the real environment is effectively overcome, and efficient, safe and natural man-machine cooperation is achieved.

The invention relates to the field of personal intelligent security evaluation, and particularly discloses a chain reasoning hidden backdoor vulnerability detection method of a vision-language-action model, which comprises the following steps of: respectively injecting micro pixel disturbance and rare character marks into vision and language input; on the basis of model autoregression prediction characteristics, designing a hybrid reasoning chain fusing normal reasoning steps and abnormal backdoor branches; adopting prefix tuning to take the hybrid reasoning sequence as a pluggable prefix injection model; and generating the vulnerability sensitivity of the abnormal action instruction through the systematic verification process detection model. Compared with an existing method, the method has the advantages that a nondestructive testing mechanism based on prefix adjustment and optimization does not need to modify model parameters or depend on training data, and the safety and reproducibility of detection are guaranteed; a multi-mode triggering mechanism is constructed, and the hidden vulnerability of the model in a complex scene is effectively revealed; the abnormal branches and the normal process are fused in a chain mode, and the defense capability of the model for the concealment logic offset can be systematically evaluated.

The invention discloses a robot control method and system based on a bidirectional selection hybrid expert mechanism, a terminal and a storage medium. The method comprises the following steps: acquiring a task language instruction and a visual perception image of a target local data set; performing scene object classification according to each task language instruction, and segmenting each visual perception image to obtain a plurality of image tokens; performing token score calculation on the plurality of image tokens, performing expert activation score calculation according to the initial expert, and determining a target expert; obtaining an expert selection vector in each target client, and carrying out model parameter aggregation processing to obtain a federal visual language action model; and obtaining a target task instruction corresponding to the current task, and inputting the target task instruction to the federated visual language action model to enable the target robot to execute the current task. According to the method, federal learning and a two-way selection hybrid expert mechanism are combined, so that the privacy data of the user is protected, and the operation capability of the robot is remarkably improved.

The invention relates to a method, a device, equipment and a program product for controlling a robot to execute tasks. The method includes acquiring a user input for a robot and an image captured by a camera of the robot. The method includes generating a sequence of actions for the robot based on the user input and the image by a visual language action model trained by reinforcement learning for a world model. In addition, the method further comprises the step of controlling the robot to execute the task corresponding to the user input by executing the action sequence.

The invention relates to the field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses an enhanced fine tuning method, device and equipment for a visual language action model and a medium. The visual language action model is used for operating the robot to execute a corresponding action task according to the visual information and the language instruction; multiple pieces of demonstration data are collected, off-line reinforcement learning is conducted on the visual language action model through the demonstration data, and an off-line fine tuning model is obtained; deploying the off-line fine tuning model into an actual environment, and controlling the robot to interact with the environment according to a task reset strategy to obtain an exploration trajectory and environment feedback; and performing online reinforcement learning on the off-line fine tuning model according to the exploration trajectory, the environment feedback and the demonstration data to obtain a fine-tuned visual language action model. Cooperative model fine tuning is realized through staged enhanced fine tuning and a task reset strategy, and the fine tuning effect is improved to ensure the reliability of robot operation.

This disclosure describes a framework for performing user-requested tasks automatically across an interactive interface using various types of machine learning models. Specifically, this disclosure outlines and describes a task execution system that utilizes a generative artificial intelligence (AI) action model and retrieval-augmented generation (RAG) to complete user-requested actions across an interactive interface. The task execution system solves many of the current limitations of LAMs by using a generative AI action model to determine a session plan, which includes a set of actions for accomplishing stages of the actionable task across the interactive interface, obtaining visual context information of each interactive interface segment, integrates RAG results to improve the accuracy of both the session plan and individual actions, and self-corrects when faced with unexpected obstacles.

The invention provides a robot action generation method and device, and the method comprises the steps: responding to a target task received by a robot, and obtaining image data and text description information associated with the target task; on the basis of the image data and the text description information, potential action information and fused visual representation information are determined by utilizing a large language model obtained by supervised training provided based on a potential action model; compressing and splicing the image data, the potential action information and the fused visual representation information to obtain control sequence information; and de-noising and splicing the state information and the control sequence information corresponding to the robot to generate action sequence information, so that the robot executes a corresponding action based on the action sequence information. By means of the method, the smoothness and coherence of actions generated by the robot are improved.

The invention provides an experiment task execution method and device.The method comprises the steps that task information of a target experiment task is divided based on a visual language model, a subtask sequence is obtained, and the subtask sequence is formed by arranging multiple subtasks from front to back according to the execution sequence; processing the text information corresponding to the current sub-task and the experiment image before the current sub-task is executed through the visual language model from the first sub-task in the sub-task sequence to obtain a visual prompt image, and executing the visual prompt image through a visual language action model. And processing based on the text information, the experiment image and the visual prompt image, guiding a robot to execute the current sub-task until the last sub-task in the sub-task sequence is completed, and determining that the target experiment task is completed. According to the method, the success rate, the operation safety and the regulation compliance of the experiment task are effectively improved, and the method has high universality and safety.

The invention provides a robot control method based on a visual language action model and related equipment thereof, and relates to the technical field of intelligent robots, and the method comprises the steps: obtaining visual observation information, a target language instruction and robot body state information of a target robot; reasoning the visual observation information, the target language instruction and the robot body state information through a pre-trained visual language action model to obtain a predicted low-dimensional potential vector; reconstructing the predicted low-dimensional potential vector into a robot action instruction through a pre-trained high-dimensional action reconstruction module; wherein the dimension of the robot action instruction is matched with the action space dimension of the target robot; and the target robot is driven to execute the high-dimensional robot action instruction. According to the method, the limitation of the existing visual language action model on the action output dimension can be solved, so that the high-degree-of-freedom robot is effectively controlled.

The invention discloses a multi-modal interaction method and interaction system applied to an intelligent robot. The method comprises the following steps: collecting a scene image and processing the scene image into a three-dimensional point cloud and a two-dimensional texture feature; the Gemini Robotics-ER model is used for extracting features, and the vision-language-action model is used for analyzing a language instruction into a sequence capable of being recognized by a machine; and fusing the features to generate an interactive decision matrix, planning a trajectory, calculating kinetic parameters, driving the robot to execute actions and feeding back in real time. The system comprises a multispectral visual information acquisition and preprocessing unit, a Gemini Robotics-ER model processing unit, a natural language instruction analysis unit, a vision-language-action cooperative processing unit, a trajectory planning and dynamics calculation unit and a motion control and feedback unit, and all the units work cooperatively. According to the method and the system, through multi-modal fusion and closed-loop control, interaction accuracy and real-time performance are improved, and industrial scene requirements are met.

The invention relates to the field of artificial intelligence and medical treatment, and discloses a VLA model generation method and device for a surgical robot, equipment and a medium, and the method comprises the steps: inputting a first data subset into a to-be-trained visual language action model for training, and obtaining an initial visual language action model; inputting the first new operation task data into the initial visual language action model, and controlling the operation robot to output and execute an operation action according to the initial visual language action model; utilizing a preset reinforcement learning algorithm to optimize action head parameters of the initial visual language action model to obtain a first visual language action model; storing task trajectory data successfully completed by the surgical robot to an online data set; and inputting the second data subset and the task trajectory data into the first visual language action model for training to obtain a second visual language action model, thereby continuously performing reinforcement learning on the VLA model, enabling the surgical robot to adjust a surgical strategy according to real-time feedback, and further improving the surgical accuracy.

The invention belongs to the technical field of artificial intelligence, embedded systems and action confrontation command, and discloses an action model learning system and method based on embedded AI. The system comprises an information collection module used for collecting historical action information and carrying out data preprocessing, and the historical action information comprises action environment data, confrontation situation information and decision data; the feature extraction and model construction module is used for extracting features from the preprocessed historical action information by using a deep learning algorithm to obtain action dynamic change features and enemy action mode features, and matching the action dynamic change features and the enemy action mode features with corresponding decision data results to form a training set; training the embedded AI model by using the training set to generate an action model; according to the invention, dynamic data from actions can be processed and learned in real time, so that the action model is continuously optimized, and the command decision-making capability is improved.

The invention provides an inertial sensing range and sensitivity adjusting method, which belongs to the technical field of parameter adjustment, and comprises the following steps: after an item opening instruction is obtained, obtaining basic information of a moving body, and calling an adaptive sensing range according to the basic information; different detection nodes are set in the whole process before a project is started, the feedback degree of a moving body is recorded in sequence when execution is carried out to each node, values of corresponding positions of an internal curve are increased or decreased according to the feedback degree, and corresponding sensitivity is dynamically set according to the internal curve. The feedback speed and external performance of the moving body are judged in the project execution process, and the follow-up content and links of the project are adjusted in combination with the action model library till the project is completed or closed. The inertial sensing range and sensitivity adjusting method provided by the invention can carry out specific dynamic adjustment according to the state of the moving body and the actual situation, has strong pertinence and ensures smooth completion of a project.

The embodiment of the invention provides a method for training an action expert model, a robot control method and electronic equipment, and relates to the technical field of artificial intelligence and robots. A first action expert model used for predicting motion direction information when the robot executes the to-be-processed task and a second action expert model used for predicting action detail information when the robot executes the to-be-processed task are trained; the training process of the first action expert model and the second action expert model is layered; moreover, the second action expert model can be trained by means of the partial training data of the first action expert model, so that the first action expert model can be utilized to provide directional guidance for the second action expert model to generate an executable action sequence, and thus, the operation efficiency is improved. The accuracy of generating an executable action sequence by a visual language action model can be improved, and then the accuracy of executing an operation task by a robot is improved.

The present disclosure provides a humanoid robot comprising a torso having an alpha model deployed on a first GPU, and wherein said alpha model includes a first number of parameters and is configured to receive a natural language command from a human and generate processed data, a beta model deployed on a second GPU, and wherein said beta model includes a second number of parameters and is configured to receive the processed data from the alpha model and provide output data used to control an extent of the left wrist, and wherein the first number of parameters is larger than the second number of parameters, and a unified training framework is used to jointly train the alpha model and the beta model.

Apparatuses, systems, and techniques are disclosed for controlling a robot to execute a task. In at least one embodiment, a current image of the robot in an environment and a text describing the task are obtained. A future image of the robot in the environment is predicted based on the current image and the text. Subsequently, one or more actions are predicted based on the current image, the future image, and the text. The one or more actions can move the robot from a first state corresponding to the current image to a second state corresponding to the future image. The robot executes the sequence of actions to move in the environment.

The invention discloses a lightweight method and system for a highway engineering three-dimensional live-action model and a medium, and relates to the technical field of three-dimensional live-action modeling. The method comprises the following steps: firstly, carrying out engineering semantic segmentation on an input three-dimensional grid model, and extracting design parameters in a BIM / GIS (Basic Information Model / Geographic Information System); then, constructing a multi-level constraint model fusing semantic constraints, engineering precision constraints and design parameter constraints; and finally, carrying out iterative simplification under the guidance and limitation of the multi-level constraint model by adopting an improved quadratic error measurement algorithm. The method solves the problems of semantic information loss, out-of-control local precision and disjunction with design intention when a traditional pure geometric simplification method is applied to highway engineering, and can intelligently reserve key engineering characteristics, control geometric errors and fit a design form while ensuring that the data size of the model is greatly reduced. And a high-quality lightweight model suitable for professional analysis and application is generated.

The present disclosure provides a humanoid robot system comprising a mechanical structure with at least 30 degrees of freedom across torso, arms, and legs, actuators driving the degrees of freedom, sensors including cameras and proprioceptive sensors, and a computing system implementing a hierarchical bipedal action model (BAM). The BAM includes: a Delta model processing sensor data and user input to generate latent representations at a first frequency; a Gamma model receiving latent representations to generate human task actions at a higher second frequency; a Beta model translating task actions into joint configurations at a higher third frequency; and an Alpha model converting joint configurations into actuator control signals at a higher fourth frequency.

The invention discloses a badminton player action posture analysis system and method based on multi-sensor fusion, and relates to the technical field of athletic training auxiliary systems.The system comprises a wearable sensing subsystem, an environment sensing subsystem and a central processing and feedback subsystem; the wearable sensing subsystem is arranged on an inertia measurement unit and a pressure sensing insole of a body to collect movement inertia and plantar pressure data; the environment perception subsystem collects global videos through multiple cameras. The central processing and feedback subsystem performs synchronous processing on multi-source data, adopts a hierarchical fusion algorithm, restrains inertia integral drift by utilizing a plantar contact state, reconstructs a three-dimensional skeleton posture and a motion trail in combination with visual key points and skeleton restraint, drives a digital twin model, compares with a standard motion model, and performs three-dimensional motion control on the three-dimensional skeleton posture and the motion trail. Quantitative evaluation results such as joint angle deviation, time sequence difference and pressure center track are generated, and visual feedback is performed through a display device and an augmented reality terminal for badminton training evaluation and technical deviation correction.

The invention relates to the technical field of artificial intelligence, intelligent and robot control, and discloses an emotion service robot system based on a multi-mode mental theory, which comprises a hierarchical mental reasoning module, a connection module and an action semantic execution module, the hierarchical mental reasoning module is used for generating a decision text containing a high-level strategy according to the multi-modal environment information collected by the robot; the connection module is used for constructing a semantic instruction according to the decision text generated by the hierarchical mental reasoning module; and the action semantic execution module is used for generating a control action of the robot according to the semantic instruction constructed by the connection module and the real-time image acquired by the robot. According to the method, a control architecture fusing a vision-language-action model and a robot center view angle mental reasoning hierarchy is constructed, so that the robot can carry out multi-order belief reasoning and implicit target inference from the view angle of the robot. According to the system, the robot is no longer a pure instruction follower and has an active service capability.

The invention discloses a wearable mechanical arm and an intelligent method thereof, relates to the field of artificial intelligence, and aims to solve the problems that a current wearable mechanical arm is insufficient in intelligence and limited in environment perception capability. According to the visual language information of the target task, the first action track is matched from the first database, or the visual language information is input into the visual language action model to obtain the second action track, then the mechanical arm action instruction is generated according to the obtained action track to be executed, and therefore the mechanical arm is driven to execute the target task. The system has the characteristics of high intelligent degree, strong self-adaption and strong environment perception capability.

The invention provides a model training method, a storage medium, electronic equipment and a program product, and relates to the technical field of deep learning. The method is used for training a visual language action model, and the model comprises a visual language module and an action expert module. The model training method comprises the following steps: acquiring action training data, wherein the action training data comprises an environment image, a task instruction and a real action sequence; a visual language module is used for processing the environment image and the task instruction to obtain a first fusion feature vector, and the first fusion feature vector comprises visual features and language features; applying an attention mask to the target visual features in the first fusion feature vector by using an action expert module to obtain a second fusion feature vector, and generating a predicted action sequence based on the second fusion feature vector; and adjusting parameters of the visual language action model based on the difference between the predicted action sequence and the real action sequence.

The invention discloses a hinged object robot grabbing method based on a visual language action model. A hinged object is put in a meta-universe environment for rendering, a geometric center point of the hinged object is used as a core reference point, six instance representation methods are adopted to mark the hinged object, and a sample database is established; generating six training tasks based on six instance representation methods; constructing a visual language action model based on LLaMA2 and SPHINX-X, and performing supervised training on the visual language action model according to a training task and a sample database to obtain an optimal visual language action model; constructing a joint optimizer, and selecting an optimal result from output results of the visual language action model by decomposing a hinged object manipulation sequence planning task; and S3, using the optimal visual language action model obtained in the step S3 and a joint optimizer to determine the hinged object robot grabbing method.

The invention discloses a vision-language-action model training method and system utilizing inference data closed-loop optimization, and belongs to the technical field of artificial intelligence. The method comprises the following steps: firstly, performing initial training on a vision-language-action model by utilizing a training data set; deploying the trained model in a task environment to execute a task, monitoring a task execution result in real time, and capturing and structurally recording failure track data of the current task when task execution failure is recognized; then, based on the captured failure trajectory data, generating a negative prompt text for guiding the model to avoid repeated error behaviors; and finally, combining failure trajectory data with the generated negative prompt text, retraining the vision-language-action model, and inhibiting the model from generating action output similar to the error action sequence. According to the method, the data utilization rate is greatly improved, the data dependence and acquisition cost are effectively reduced, and the complete closed loop of the VLA model training and reasoning process is realized.

The present disclosure provides a humanoid robot system comprising a mechanical structure including a torso, two arms, and two legs providing at least 30 degrees of freedom, actuators coupled to the degrees of freedom, a sensor suite comprising at least one camera and proprioceptive sensors including joint encoders and an inertial measurement unit, a computing system comprising at least one processor and memory storing instructions which, when executed, implement a hierarchical bipedal action model including a Beta model configured to receive multimodal input data and generate a token sequence indicative of task intent and environmental state, and an Alpha model configured to condition on the token sequence and current robot pose data to output continuous action chunks comprising sequences of future target joint states over a finite horizon, and a low-level controller configured to convert the continuous action chunks into actuator control signals for execution.