269 results about "Action prediction" patented technology

The invention provides an agent visual language navigation method and system based on task completion prediction. The method comprises the step of constructing a dual-drive structure composed of a self-adaptive mixed pooling mechanism and a task completion analysis module. Firstly, in the visual information processing process, a dynamic weight distribution strategy is adopted to carry out multi-scale adaptive mixed pooling on panoramic features, so that the fusion effect of local and global semantic information is optimized, and the retention capability and semantic integrity of navigation historical information in a dynamic topological map are improved. And then, inspired by a human navigation cognitive behavior mechanism, a task completion analysis module is designed and introduced, and the task execution progress is dynamically estimated based on the recognition condition of a key landmark in a navigation path, so that an intelligent agent is driven to preferentially select a key path node and invalid exploration is reduced. And finally, realizing efficient understanding and execution of the natural language instruction by the intelligent agent through a multi-round cyclic cross-modal reasoning and action prediction mechanism.

The invention belongs to the technical field of robot operation and control, and discloses a robot action generation method and a related device. The robot action generation method comprises the steps of obtaining a task instruction, ontology perception of a to-be-operated robot at the current moment, a multi-view RGB-D image and an initialized current noise action; and utilizing the trained action consistency diffusion strategy model to perform iterative action denoising according to the obtained task instruction, the ontology perception of the to-be-operated robot at the current moment, the multi-view RGB-D image and the initialized current noise action, and generating the action of the next step of the to-be-operated robot. According to the technical scheme disclosed by the invention, the technical problems of high training modeling difficulty, insufficient action prediction precision, low action generation speed and the like of the existing robot action generation technology in a high-precision operation task of the robot can be solved.

The invention discloses a visual language navigation method based on memory driving, and the method comprises the following steps: S1, constructing an MDQT model which comprises a panoramic encoder, a text embedding layer, a Q-Former, an action prediction module and a memory updating module; s2, training an MDQT model by using three pre-training tasks of image-concerned mask language modeling, instruction track matching and instruction track contrast learning; and S3, performing fine tuning on the MDQT model by using imitation learning and reinforcement learning. According to the invention, a learnable memory vector with a fixed length is used to encode historical information. In each navigation step, the memory vector interacts with the extracted panoramic information and instruction information, and visual information most relevant to the current instruction information is extracted according to the current memory state of the robot for decision making. The memory of the robot to the historical navigation steps is effectively maintained under limited resources, and the navigation success rate and efficiency of the robot are improved.

The invention discloses a progressive end-to-end trajectory planning method and system based on BEV features, and the method comprises the steps: extracting rich scene features of an online mapping module, a three-dimensional occupation module, an action prediction module and the like through multi-module parallel pre-training, and generating BEV features fused with comprehensive information; then, a lightweight trajectory planner is constructed, the planner performs trajectory planning only by depending on the BEV features, a probability planning method is introduced to deal with uncertainty in planning, probability distribution of actions is output, and one action is sampled from the probability distribution to control the vehicle; the system comprises a storage medium and a processor, and is used for operating according to the instruction to execute the method. According to the method, by simplifying the model structure, interference among multiple tasks is avoided, and the operation efficiency and the trajectory planning precision are improved.

The invention provides a multi-mode intelligent control method and device for a robot body and electronic equipment. The multi-mode intelligent control method comprises the following steps: acquiring multi-view images and task texts of cameras at multiple parts of the robot; generating two-dimensional track supervision information and key point supervision information based on the two, and converting the two-dimensional track supervision information and the key point supervision information into a visual supervision graph; key point features are generated through the supervision graph, and track features are generated in combination with the multi-view image; extracting a depth image, optical flow information, a matched segmentation mask, an execution mechanism attitude and a motion frequency, and generating corresponding features; inputting each feature into a multi-modal fusion reasoning model, generating a relative action prediction feature and converting the relative action prediction feature into an action control instruction; and controlling an execution mechanism to complete the task. According to the method, track supervision, key point supervision and multi-modal prior are introduced under the vision-language condition, and perception integrity, robustness and execution precision are remarkably improved.

The invention provides a visual language navigation and model training method, device, equipment and medium, and the method comprises the steps: obtaining a training sample, processing a to-be-executed instruction and environment information in the training sample through a core model, obtaining a navigation feature corresponding to the to-be-executed instruction, and inputting the navigation feature into an action prediction head. Obtaining predicted action information corresponding to the navigation features, and inputting the navigation features into a thinking chain generation head to obtain predicted thinking chain information corresponding to the navigation features; and then, based on the predicted action information, the reference action information in the training sample, the predicted thinking chain information and the reference thinking chain information in the training sample, training the core model, the action prediction head and the thinking chain generation head so as to take the trained core model and the trained action prediction head as a visual language navigation model. According to the method, the reasoning efficiency and the reasoning accuracy can be improved.

The invention provides a virtual object processing method and device, equipment, a computer program product and a computer readable storage medium. The method comprises the following steps: acquiring text description information of an image of a virtual scene at a current time step; obtaining semantic understanding information of the virtual scene in the previous time step; encoding the text description information and the semantic comprehension information of the previous time step to obtain the semantic comprehension information of the virtual scene at the current time step; acquiring an action parameter of the virtual object in the previous time step, and acquiring feature information of the virtual object in the current time step; action prediction processing is carried out through the semantic understanding information of the current time step, the feature information of the current time step and the action parameters of the previous time step, the action parameters of the virtual object in the current time step are obtained, and the action parameters of the current time step are used for driving the virtual object to execute actions in the current time step. According to the method and the device, the virtual object can be intelligently and automatically controlled.

The invention provides a time sequence knowledge graph-based target action prediction method, which comprises the following steps of: acquiring multi-source data which comprises a plurality of entities; based on a relationship classification model, obtaining an association relationship between the entities, and completing redundancy check and alignment de-duplication of the association relationship, the relationship classification model being obtained by classification learning; after target entities in the multiple entities are determined according to the positions and the feature information of the entities, an association tree of association relationships among the target entities is dynamically constructed to complete construction of the time sequence knowledge graph; based on the time sequence knowledge graph, modeling and classifying the time sequence nonlinear incidence relation between the target entities by using a deep neural network to generate a target space evolution model; and predicting a behavior pattern and a motion track of the target entity based on the target spatial evolution model.

The invention belongs to the technical field of artificial intelligence, and relates to a behavior prediction method and system based on a generative visual large model. The method comprises the steps of collecting human body action video data, and extracting human body behavior action data; preprocessing the data; constructing a generative visual large model, performing feature extraction, and performing time sequence analysis by using a long-short-term memory network; performing end-to-end training, calculating a loss function, and obtaining a personnel behavior prediction model; and outputting prediction results of the human body behavior actions in a plurality of future time frames. According to the method, high-level features can be extracted by adopting the generative visual model, so that the efficiency and accuracy of feature extraction are improved; a long-short-term memory network is adopted to perform time sequence analysis on the feature vectors, so that the accuracy and stability of action prediction are improved, the dynamic change of the action can be captured, and the future action trend can be predicted; the performance of the whole model can be optimized by adopting an end-to-end training method, the efficiency and robustness are improved, and the method adapts to variable practical application environments.

Implementations are provided for leveraging training data that is less costly to collect than state-action sequences to perform semi-supervised training of robot control policies. In various implementations, a first input prompt may be assembled with representations of an observed initial state of a robot and a goal state of the robot. The first input prompt may be processed using a goal-conditioned trajectory model to generate first output indicative of a sequence of predicted states to be reached by the robot between the observed initial and goal states. A second input prompt may be assembled to include representations of the sequence of predicted states. The second input prompt may be processed using an action prediction model to generate second output indicative of a sequence of predicted actions to be performed by the robot to reach the sequence of predicted states.

Aspects of the present disclosure provide techniques for machine learning based action prediction. Embodiments include providing, as inputs to a recurrent neural network (RNN), an ordered sequence of strings representing actions performed by a user within a software application, the RNN having been trained through a supervised learning process to generate embeddings of the ordered sequence of strings and generate a numerical score relating to a target action based on the embeddings and an order of the ordered sequence of strings. Embodiments include providing, as respective inputs to a tree-based classification machine learning model, the numerical score and an additional feature relating to the user and receiving, as a respective output from the tree-based classification machine learning model in response to the respective inputs, a propensity score indicating a likelihood of the user to perform the target action.

The invention provides a nuclear decommissioning operation master-slave manipulator control system and method. The system comprises an acquisition module for acquiring real-time attitude data, corresponding communication delay data and hot cell environment data of a master manipulator and a slave manipulator respectively; the data processing module preprocesses the collected data, extracts real-time posture standard feature information, communication delay standard feature information and hot cell environment feature data of the master and slave manipulators, and stores the information into a motion information database; the action prediction module constructs a slave manipulator track prediction model based on a neural network, and outputs motion action track prediction information of a slave manipulator; according to the system, in a nuclear decommissioning hot cell environment, real-time attitude data, communication delay data and hot cell environment data of a master manipulator and a slave manipulator are collected, and a high-precision slave manipulator track prediction model is constructed in combination with a neural network, so that the environmental adaptability of the system is enhanced, and the following precision and stability of the slave manipulator to the action of the master manipulator are improved; and accurate synchronous operation of the master and slave manipulators in a complex environment is ensured.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an action queue control method, device, equipment and medium, and the method comprises the steps: obtaining initial observation data, sending the initial observation data to a strategy server, and receiving a generated initial action queue; executing the action in the action queue; when the residual amount of the queue is lower than a threshold value, obtaining new observation data; judging whether action prediction is triggered or not based on the similarity between the data and historical observation data; if yes, the new observation data are sent to the strategy server in a non-blocking mode, and a new action block is generated; and receiving a new action block, integrating the new action block with the action queue to form an updated action queue, and repeatedly executing the updated action queue as a new action queue until all actions are completed. According to the method, the action blocks are generated in parallel in the action execution process in an asynchronous reasoning mode, control blind areas and waiting delay are reduced, response timeliness and continuous control capacity are improved, and the requirement of edge equipment for resource efficiency is met.

The invention provides a dynamic sports data visualization system and method, and relates to the technical field of data visualization analysis. Competition behavior data of athletes in a sports event are captured in real time; determining a plurality of exercise interaction nodes of the athlete in the current exercise item, and determining the action evolution trend of each exercise interaction node according to the action tendency track of the athlete; performing feedback evaluation on the real-time state data to obtain a motion feedback strategy of the athlete, and determining an action linkage attribute of the athlete according to the motion feedback strategy and the motion posture information of the athlete in the current sports event; and inputting the action evolution trend and the action linkage attribute into a preset action prediction model for real-time training, and visually displaying the output action prediction data. According to the method and the device, the dynamic sports data of the athletes can be subjected to linkage analysis in a data environment with real-time capture capability and a multi-modal perception basis, so that the effectiveness of collaborative interaction sports dynamic data visualization processing is improved.

The invention belongs to the technical field of data processing, and provides a vector building element capping conflict processing method based on deep reinforcement learning, which comprises the following steps: determining and constructing a constraint Delaunary triangulation network and a skeleton line segment between vector building elements in a target space region, and generating a space conflict detection field model; connecting each vector building element with center points of other surrounding vector building elements, and determining abnormal vector building elements with gland conflicts in combination with a space conflict detection field model; determining the current state information of the abnormal vector building elements, performing data grouping according to the area scale of the abnormal vector building elements so as to set constraint condition parameters and action reward information of each group, and obtaining action prediction information in combination with a DQN model; and according to the action prediction information, controlling the abnormal vector building elements to execute the position movement action until the optimal position. According to the scheme provided by the invention, the processing efficiency, precision and feasibility of the capping conflict are improved.

The invention relates to the technical field of intelligent control of a tool, in particular to an intelligent control method, device and equipment for a tool based on a visual language and a storage medium. Comprising the following steps: acquiring an original image, and obtaining a control text instruction; inputting the original image and the control text instruction into a visual language model, and determining target coordinate information of an execution target from the original image based on the control text instruction through the visual language model; based on the target coordinate information, determining an execution area containing an execution target, and performing mask processing on the original image according to the execution area; and inputting the mask image obtained after processing and a preset input feature set into an imitation learning model, and outputting action prediction based on the mask image and the preset input feature set through the imitation learning model so as to control the intelligent device with the body to execute a corresponding action. According to the method, the input of subsequent action prediction is simplified through mask processing, so that the flexibility and the response speed of the intelligent equipment with the body are improved.

The invention discloses a fall and turn-over control method and device of a foot type robot and a storage medium. The fall and turn-over control method comprises the steps of firstly obtaining fall posture information of the foot type robot; calling an action selection model to perform multi-stage iterative turnover action prediction based on the falling posture information to obtain an optimal turnover action of each iteration stage, controlling the foot type robot to execute the optimal turnover action of the current iteration stage in each iteration stage, and controlling the foot type robot to execute the optimal turnover action of the current iteration stage after the foot type robot executes each optimal turnover action. The foot-type robot is in a stable posture when the foot-type robot turns over, and the foot-type robot turns over successfully after executing the optimal turning-over action of the last iteration stage. Wherein when the action selection model is called to carry out multi-stage iterative turning-over action prediction based on the falling posture information, the action selection model predicts the optimal turning-over action of the next iteration stage according to the stable posture of the foot robot in the current iteration stage. According to the embodiment, the manual design cost can be reduced, and the applicability is high.

The invention provides a training method of an action prediction model and an action prediction method. According to the main technical scheme, the method comprises the steps of obtaining training data comprising a plurality of training samples; the training sample at least comprises an instruction received by the robot at a first time point, a collected image and an action state, and an action executed at a second time point; identifying whether a preset type of key action exists between the first time point and the second time point; training an action prediction model based on the training data; the training comprises the steps of inputting an instruction received at a first time point, a collected image and an action state into an action prediction model, if a key action of a preset type is identified, taking the key action of the preset type as a prediction target of the action prediction model, otherwise, taking the key action of the preset type as a prediction target of the action prediction model; and taking the action executed at the second time point as a prediction target of the action prediction model. The execution speed and accuracy of the robot action can be improved.

A system may access a first set of images captured by cameras coupled to a shopping cart, wherein each image depicts a user associated with the shopping cart. A system may apply a pose detection model to each of the images to predict a user's pose. A system may apply an action prediction model to the set of images and the predicted poses to predict whether the user performed an action to change the contents of a storage area of the shopping cart. A system may, responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action. A system may update an item list of the user based on the change action and the identified item.

The invention provides a universal dexterous hand action redirection method based on modular residual reinforcement learning, and the method comprises the steps: S110, receiving MANO parameters subjected to structural splitting through a finger sub-strategy network, and outputting initial action prediction; s120, simultaneously receiving the initial motion prediction and the complete MANO parameter representation as input through a residual error coordination network, correcting the local prediction of each finger under the guidance of the global hand intention, and outputting residual error correction, so as to obtain a final motion; s130, executing two-stage training: in the first stage, training each finger sub-strategy network in parallel; in the second stage, parameters of the finger sub-strategy network are frozen, and only the residual error coordination network is trained. According to the technical scheme, the real-time performance of a learning type method and the accuracy of an optimization type method are both achieved, data dependence is reduced, the generalization ability is improved, and meanwhile the equipment universality and action flexibility are both considered.

In accordance with an embodiment, described herein are systems and methods for auto-completion of ICS flow using artificial intelligence / machine learning. Next actions prediction is a service that assists users in modeling the flows quickly by predicting and suggesting the next set of actions a user might be thinking of adding. The service also assists the user to follow some of the best practices while creating an integration flow.

The invention relates to the technical field of three-dimensional scene modeling, in particular to a dynamic event scene reproduction method and system and electronic equipment, and the method comprises the steps: collecting scene multi-modal data, and generating a three-dimensional static scene model; generating a static object supplementary description model and a dynamic object motion prediction model by using a large language model; fusing the text semantic features and the image visual features, and integrating with a dynamic object motion prediction model to generate a total behavior prediction model; motion planning and real-time simulation are carried out on a virtual role, and a virtual role model with intelligent behavior ability is output; combining the three-dimensional static scene model, the static object supplement description model, the total behavior prediction model and the virtual role model to generate a complete dynamic event scene; according to the invention, through the innovative architecture of dynamic and static separation modeling and large language model driving, the bottleneck of static property and manual dependence of the traditional scene modeling technology is broken through, and efficient and real dynamic scene automatic reproduction is realized.

The invention provides an automatic task-independent data acquisition method for a body-equipped agent, a training method and device for an action prediction model, equipment and a storage medium, and relates to the technical field of body-equipped agents. The task-independent data acquisition method comprises the following steps: defining a 3D position feasible range of an end effector as a bounded working space cube; training the reinforcement learning network through a near-end strategy optimization algorithm; randomly sampling a target point as a target point 3D position in the workspace cube, generating a corresponding joint position of the mechanical arm by using a reinforcement learning network, and constructing position mapping; the method comprises the following steps: driving an end effector of the mechanical arm to perform random sampling in an actual space, recording a sampling image obtained when the end effector reaches a 3D sampling position in the random sampling process of the end effector, and taking the sampling image and a corresponding joint position as task irrelevant data of the mechanical arm. The data collection process is simpler and is easy to expand, and cross-task multiplexing of data is supported.

The invention provides a robot action prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting multiple pieces of visual information of a robot and task execution instruction information into a first network layer of a pre-trained robot action prediction model for key point prediction processing, outputting key point features of the operated object grabbed by the robot; wherein different visual information comes from different mechanical arms of the robot; and performing action prediction processing on current mechanical arm state information, multiple pieces of visual information, task execution instruction information and the key point features based on a second network layer of the robot action prediction model, and outputting an action sequence of the robot. The key point features of the clamping jaw on the mechanical arm of the robot and the operated object are accurately determined through the robot action prediction model, and then action prediction processing is conducted through the key point features to improve the accuracy of robot action sequence determination.

The invention provides a robot action sequence generation method and device, equipment and a medium, and the method comprises the steps: inputting real-time multi-modal observation data and a current joint state into a pre-trained action prediction model, and generating an initial action prediction sequence containing a plurality of future time steps; constructing a space-time heterogeneous guide mask matrix by utilizing the current tail end linear speed and the execution state of a tail end operation part in the target robot; and correcting the initial action prediction sequence by using the space-time heterogeneous guide mask matrix and the historical action sequence queue to obtain a target action sequence, and sending the target action sequence to a controller of the target robot. By means of the method and device, the problems that in the track splicing process of a traditional method, a mechanical arm shakes, and an end effector is switched in an uncertain state are effectively solved, and the stability and response precision of target robot control are improved.

The invention provides a visual language action model training method and device, equipment and a storage medium, and the method comprises the steps: extracting a corresponding two-dimensional visual feature and a four-dimensional visual feature based on an environment image, carrying out the multi-modal feature fusion of the two-dimensional visual feature, the four-dimensional visual feature, the language feature of a natural language instruction, and the state feature of state data, and carrying out the multi-modal feature fusion of the two-dimensional visual feature and the four-dimensional visual feature. Motion prediction and visual feature reconstruction are carried out based on multi-modal fusion features, model parameter updating is carried out by using generated multi-modal output data, and the model has the capability of extracting spatio-temporal dynamic information under the conditions of not increasing additional sensors and ensuring that the overall parameter quantity of the model is small. The time and space reasoning performance of the visual language action model in the dynamic environment can be improved, so that the actuator can be accurately controlled to execute the natural language instruction in the dynamic environment in the model reasoning stage.

The invention discloses a time sequence dynamic gesture recognition method and system. The time sequence dynamic gesture recognition method mainly comprises the steps of S1, hand target detection; step S2, hand target tracking; s3, outputting category information; s4, setting an information container; and S5, gesture motion prediction. According to the method, the problem of time sequence prediction of gesture actions is simplified, meanwhile, the problem of easy ID exchange during target tracking is solved, YOLO is mainly used for detecting a hand and positioning key point information of a hand skeleton, a neural network is realized based on a 1D operator, and gesture action prediction is carried out on time sequence key point information. According to the dynamic gesture recognition method based on combination of the multiple models, lightweight design is carried out on the YOLOv8 model, real-time recognition is achieved on equipment with limited resources (for example, the real-time recognition can reach 25FPS on RK3566), and the problem that ID information is likely to be exchanged when hands are waved crosswise is solved through dynamic threshold control parameters of the target tracking module.

The invention discloses an off-line reinforcement learning action exploration agent method based on expected reward regularization, and belongs to the field of reinforcement learning. The problem that reliable track splicing and strategy generalization are difficult to realize in a complex task by the existing method is solved. The method comprises the following steps: constructing state loss and action loss based on sequence modeling, and carrying out iterative training of states and actions; designing an RTG loss function based on a weighted square error; a double-Q learning framework is adopted to maintain two independent Q functions, conservative constraints of conservative Q learning are applied in the Q function updating process, and action exploration optimization is achieved in combination with Boltzmann distribution; combining state loss, action loss, RTG regularization loss and Q value loss to form a joint optimization objective function; performing noise disturbance sampling on the plurality of RTG candidate values to generate diversified action prediction; and value evaluation is performed on the candidate actions based on a double-conservative Q function, and the action with the highest Q value is selected for execution. The method is mainly used in the intelligent agent exploration field.

The invention provides an on-line motion prediction method based on space-time consistency information decoupling and integration, and the method comprises the steps: obtaining a video stream, and extracting an RGB image and an optical flow field image from the video stream; and inputting the RGB image and the optical flow field into a trained ST-Mama model to obtain a prediction action of a first frame of the video stream. According to the method, the Mama is introduced into the online action detection task, so that the long sequence action detection capability is improved, the computing resource consumption during reasoning is greatly reduced, and the OAD requirement with high real-time performance is met. Moreover, a plurality of scanning strategies are introduced into the decoupling modules for feature supplementation and enhancement, so that the perception capability of the model to a complex process can be effectively improved. Besides, the semantic expression of the integrated module is enhanced by means of historical frame information, and progressive optimization of action features is completed through double Mama branches, so that the recognition ability of the model to the current state and the memory ability of the model to the context are enhanced.

A database system accesses historical data, including characteristics of employees before their departure from past employers. It then generates a normalized training set based on this data and geographic details of each entity. Using this set, the system trains a neural network to forecast employee exits from current employers. The system applies this model to predict the departure dates for specific employees from a target company. The system updates the training set with both predicted and actual departure dates, improving the neural network through a second stage of retraining.