316 results about "Action prediction" patented technology

The invention provides an agent visual language navigation method and system based on task completion prediction. The method comprises the step of constructing a dual-drive structure composed of a self-adaptive mixed pooling mechanism and a task completion analysis module. Firstly, in the visual information processing process, a dynamic weight distribution strategy is adopted to carry out multi-scale adaptive mixed pooling on panoramic features, so that the fusion effect of local and global semantic information is optimized, and the retention capability and semantic integrity of navigation historical information in a dynamic topological map are improved. And then, inspired by a human navigation cognitive behavior mechanism, a task completion analysis module is designed and introduced, and the task execution progress is dynamically estimated based on the recognition condition of a key landmark in a navigation path, so that an intelligent agent is driven to preferentially select a key path node and invalid exploration is reduced. And finally, realizing efficient understanding and execution of the natural language instruction by the intelligent agent through a multi-round cyclic cross-modal reasoning and action prediction mechanism.

The invention belongs to the technical field of robot operation and control, and discloses a robot action generation method and a related device. The robot action generation method comprises the steps of obtaining a task instruction, ontology perception of a to-be-operated robot at the current moment, a multi-view RGB-D image and an initialized current noise action; and utilizing the trained action consistency diffusion strategy model to perform iterative action denoising according to the obtained task instruction, the ontology perception of the to-be-operated robot at the current moment, the multi-view RGB-D image and the initialized current noise action, and generating the action of the next step of the to-be-operated robot. According to the technical scheme disclosed by the invention, the technical problems of high training modeling difficulty, insufficient action prediction precision, low action generation speed and the like of the existing robot action generation technology in a high-precision operation task of the robot can be solved.

The invention discloses a visual language navigation method based on memory driving, and the method comprises the following steps: S1, constructing an MDQT model which comprises a panoramic encoder, a text embedding layer, a Q-Former, an action prediction module and a memory updating module; s2, training an MDQT model by using three pre-training tasks of image-concerned mask language modeling, instruction track matching and instruction track contrast learning; and S3, performing fine tuning on the MDQT model by using imitation learning and reinforcement learning. According to the invention, a learnable memory vector with a fixed length is used to encode historical information. In each navigation step, the memory vector interacts with the extracted panoramic information and instruction information, and visual information most relevant to the current instruction information is extracted according to the current memory state of the robot for decision making. The memory of the robot to the historical navigation steps is effectively maintained under limited resources, and the navigation success rate and efficiency of the robot are improved.

The invention discloses a driving track prediction method and device based on sparse BEV features, equipment and a medium, and relates to the technical field of vehicle simulation, and the method comprises the steps: obtaining driving scene information, carrying out the mapping according to the driving scene information, obtaining sparse vehicle features, carrying out the interaction prediction according to the driving scene information, and obtaining motion prediction features, according to the action prediction features and the sparse automobile features, enhanced automobile features are obtained, and according to the enhanced automobile features, a self-automobile prediction trajectory is obtained. Through obtaining the scene information, generating the sparse features through mapping and obtaining the predicted action features, the two features are fused to obtain the enhanced automobile features, the self-vehicle trajectory prediction is realized according to the enhanced automobile features, the driving trajectory prediction efficiency and accuracy are improved, and the collision risk is reduced.

The invention discloses a progressive end-to-end trajectory planning method and system based on BEV features, and the method comprises the steps: extracting rich scene features of an online mapping module, a three-dimensional occupation module, an action prediction module and the like through multi-module parallel pre-training, and generating BEV features fused with comprehensive information; then, a lightweight trajectory planner is constructed, the planner performs trajectory planning only by depending on the BEV features, a probability planning method is introduced to deal with uncertainty in planning, probability distribution of actions is output, and one action is sampled from the probability distribution to control the vehicle; the system comprises a storage medium and a processor, and is used for operating according to the instruction to execute the method. According to the method, by simplifying the model structure, interference among multiple tasks is avoided, and the operation efficiency and the trajectory planning precision are improved.

The invention provides a multi-mode intelligent control method and device for a robot body and electronic equipment. The multi-mode intelligent control method comprises the following steps: acquiring multi-view images and task texts of cameras at multiple parts of the robot; generating two-dimensional track supervision information and key point supervision information based on the two, and converting the two-dimensional track supervision information and the key point supervision information into a visual supervision graph; key point features are generated through the supervision graph, and track features are generated in combination with the multi-view image; extracting a depth image, optical flow information, a matched segmentation mask, an execution mechanism attitude and a motion frequency, and generating corresponding features; inputting each feature into a multi-modal fusion reasoning model, generating a relative action prediction feature and converting the relative action prediction feature into an action control instruction; and controlling an execution mechanism to complete the task. According to the method, track supervision, key point supervision and multi-modal prior are introduced under the vision-language condition, and perception integrity, robustness and execution precision are remarkably improved.

The invention discloses an agent action prediction method based on multi-scale spatial perception, which comprises the steps of map spatial structure modeling, historical trajectory feature extraction, feature fusion and multi-modal action prediction. According to the map space structure, map feature extraction is conducted on two-dimensional vector map data in an application scene map through a multi-scale map convolutional neural network, and high-dimensional map feature information is obtained; the historical trajectory feature extraction adopts a convolutional neural network and a feature pyramid network to extract high-dimensional trajectory data features of all agents; according to feature fusion, correlation modeling fusion is carried out on high-dimensional map feature information and high-dimensional trajectory data feature information through a self-attention mechanism, and agent trajectory fusion features with direction information are obtained; according to the multi-modal trajectory prediction, intelligent agent trajectory fusion features are used for prediction regression and confidence scoring, multi-modal complete trajectory coordinates of action prediction and corresponding confidence scores are output, and a reasonable auxiliary decision is provided for intelligent agent actions.

An electronic device and method for object-centric video representation for action prediction is provided. The electronic device extracts a first sequence of video segments from video content associated with a domain and detects a set of objects in the first sequence of video segments. The electronic device generates a set of embeddings based on the first sequence of video segments and the set of objects. The electronic device applies a PTE model on the set of embeddings. The electronic device predicts, based on the application, a set of object-action pairs associated with a second sequence of video segments of the video content. Each object-action pair includes an action to be executed using an object of the set of objects in a video segment of the second sequence of video segments. The second sequence of video segments succeeds the first sequence of video segments in a timeline of the video content.

The invention belongs to the technical field of computer vision, and particularly relates to a human body action prediction method and system based on a space-time interaction attention mechanism. The method comprises the steps that a human body historical motion data set is obtained and preprocessed to obtain a historical motion sequence, the historical motion sequence is input into a Transform model based on a space-time interactive attention mechanism, firstly, time features and space features are extracted through a feature extraction branch, then the extracted features are interacted through the interactive attention mechanism, and the human body historical motion data set is obtained. And finally, obtaining a human motion sequence prediction result through the time features and the space features after joint interaction. According to the method, consistency and interaction information of 3D human body motion space and time characteristics are learned through an interaction attention mechanism, and more real 3D human body motion can be generated in the aspect of long-time modeling.

The invention provides a visual language navigation and model training method, device, equipment and medium, and the method comprises the steps: obtaining a training sample, processing a to-be-executed instruction and environment information in the training sample through a core model, obtaining a navigation feature corresponding to the to-be-executed instruction, and inputting the navigation feature into an action prediction head. Obtaining predicted action information corresponding to the navigation features, and inputting the navigation features into a thinking chain generation head to obtain predicted thinking chain information corresponding to the navigation features; and then, based on the predicted action information, the reference action information in the training sample, the predicted thinking chain information and the reference thinking chain information in the training sample, training the core model, the action prediction head and the thinking chain generation head so as to take the trained core model and the trained action prediction head as a visual language navigation model. According to the method, the reasoning efficiency and the reasoning accuracy can be improved.

The invention provides a virtual object processing method and device, equipment, a computer program product and a computer readable storage medium. The method comprises the following steps: acquiring text description information of an image of a virtual scene at a current time step; obtaining semantic understanding information of the virtual scene in the previous time step; encoding the text description information and the semantic comprehension information of the previous time step to obtain the semantic comprehension information of the virtual scene at the current time step; acquiring an action parameter of the virtual object in the previous time step, and acquiring feature information of the virtual object in the current time step; action prediction processing is carried out through the semantic understanding information of the current time step, the feature information of the current time step and the action parameters of the previous time step, the action parameters of the virtual object in the current time step are obtained, and the action parameters of the current time step are used for driving the virtual object to execute actions in the current time step. According to the method and the device, the virtual object can be intelligently and automatically controlled.

The invention provides a time sequence knowledge graph-based target action prediction method, which comprises the following steps of: acquiring multi-source data which comprises a plurality of entities; based on a relationship classification model, obtaining an association relationship between the entities, and completing redundancy check and alignment de-duplication of the association relationship, the relationship classification model being obtained by classification learning; after target entities in the multiple entities are determined according to the positions and the feature information of the entities, an association tree of association relationships among the target entities is dynamically constructed to complete construction of the time sequence knowledge graph; based on the time sequence knowledge graph, modeling and classifying the time sequence nonlinear incidence relation between the target entities by using a deep neural network to generate a target space evolution model; and predicting a behavior pattern and a motion track of the target entity based on the target spatial evolution model.

The invention provides a mechanical arm control method, a server, equipment, a system and a medium, and relates to the technical field of industrial automation. According to the method, a cloud server responds to an input target task operation instruction, and a full-view observation image of a mechanical arm operation scene is obtained; the target task operation instruction and the full-view-angle observation image are input into a preset multi-mode large model, and a mechanical arm action planning instruction is obtained; the mechanical arm action planning instruction is sent to terminal equipment; the terminal equipment responds to a mechanical arm action planning instruction sent by the cloud server and obtains a current observation image of the mechanical arm; the mechanical arm action planning instruction and the current observation image of the mechanical arm are input into a preset action prediction model, and a corresponding action sequence is obtained; and controlling the mechanical arm to execute the target task according to the action sequence. According to the method, the generalization, migration ability and accuracy of mechanical arm control are improved through end-cloud cooperative control.

The invention belongs to the technical field of artificial intelligence, and relates to a behavior prediction method and system based on a generative visual large model. The method comprises the steps of collecting human body action video data, and extracting human body behavior action data; preprocessing the data; constructing a generative visual large model, performing feature extraction, and performing time sequence analysis by using a long-short-term memory network; performing end-to-end training, calculating a loss function, and obtaining a personnel behavior prediction model; and outputting prediction results of the human body behavior actions in a plurality of future time frames. According to the method, high-level features can be extracted by adopting the generative visual model, so that the efficiency and accuracy of feature extraction are improved; a long-short-term memory network is adopted to perform time sequence analysis on the feature vectors, so that the accuracy and stability of action prediction are improved, the dynamic change of the action can be captured, and the future action trend can be predicted; the performance of the whole model can be optimized by adopting an end-to-end training method, the efficiency and robustness are improved, and the method adapts to variable practical application environments.

Implementations are provided for leveraging training data that is less costly to collect than state-action sequences to perform semi-supervised training of robot control policies. In various implementations, a first input prompt may be assembled with representations of an observed initial state of a robot and a goal state of the robot. The first input prompt may be processed using a goal-conditioned trajectory model to generate first output indicative of a sequence of predicted states to be reached by the robot between the observed initial and goal states. A second input prompt may be assembled to include representations of the sequence of predicted states. The second input prompt may be processed using an action prediction model to generate second output indicative of a sequence of predicted actions to be performed by the robot to reach the sequence of predicted states.

The invention provides a mechanical arm reward-free reinforcement learning method based on expert state learning, belongs to the technical field of automation, particularly focuses on operation control of a robot arm, and comprises two core modules: an expected state prediction network and a reward-free strategy network. The expected state prediction network is used for predicting state transition, and the non-reward strategy network optimizes action through an error between a prediction state of the expected state prediction network and a target state provided by an expert under the condition of no reward signal; according to the method, the mechanical arm can directly learn an effective grabbing strategy from state data observed by experts, complex reward function design is avoided, dependence on expert action data is not needed, and therefore understanding of dynamic characteristics of grabbing tasks is improved, and accuracy of action prediction is enhanced.

Aspects of the present disclosure provide techniques for machine learning based action prediction. Embodiments include providing, as inputs to a recurrent neural network (RNN), an ordered sequence of strings representing actions performed by a user within a software application, the RNN having been trained through a supervised learning process to generate embeddings of the ordered sequence of strings and generate a numerical score relating to a target action based on the embeddings and an order of the ordered sequence of strings. Embodiments include providing, as respective inputs to a tree-based classification machine learning model, the numerical score and an additional feature relating to the user and receiving, as a respective output from the tree-based classification machine learning model in response to the respective inputs, a propensity score indicating a likelihood of the user to perform the target action.

The invention provides a nuclear decommissioning operation master-slave manipulator control system and method. The system comprises an acquisition module for acquiring real-time attitude data, corresponding communication delay data and hot cell environment data of a master manipulator and a slave manipulator respectively; the data processing module preprocesses the collected data, extracts real-time posture standard feature information, communication delay standard feature information and hot cell environment feature data of the master and slave manipulators, and stores the information into a motion information database; the action prediction module constructs a slave manipulator track prediction model based on a neural network, and outputs motion action track prediction information of a slave manipulator; according to the system, in a nuclear decommissioning hot cell environment, real-time attitude data, communication delay data and hot cell environment data of a master manipulator and a slave manipulator are collected, and a high-precision slave manipulator track prediction model is constructed in combination with a neural network, so that the environmental adaptability of the system is enhanced, and the following precision and stability of the slave manipulator to the action of the master manipulator are improved; and accurate synchronous operation of the master and slave manipulators in a complex environment is ensured.

The invention provides an action prediction and training method, a program product, electronic equipment and a chip system, and relates to the technical field of chip design, and the method can expand the exploration space of an operator sequence and improve the optimization effect. The method is executed based on reinforcement learning, and the reinforcement learning comprises a strategy network. The execution object of the action is a logic circuit or other objects; the method comprises the following steps: determining a tth state feature vector; wherein the tth state feature vector is used for representing the environment state after the (t-1) th action is executed; t is an integer variable and is greater than or equal to 1; inputting the t-th state feature vector into a strategy network, and at least outputting a first parameter; based on the first parameter, determining a tth operator to obtain a tth action; wherein the tth action is used for executing the tth operator on the execution object; the tth operator and the (t-1) th operator corresponding to the (t-1) th action are operators of the same type or different types.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an action queue control method, device, equipment and medium, and the method comprises the steps: obtaining initial observation data, sending the initial observation data to a strategy server, and receiving a generated initial action queue; executing the action in the action queue; when the residual amount of the queue is lower than a threshold value, obtaining new observation data; judging whether action prediction is triggered or not based on the similarity between the data and historical observation data; if yes, the new observation data are sent to the strategy server in a non-blocking mode, and a new action block is generated; and receiving a new action block, integrating the new action block with the action queue to form an updated action queue, and repeatedly executing the updated action queue as a new action queue until all actions are completed. According to the method, the action blocks are generated in parallel in the action execution process in an asynchronous reasoning mode, control blind areas and waiting delay are reduced, response timeliness and continuous control capacity are improved, and the requirement of edge equipment for resource efficiency is met.

The invention provides a dynamic sports data visualization system and method, and relates to the technical field of data visualization analysis. Competition behavior data of athletes in a sports event are captured in real time; determining a plurality of exercise interaction nodes of the athlete in the current exercise item, and determining the action evolution trend of each exercise interaction node according to the action tendency track of the athlete; performing feedback evaluation on the real-time state data to obtain a motion feedback strategy of the athlete, and determining an action linkage attribute of the athlete according to the motion feedback strategy and the motion posture information of the athlete in the current sports event; and inputting the action evolution trend and the action linkage attribute into a preset action prediction model for real-time training, and visually displaying the output action prediction data. According to the method and the device, the dynamic sports data of the athletes can be subjected to linkage analysis in a data environment with real-time capture capability and a multi-modal perception basis, so that the effectiveness of collaborative interaction sports dynamic data visualization processing is improved.

The invention belongs to the technical field of data processing, and provides a vector building element capping conflict processing method based on deep reinforcement learning, which comprises the following steps: determining and constructing a constraint Delaunary triangulation network and a skeleton line segment between vector building elements in a target space region, and generating a space conflict detection field model; connecting each vector building element with center points of other surrounding vector building elements, and determining abnormal vector building elements with gland conflicts in combination with a space conflict detection field model; determining the current state information of the abnormal vector building elements, performing data grouping according to the area scale of the abnormal vector building elements so as to set constraint condition parameters and action reward information of each group, and obtaining action prediction information in combination with a DQN model; and according to the action prediction information, controlling the abnormal vector building elements to execute the position movement action until the optimal position. According to the scheme provided by the invention, the processing efficiency, precision and feasibility of the capping conflict are improved.

The invention discloses a human body action recognition method and system based on key frames, and the method comprises the steps: obtaining a target video, and extracting the key frames of actions from the target video; constructing motion features of the global features and the local features based on the key frame, wherein the motion features comprise a global motion image and a quantitative motion feature; constructing a multi-scale motion matrix through quantitative motion features calculated by different original points; and performing action recognition on the multi-scale motion matrix and the global features through a two-stage cascade neural network. The input parameters can be remarkably reduced by using the key frame. In the key frame extraction process, the time scales of different actions can be unified at the same time. Secondly, constructing a motion matrix of which the scale is much larger than that of only one original point by adopting a plurality of original points; therefore, higher precision can be achieved with the CNN having a deeper layer currently. Thirdly, a cascade neural network is provided, and efficient and accurate action prediction is carried out through global features and local features.

The invention relates to the field of robot operation and deep learning model algorithm design, in particular to a robot operation method and system based on a waypoint perception world model.The robot operation method comprises the steps that firstly, operation information of a robot is collected, and the operation information comprises an observation image, current state information of the robot and an instruction given by a user; the operation information is input into a preset waypoint sensing world model, the model generates waypoint information of the robot based on input, the waypoint information and state information of the robot are used for calculating motion information of the robot through a preset motion prediction module, the calculated motion information of the robot is combined with the waypoint information, and the motion information of the robot is obtained. According to the method, the waypoint sensing world model captures key information, and good generalization ability is kept in a changing scene.

The invention relates to the technical field of intelligent control of a tool, in particular to an intelligent control method, device and equipment for a tool based on a visual language and a storage medium. Comprising the following steps: acquiring an original image, and obtaining a control text instruction; inputting the original image and the control text instruction into a visual language model, and determining target coordinate information of an execution target from the original image based on the control text instruction through the visual language model; based on the target coordinate information, determining an execution area containing an execution target, and performing mask processing on the original image according to the execution area; and inputting the mask image obtained after processing and a preset input feature set into an imitation learning model, and outputting action prediction based on the mask image and the preset input feature set through the imitation learning model so as to control the intelligent device with the body to execute a corresponding action. According to the method, the input of subsequent action prediction is simplified through mask processing, so that the flexibility and the response speed of the intelligent equipment with the body are improved.

The invention discloses a fall and turn-over control method and device of a foot type robot and a storage medium. The fall and turn-over control method comprises the steps of firstly obtaining fall posture information of the foot type robot; calling an action selection model to perform multi-stage iterative turnover action prediction based on the falling posture information to obtain an optimal turnover action of each iteration stage, controlling the foot type robot to execute the optimal turnover action of the current iteration stage in each iteration stage, and controlling the foot type robot to execute the optimal turnover action of the current iteration stage after the foot type robot executes each optimal turnover action. The foot-type robot is in a stable posture when the foot-type robot turns over, and the foot-type robot turns over successfully after executing the optimal turning-over action of the last iteration stage. Wherein when the action selection model is called to carry out multi-stage iterative turning-over action prediction based on the falling posture information, the action selection model predicts the optimal turning-over action of the next iteration stage according to the stable posture of the foot robot in the current iteration stage. According to the embodiment, the manual design cost can be reduced, and the applicability is high.

The invention provides a training method of an action prediction model and an action prediction method. According to the main technical scheme, the method comprises the steps of obtaining training data comprising a plurality of training samples; the training sample at least comprises an instruction received by the robot at a first time point, a collected image and an action state, and an action executed at a second time point; identifying whether a preset type of key action exists between the first time point and the second time point; training an action prediction model based on the training data; the training comprises the steps of inputting an instruction received at a first time point, a collected image and an action state into an action prediction model, if a key action of a preset type is identified, taking the key action of the preset type as a prediction target of the action prediction model, otherwise, taking the key action of the preset type as a prediction target of the action prediction model; and taking the action executed at the second time point as a prediction target of the action prediction model. The execution speed and accuracy of the robot action can be improved.

The invention relates to the field of data processing, and discloses an electric vehicle intelligent action prediction and helmet illuminating lamp linkage method and system. The method comprises the steps that a vehicle machine system of the electric vehicle monitors the working states of the vehicle and driving data of the electric vehicle in real time, the working states comprise the left and right steering lamp states, the double-flash signal state and the brake state, and the state information is packaged into a Bluetooth broadcast frame; the helmet Bluetooth equipment continuously monitors Bluetooth broadcast information from the electric vehicle, and analyzes state information in the broadcast frame when receiving the broadcast frame; according to the analyzed state information, an LED lamp on the helmet is controlled to carry out corresponding display; motion prediction and blind area monitoring of the electric vehicle are carried out based on the driving data of the electric vehicle, earphones on the helmet are controlled to give out corresponding prompt tones according to the prediction result and the monitoring result, and a warning light effect is automatically triggered to remind a driver of a following vehicle. The riding safety and convenience are improved, and the interaction experience between the user and other traffic participants is enhanced.

A system may access a first set of images captured by cameras coupled to a shopping cart, wherein each image depicts a user associated with the shopping cart. A system may apply a pose detection model to each of the images to predict a user's pose. A system may apply an action prediction model to the set of images and the predicted poses to predict whether the user performed an action to change the contents of a storage area of the shopping cart. A system may, responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action. A system may update an item list of the user based on the change action and the identified item.

The invention provides a universal dexterous hand action redirection method based on modular residual reinforcement learning, and the method comprises the steps: S110, receiving MANO parameters subjected to structural splitting through a finger sub-strategy network, and outputting initial action prediction; s120, simultaneously receiving the initial motion prediction and the complete MANO parameter representation as input through a residual error coordination network, correcting the local prediction of each finger under the guidance of the global hand intention, and outputting residual error correction, so as to obtain a final motion; s130, executing two-stage training: in the first stage, training each finger sub-strategy network in parallel; in the second stage, parameters of the finger sub-strategy network are frozen, and only the residual error coordination network is trained. According to the technical scheme, the real-time performance of a learning type method and the accuracy of an optimization type method are both achieved, data dependence is reduced, the generalization ability is improved, and meanwhile the equipment universality and action flexibility are both considered.