159 results about "Action prediction" patented technology

The invention provides an agent visual language navigation method and system based on task completion prediction. The method comprises the step of constructing a dual-drive structure composed of a self-adaptive mixed pooling mechanism and a task completion analysis module. Firstly, in the visual information processing process, a dynamic weight distribution strategy is adopted to carry out multi-scale adaptive mixed pooling on panoramic features, so that the fusion effect of local and global semantic information is optimized, and the retention capability and semantic integrity of navigation historical information in a dynamic topological map are improved. And then, inspired by a human navigation cognitive behavior mechanism, a task completion analysis module is designed and introduced, and the task execution progress is dynamically estimated based on the recognition condition of a key landmark in a navigation path, so that an intelligent agent is driven to preferentially select a key path node and invalid exploration is reduced. And finally, realizing efficient understanding and execution of the natural language instruction by the intelligent agent through a multi-round cyclic cross-modal reasoning and action prediction mechanism.

The invention provides a multi-mode intelligent control method and device for a robot body and electronic equipment. The multi-mode intelligent control method comprises the following steps: acquiring multi-view images and task texts of cameras at multiple parts of the robot; generating two-dimensional track supervision information and key point supervision information based on the two, and converting the two-dimensional track supervision information and the key point supervision information into a visual supervision graph; key point features are generated through the supervision graph, and track features are generated in combination with the multi-view image; extracting a depth image, optical flow information, a matched segmentation mask, an execution mechanism attitude and a motion frequency, and generating corresponding features; inputting each feature into a multi-modal fusion reasoning model, generating a relative action prediction feature and converting the relative action prediction feature into an action control instruction; and controlling an execution mechanism to complete the task. According to the method, track supervision, key point supervision and multi-modal prior are introduced under the vision-language condition, and perception integrity, robustness and execution precision are remarkably improved.

The invention provides a visual language navigation and model training method, device, equipment and medium, and the method comprises the steps: obtaining a training sample, processing a to-be-executed instruction and environment information in the training sample through a core model, obtaining a navigation feature corresponding to the to-be-executed instruction, and inputting the navigation feature into an action prediction head. Obtaining predicted action information corresponding to the navigation features, and inputting the navigation features into a thinking chain generation head to obtain predicted thinking chain information corresponding to the navigation features; and then, based on the predicted action information, the reference action information in the training sample, the predicted thinking chain information and the reference thinking chain information in the training sample, training the core model, the action prediction head and the thinking chain generation head so as to take the trained core model and the trained action prediction head as a visual language navigation model. According to the method, the reasoning efficiency and the reasoning accuracy can be improved.

Implementations are provided for leveraging training data that is less costly to collect than state-action sequences to perform semi-supervised training of robot control policies. In various implementations, a first input prompt may be assembled with representations of an observed initial state of a robot and a goal state of the robot. The first input prompt may be processed using a goal-conditioned trajectory model to generate first output indicative of a sequence of predicted states to be reached by the robot between the observed initial and goal states. A second input prompt may be assembled to include representations of the sequence of predicted states. The second input prompt may be processed using an action prediction model to generate second output indicative of a sequence of predicted actions to be performed by the robot to reach the sequence of predicted states.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an action queue control method, device, equipment and medium, and the method comprises the steps: obtaining initial observation data, sending the initial observation data to a strategy server, and receiving a generated initial action queue; executing the action in the action queue; when the residual amount of the queue is lower than a threshold value, obtaining new observation data; judging whether action prediction is triggered or not based on the similarity between the data and historical observation data; if yes, the new observation data are sent to the strategy server in a non-blocking mode, and a new action block is generated; and receiving a new action block, integrating the new action block with the action queue to form an updated action queue, and repeatedly executing the updated action queue as a new action queue until all actions are completed. According to the method, the action blocks are generated in parallel in the action execution process in an asynchronous reasoning mode, control blind areas and waiting delay are reduced, response timeliness and continuous control capacity are improved, and the requirement of edge equipment for resource efficiency is met.

The invention relates to the technical field of intelligent control of a tool, in particular to an intelligent control method, device and equipment for a tool based on a visual language and a storage medium. Comprising the following steps: acquiring an original image, and obtaining a control text instruction; inputting the original image and the control text instruction into a visual language model, and determining target coordinate information of an execution target from the original image based on the control text instruction through the visual language model; based on the target coordinate information, determining an execution area containing an execution target, and performing mask processing on the original image according to the execution area; and inputting the mask image obtained after processing and a preset input feature set into an imitation learning model, and outputting action prediction based on the mask image and the preset input feature set through the imitation learning model so as to control the intelligent device with the body to execute a corresponding action. According to the method, the input of subsequent action prediction is simplified through mask processing, so that the flexibility and the response speed of the intelligent equipment with the body are improved.

A system may access a first set of images captured by cameras coupled to a shopping cart, wherein each image depicts a user associated with the shopping cart. A system may apply a pose detection model to each of the images to predict a user's pose. A system may apply an action prediction model to the set of images and the predicted poses to predict whether the user performed an action to change the contents of a storage area of the shopping cart. A system may, responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action. A system may update an item list of the user based on the change action and the identified item.

The invention provides a universal dexterous hand action redirection method based on modular residual reinforcement learning, and the method comprises the steps: S110, receiving MANO parameters subjected to structural splitting through a finger sub-strategy network, and outputting initial action prediction; s120, simultaneously receiving the initial motion prediction and the complete MANO parameter representation as input through a residual error coordination network, correcting the local prediction of each finger under the guidance of the global hand intention, and outputting residual error correction, so as to obtain a final motion; s130, executing two-stage training: in the first stage, training each finger sub-strategy network in parallel; in the second stage, parameters of the finger sub-strategy network are frozen, and only the residual error coordination network is trained. According to the technical scheme, the real-time performance of a learning type method and the accuracy of an optimization type method are both achieved, data dependence is reduced, the generalization ability is improved, and meanwhile the equipment universality and action flexibility are both considered.

The invention relates to the technical field of three-dimensional scene modeling, in particular to a dynamic event scene reproduction method and system and electronic equipment, and the method comprises the steps: collecting scene multi-modal data, and generating a three-dimensional static scene model; generating a static object supplementary description model and a dynamic object motion prediction model by using a large language model; fusing the text semantic features and the image visual features, and integrating with a dynamic object motion prediction model to generate a total behavior prediction model; motion planning and real-time simulation are carried out on a virtual role, and a virtual role model with intelligent behavior ability is output; combining the three-dimensional static scene model, the static object supplement description model, the total behavior prediction model and the virtual role model to generate a complete dynamic event scene; according to the invention, through the innovative architecture of dynamic and static separation modeling and large language model driving, the bottleneck of static property and manual dependence of the traditional scene modeling technology is broken through, and efficient and real dynamic scene automatic reproduction is realized.

The invention provides a robot action prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting multiple pieces of visual information of a robot and task execution instruction information into a first network layer of a pre-trained robot action prediction model for key point prediction processing, outputting key point features of the operated object grabbed by the robot; wherein different visual information comes from different mechanical arms of the robot; and performing action prediction processing on current mechanical arm state information, multiple pieces of visual information, task execution instruction information and the key point features based on a second network layer of the robot action prediction model, and outputting an action sequence of the robot. The key point features of the clamping jaw on the mechanical arm of the robot and the operated object are accurately determined through the robot action prediction model, and then action prediction processing is conducted through the key point features to improve the accuracy of robot action sequence determination.

The invention provides a robot action sequence generation method and device, equipment and a medium, and the method comprises the steps: inputting real-time multi-modal observation data and a current joint state into a pre-trained action prediction model, and generating an initial action prediction sequence containing a plurality of future time steps; constructing a space-time heterogeneous guide mask matrix by utilizing the current tail end linear speed and the execution state of a tail end operation part in the target robot; and correcting the initial action prediction sequence by using the space-time heterogeneous guide mask matrix and the historical action sequence queue to obtain a target action sequence, and sending the target action sequence to a controller of the target robot. By means of the method and device, the problems that in the track splicing process of a traditional method, a mechanical arm shakes, and an end effector is switched in an uncertain state are effectively solved, and the stability and response precision of target robot control are improved.

The invention provides a visual language action model training method and device, equipment and a storage medium, and the method comprises the steps: extracting a corresponding two-dimensional visual feature and a four-dimensional visual feature based on an environment image, carrying out the multi-modal feature fusion of the two-dimensional visual feature, the four-dimensional visual feature, the language feature of a natural language instruction, and the state feature of state data, and carrying out the multi-modal feature fusion of the two-dimensional visual feature and the four-dimensional visual feature. Motion prediction and visual feature reconstruction are carried out based on multi-modal fusion features, model parameter updating is carried out by using generated multi-modal output data, and the model has the capability of extracting spatio-temporal dynamic information under the conditions of not increasing additional sensors and ensuring that the overall parameter quantity of the model is small. The time and space reasoning performance of the visual language action model in the dynamic environment can be improved, so that the actuator can be accurately controlled to execute the natural language instruction in the dynamic environment in the model reasoning stage.

The invention discloses a robot virtual-real cooperative training decision optimization system and method based on digital twinning, and the method comprises the following steps: collecting state data and disturbance data of an entity robot in a real environment, and carrying out the preprocessing of the data to generate standardized input; joint coding and state perturbation mapping are carried out on the standardized data, and a feature vector sequence embedded in a hyperspherical manifold space is generated; inputting to a virtual twin control body based on a hypersurface neural element structure, executing disturbance direction sensitive activation, and outputting an activated state vector sequence; virtual and entity control action prediction sequences are generated respectively, an embedded space difference vector is calculated, and control body parameters are updated based on a consistency optimization criterion; after convergence, the control body executes reasoning to generate a target control action sequence, the entity robot is driven to complete action execution, and control strategy optimization is achieved. According to the method, high-precision migration and rapid convergence of a robot control strategy are realized, and the execution stability in a complex disturbance environment is improved.

The invention provides a robot imitation learning system and method, electronic equipment and a medium, and the method comprises the steps: a multi-mode sensing module is used for collecting the sensing data of a robot operation environment; the feature coding module is used for carrying out feature coding on the perceptual data to obtain corresponding visual feature vectors, tactile feature vectors, language feature vectors and joint feature vectors, and aligning all the feature vectors to a unified time sequence; the cross-modal alignment and fusion module is used for generating fused multi-modal collaborative representation through the fusion network; and the action prediction and control module is used for decoding the multi-mode cooperative representation, generating an action instruction of the robot, and performing closed-loop control and error recovery on execution of the action instruction based on real-time feedback tactile data and joint sensor data. According to the invention, more stable closed-loop control of the robot is realized, and the fine operation capability of the robot is improved.

The invention provides a light-weight pedestrian motion prediction method based on a mask, and is suitable for the technical field of human-computer interaction. According to the method, human body 3D skeleton point data is processed through space and time masks, key features are extracted by using a local perceptron constructed by a lightweight multilayer perceptron module and a cross-frame fusion device, the features are fused by using 1 * 1 convolution and splicing technologies, then a frequency-space domain pedestrian prediction action sequence is generated through a prediction module, and the frequency-space domain pedestrian prediction action sequence is obtained. And converting into a time-space domain sequence through an inverse discrete cosine converter. The method has the advantages of light model, quick response and suitability for real-time interaction. In addition, the strategy of gradually increasing the number of training frames improves the accuracy and stability of prediction.

The application provides an instance retrieval method based on a cross-semantic attention model and related equipment. The method comprises the following steps: selecting any shot as a target shot from multiple shots, and obtaining an action prediction score vector and a location prediction score vector in the target shot; correcting the location prediction score vector and the action prediction score vector to obtain a corrected location prediction score vector and a corrected action prediction score vector; calculating an association retrieval score based on the corrected action prediction score vector, the corrected location prediction score vector, a to-be-queried action and a to-be-queried location; repeating the above steps until the multiple shots are selected to obtain multiple association retrieval scores; and obtaining a retrieval result corresponding to the to-be-queried action and the to-be-queried location based on the multiple association retrieval scores. Through the application, a shot with a semantic contradiction in a retrieval result can be effectively avoided, and the precision of "action-location" compound semantic retrieval is improved.

The present disclosure relates to methods and systems for spatiotemporal graph modelling of road users in observed frames of an environment in which an autonomous vehicle operates (i.e. a traffic scene), clustering of the road users into categories, and providing the spatiotemporal graph to a trained graphical convolutional neural network (GNN) to predict a future pedestrian action. The future pedestrian action can be: one of the pedestrian will cross a road and the pedestrian will not cross the road. The spatiotemporal graph includes a better understanding of the observed frames (i.e. traffic scene).

The application provides a robot action prediction method and device, equipment and a storage medium, wherein the method comprises: extracting global features and local features of an action sequence based on a reinforcement learning action prediction model, and predicting a predicted action sequence based on the extracted global features and local features. By extracting the global features and local features of the action sequence respectively, the continuity of the historical action sequence in time can be concerned while focusing on the local correlation of the action sequence. By extracting the features through the Mamba module, the selectivity and context awareness of the model can be further enhanced, thereby improving the performance of the model. In addition, since the Mamba module has a significant lightweight advantage, the action prediction model improved based on the Mamba module also has the advantages of being lightweight and easy to deploy.

The present disclosure provides a deep reinforcement learning model training method for generating a treatment plan, a treatment plan generation method, equipment and media, relating to the technical field of medical treatment, especially to the technical field of radiotherapy. The method comprises: obtaining initial dose distribution state data of a target target region; determining target data based on the initial dose distribution state data of the target target region, the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer; updating the current policy data of the plurality of action prediction network layers and the current policy data of the evaluation prediction network layer based on the target data, completing the current training of the deep reinforcement learning model; iteratively performing the above training process until the training of the deep reinforcement learning model reaches a preset number of times, and obtaining the trained deep reinforcement learning model.

The invention relates to the technical field of artificial intelligence, provides a robot control method and device based on an image-text interleaving instruction, equipment and a medium, is applied to financial and medical health care service scenes, and can construct an initial model which takes an image-text mixed format as an input data format and takes a visual language model as a backbone model. The limitation that a traditional vision-language-action model can only process image observation and plain text instructions is broken through; performing language-action pre-training on the initial model based on a potential action distribution function and a flow matching mechanism, so that the model can learn potential representation of action intention from a language; instruction tuning training is carried out based on a hybrid expert mechanism to ensure that the model is flexibly switched between language reasoning and action prediction, so that the model can accurately reasone from a complex and natural image-text instruction and carry out action planning; and generating a predicted action sequence by using the robot action generation model, so as to accurately control the robot based on the image-text interlacing instruction.

The embodiments of the present disclosure disclose a movable platform action prediction method and a training method of an action prediction model, wherein the movable platform action prediction method comprises: acquiring multi-modal dynamic related information corresponding to the movable platform at a current time; determining visual latent features and action latent features in a unified latent space based on the dynamic related information, realizing short-time sequence real-time response; performing prediction processing on at least one of the multi-modal dynamic related information to obtain prediction latent variable features at future k time points relative to the current time, obtaining k prediction latent variable features, and realizing long-time sequence dynamic perception planning; and performing action prediction based on the visual latent features, the action latent features and the k prediction latent variable features to obtain an action prediction result at a next time point relative to the current time. The embodiments of the present disclosure improve the foresight, real-time and stability of action prediction.

The invention discloses a quick-response multi-worker dangerous action prediction and real-time early warning method and system, and belongs to the technical field of computer vision and deep learning, and the method comprises the steps: obtaining video frames of workers and an environment, carrying out the preprocessing, carrying out the real-time detection and tracking of a plurality of workers in the video frames through employing YOLOv5, generating and screening a high-confidence bounding box, and carrying out the cutting; extracting two-dimensional coordinates of the joints of the worker based on attitude estimation and generating a skeleton graph; carrying out feature modeling on a nonlinear relation between skeleton joints by adopting a KAN (Karan Area Network) so as to capture subtle attitude changes, and obtaining a high-dimensional feature vector of an action; time sequence modeling is carried out, future potential dangerous actions are predicted, danger scores are output, when the scores exceed a threshold value, early warning is triggered, and advanced intervention and real-time alarm of high-risk actions are achieved. According to the method, mutual interference in a multi-worker scene can be effectively reduced, the recognition sensitivity and real-time response capability of fine-grained action abnormity are improved, and the method is suitable for industrial production safety monitoring and accident prevention.

The invention relates to the technical field of robot control, and discloses a visual motion joint modeling and prediction method, device, equipment and medium, and the method comprises the steps: constructing a unified input data set, carrying out the preprocessing and time alignment of a visual image and a motion state vector, and generating standardized input data; inputting the visual image into a visual backbone network to obtain a visual feature sequence; the visual feature sequence is sent to a time sequence modeling module to generate feature representation fused with a time sequence context; inputting the feature representation fused with the time sequence context into an action prediction branch to generate an action prediction vector; generating a training loss item set based on the motion prediction vector and the demonstration label, and jointly updating the model to obtain an optimization model; constructing a contrast model based on the optimization model and generating an evaluation record; and determining optimization configuration based on the evaluation record and updating the optimization model. According to the method, through unified perception and prediction and dynamic optimization, the precision and efficiency are improved, and the autonomous operation capability in a complex environment is enhanced.

The invention discloses a double-arm robot control method based on single-arm trajectory prior guidance. The method comprises the following steps: acquiring a double-arm robot data set; skill primitives of different action types are obtained based on a single-arm action prediction model, the skill primitives of different action types are selected for a left arm and a right arm, and next action intentions of the left arm and the right arm are predicted respectively; constructing a double-arm diffusion generation model for establishing a mapping relation from initial action distribution to real double-arm coordination action distribution; training a single-arm action prediction model and a double-arm diffusion generation model based on the double-arm robot training data set; and acquiring motion prediction priori of the left arm and the right arm, constructing Gaussian distribution taking the single-arm priori as a mean value and a preset variance as a radius, sampling from the Gaussian distribution to obtain an initial motion vector, inputting the initial motion vector into a trained double-arm diffusion generation model, and de-noising the initial motion vector to generate double-arm motion. According to the invention, the independence of single-arm motion and the collaboration of double-arm control can be considered, and the double-arm motion control effect is optimized.

This invention discloses a closed-loop policy prediction method and apparatus based on a world model, relating to the field of action prediction. The method includes: during iterative prediction at the current time step, using a pre-trained world model, the core hidden state of the current time step is obtained based on the visual observation frame and action command of the current time step, as well as the core hidden state of the previous time step. The core hidden state of the current time step is then input into a trained policy model to obtain the action command for the next time step. The pre-trained world model then generates the visual observation frame for the next time step based on the action command for the next time step and the core hidden state of the current time step. This iterative prediction process is repeated to obtain action commands for different time steps, which are then sent to the machine to execute the corresponding actions. This invention addresses the problem of existing world models causing inaccurate action predictions due to the predicted state deviating from the actual physical state.

The invention belongs to the technical field of 3D projection, and particularly relates to an immersive deduction system and method based on 3D projection fusion, and the system comprises a sensing module which is used for collecting the rapid motion data and height parameters of an actor, a motion pre-judgment module which is used for judging the motion parameters of the actor, and a control module which is used for controlling the motion pre-judgment module and the motion pre-judgment module. The motion pre-judgment module is in communication connection with the sensing module and comprises a motion association sub-model and a motion time sequence prediction sub-module, and the motion association sub-model establishes a mapping relation among the body height of the actor, the motion amplitude and the track span. And data is provided for follow-up action pre-judgment and special effect adaptation, and the problem that the projection special effect is disjointed with the action of the actor due to action acquisition omission or insufficient data precision is avoided.

The application belongs to the technical field of computer vision, and relates to a human action recognition method and system based on a double-flow self-attention mechanism, which comprises the following steps: collecting video data of human actions; extracting multiple modal features of the video data through a basic network of a double-flow self-attention mechanism model, and fusing the multiple modal features with a time flow; inputting the fused space-time features into a space-time memory network of the double-flow self-attention mechanism model, performing space-time feature interaction of different scales, and generating a preliminary action classification result; and estimating the action boundary and action center offset position of the preliminary action classification result through an action prediction head, and generating a final human action recognition result. The application can effectively extract human action video features, can extract human features from two levels of a macro instance level and a micro fine-grained level, and thus can reduce the interference of an action video background environment on an action recognition result.

The present application relates to the technical field of artificial intelligence, is applied to robot control, machine vision, automatic driving and other business scenes, performs action prediction by using an action prediction model obtained by performing potential action pre-training and real action post-training based on an unsupervised mode, can effectively improve the accuracy of action prediction, and improves the accuracy of action prediction. And therefore, the probability of operation errors when task operation is performed according to the predicted action is reduced. Relates to an action prediction method and device, equipment and a medium. The method comprises the following steps: acquiring video stream data of a target object and a language instruction corresponding to the video stream data; inputting a language instruction corresponding to the video stream data and the video stream data into a trained action prediction model for action prediction to obtain an action prediction result, the action prediction model being a model obtained by performing potential action pre-training and real action post-training based on an unsupervised mode in advance; and determining an action instruction according to the action prediction result, and controlling the target object to perform task operation based on the action instruction.

The invention belongs to the technical field of communication, and discloses a dynamic multi-network switching method for power satellite communication and a related device. The dynamic multi-network switching method comprises the following steps: predicting and acquiring a cellular link performance prediction result in a future preset time period based on an acquired network state joint feature vector; splicing the network state joint feature vector, the power service scene label and the cellular link performance prediction result to form a network fusion feature vector, taking the network fusion feature vector as input, and performing action prediction by using the trained double-depth Q network to obtain a prediction action set; based on the predicted action set, illegal action elimination and overcharge action shielding are carried out, effective actions are obtained, corresponding configuration files are activated, and network connection is carried out. According to the technical scheme disclosed by the invention, flexible switching between the ground network and the satellite network can be realized, and meanwhile, the switching packet loss rate can be reduced.

The invention provides a device control method and system, a device and a storage medium, and the method comprises the steps: an edge side device receives multi-modal input data sent by each end side device; the edge side device performs reasoning based on the multi-modal input data sent by each end side device, generates future action prediction information of each end side device, and sends the future action prediction information of each end side device to each end side device; and each end-side device updates an annular action cache queue in the end-side device based on the future action prediction information, determines a current control instruction from the annular action cache queue according to the current moment, and executes the current control instruction. According to the method, a calculation-intensive reasoning process can be separated from an execution process with an extremely high real-time requirement, so that continuity and smoothness of action output are realized, and motion discontinuity is avoided fundamentally. And meanwhile, the robustness and the stability of the control process can be remarkably improved.