226 results about "Action prediction" patented technology

The invention provides an agent visual language navigation method and system based on task completion prediction. The method comprises the step of constructing a dual-drive structure composed of a self-adaptive mixed pooling mechanism and a task completion analysis module. Firstly, in the visual information processing process, a dynamic weight distribution strategy is adopted to carry out multi-scale adaptive mixed pooling on panoramic features, so that the fusion effect of local and global semantic information is optimized, and the retention capability and semantic integrity of navigation historical information in a dynamic topological map are improved. And then, inspired by a human navigation cognitive behavior mechanism, a task completion analysis module is designed and introduced, and the task execution progress is dynamically estimated based on the recognition condition of a key landmark in a navigation path, so that an intelligent agent is driven to preferentially select a key path node and invalid exploration is reduced. And finally, realizing efficient understanding and execution of the natural language instruction by the intelligent agent through a multi-round cyclic cross-modal reasoning and action prediction mechanism.

The invention belongs to the technical field of robot operation and control, and discloses a robot action generation method and a related device. The robot action generation method comprises the steps of obtaining a task instruction, ontology perception of a to-be-operated robot at the current moment, a multi-view RGB-D image and an initialized current noise action; and utilizing the trained action consistency diffusion strategy model to perform iterative action denoising according to the obtained task instruction, the ontology perception of the to-be-operated robot at the current moment, the multi-view RGB-D image and the initialized current noise action, and generating the action of the next step of the to-be-operated robot. According to the technical scheme disclosed by the invention, the technical problems of high training modeling difficulty, insufficient action prediction precision, low action generation speed and the like of the existing robot action generation technology in a high-precision operation task of the robot can be solved.

The invention discloses a visual language navigation method based on memory driving, and the method comprises the following steps: S1, constructing an MDQT model which comprises a panoramic encoder, a text embedding layer, a Q-Former, an action prediction module and a memory updating module; s2, training an MDQT model by using three pre-training tasks of image-concerned mask language modeling, instruction track matching and instruction track contrast learning; and S3, performing fine tuning on the MDQT model by using imitation learning and reinforcement learning. According to the invention, a learnable memory vector with a fixed length is used to encode historical information. In each navigation step, the memory vector interacts with the extracted panoramic information and instruction information, and visual information most relevant to the current instruction information is extracted according to the current memory state of the robot for decision making. The memory of the robot to the historical navigation steps is effectively maintained under limited resources, and the navigation success rate and efficiency of the robot are improved.

The invention provides a multi-mode intelligent control method and device for a robot body and electronic equipment. The multi-mode intelligent control method comprises the following steps: acquiring multi-view images and task texts of cameras at multiple parts of the robot; generating two-dimensional track supervision information and key point supervision information based on the two, and converting the two-dimensional track supervision information and the key point supervision information into a visual supervision graph; key point features are generated through the supervision graph, and track features are generated in combination with the multi-view image; extracting a depth image, optical flow information, a matched segmentation mask, an execution mechanism attitude and a motion frequency, and generating corresponding features; inputting each feature into a multi-modal fusion reasoning model, generating a relative action prediction feature and converting the relative action prediction feature into an action control instruction; and controlling an execution mechanism to complete the task. According to the method, track supervision, key point supervision and multi-modal prior are introduced under the vision-language condition, and perception integrity, robustness and execution precision are remarkably improved.

The invention provides a visual language navigation and model training method, device, equipment and medium, and the method comprises the steps: obtaining a training sample, processing a to-be-executed instruction and environment information in the training sample through a core model, obtaining a navigation feature corresponding to the to-be-executed instruction, and inputting the navigation feature into an action prediction head. Obtaining predicted action information corresponding to the navigation features, and inputting the navigation features into a thinking chain generation head to obtain predicted thinking chain information corresponding to the navigation features; and then, based on the predicted action information, the reference action information in the training sample, the predicted thinking chain information and the reference thinking chain information in the training sample, training the core model, the action prediction head and the thinking chain generation head so as to take the trained core model and the trained action prediction head as a visual language navigation model. According to the method, the reasoning efficiency and the reasoning accuracy can be improved.

The invention provides a virtual object processing method and device, equipment, a computer program product and a computer readable storage medium. The method comprises the following steps: acquiring text description information of an image of a virtual scene at a current time step; obtaining semantic understanding information of the virtual scene in the previous time step; encoding the text description information and the semantic comprehension information of the previous time step to obtain the semantic comprehension information of the virtual scene at the current time step; acquiring an action parameter of the virtual object in the previous time step, and acquiring feature information of the virtual object in the current time step; action prediction processing is carried out through the semantic understanding information of the current time step, the feature information of the current time step and the action parameters of the previous time step, the action parameters of the virtual object in the current time step are obtained, and the action parameters of the current time step are used for driving the virtual object to execute actions in the current time step. According to the method and the device, the virtual object can be intelligently and automatically controlled.

The invention provides a time sequence knowledge graph-based target action prediction method, which comprises the following steps of: acquiring multi-source data which comprises a plurality of entities; based on a relationship classification model, obtaining an association relationship between the entities, and completing redundancy check and alignment de-duplication of the association relationship, the relationship classification model being obtained by classification learning; after target entities in the multiple entities are determined according to the positions and the feature information of the entities, an association tree of association relationships among the target entities is dynamically constructed to complete construction of the time sequence knowledge graph; based on the time sequence knowledge graph, modeling and classifying the time sequence nonlinear incidence relation between the target entities by using a deep neural network to generate a target space evolution model; and predicting a behavior pattern and a motion track of the target entity based on the target spatial evolution model.

Implementations are provided for leveraging training data that is less costly to collect than state-action sequences to perform semi-supervised training of robot control policies. In various implementations, a first input prompt may be assembled with representations of an observed initial state of a robot and a goal state of the robot. The first input prompt may be processed using a goal-conditioned trajectory model to generate first output indicative of a sequence of predicted states to be reached by the robot between the observed initial and goal states. A second input prompt may be assembled to include representations of the sequence of predicted states. The second input prompt may be processed using an action prediction model to generate second output indicative of a sequence of predicted actions to be performed by the robot to reach the sequence of predicted states.

Aspects of the present disclosure provide techniques for machine learning based action prediction. Embodiments include providing, as inputs to a recurrent neural network (RNN), an ordered sequence of strings representing actions performed by a user within a software application, the RNN having been trained through a supervised learning process to generate embeddings of the ordered sequence of strings and generate a numerical score relating to a target action based on the embeddings and an order of the ordered sequence of strings. Embodiments include providing, as respective inputs to a tree-based classification machine learning model, the numerical score and an additional feature relating to the user and receiving, as a respective output from the tree-based classification machine learning model in response to the respective inputs, a propensity score indicating a likelihood of the user to perform the target action.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an action queue control method, device, equipment and medium, and the method comprises the steps: obtaining initial observation data, sending the initial observation data to a strategy server, and receiving a generated initial action queue; executing the action in the action queue; when the residual amount of the queue is lower than a threshold value, obtaining new observation data; judging whether action prediction is triggered or not based on the similarity between the data and historical observation data; if yes, the new observation data are sent to the strategy server in a non-blocking mode, and a new action block is generated; and receiving a new action block, integrating the new action block with the action queue to form an updated action queue, and repeatedly executing the updated action queue as a new action queue until all actions are completed. According to the method, the action blocks are generated in parallel in the action execution process in an asynchronous reasoning mode, control blind areas and waiting delay are reduced, response timeliness and continuous control capacity are improved, and the requirement of edge equipment for resource efficiency is met.

The invention relates to the technical field of intelligent control of a tool, in particular to an intelligent control method, device and equipment for a tool based on a visual language and a storage medium. Comprising the following steps: acquiring an original image, and obtaining a control text instruction; inputting the original image and the control text instruction into a visual language model, and determining target coordinate information of an execution target from the original image based on the control text instruction through the visual language model; based on the target coordinate information, determining an execution area containing an execution target, and performing mask processing on the original image according to the execution area; and inputting the mask image obtained after processing and a preset input feature set into an imitation learning model, and outputting action prediction based on the mask image and the preset input feature set through the imitation learning model so as to control the intelligent device with the body to execute a corresponding action. According to the method, the input of subsequent action prediction is simplified through mask processing, so that the flexibility and the response speed of the intelligent equipment with the body are improved.

The invention discloses a fall and turn-over control method and device of a foot type robot and a storage medium. The fall and turn-over control method comprises the steps of firstly obtaining fall posture information of the foot type robot; calling an action selection model to perform multi-stage iterative turnover action prediction based on the falling posture information to obtain an optimal turnover action of each iteration stage, controlling the foot type robot to execute the optimal turnover action of the current iteration stage in each iteration stage, and controlling the foot type robot to execute the optimal turnover action of the current iteration stage after the foot type robot executes each optimal turnover action. The foot-type robot is in a stable posture when the foot-type robot turns over, and the foot-type robot turns over successfully after executing the optimal turning-over action of the last iteration stage. Wherein when the action selection model is called to carry out multi-stage iterative turning-over action prediction based on the falling posture information, the action selection model predicts the optimal turning-over action of the next iteration stage according to the stable posture of the foot robot in the current iteration stage. According to the embodiment, the manual design cost can be reduced, and the applicability is high.

A system may access a first set of images captured by cameras coupled to a shopping cart, wherein each image depicts a user associated with the shopping cart. A system may apply a pose detection model to each of the images to predict a user's pose. A system may apply an action prediction model to the set of images and the predicted poses to predict whether the user performed an action to change the contents of a storage area of the shopping cart. A system may, responsive to predicting that the user performed a change action, apply an item identification model to a second set of images of a storage area of the shopping cart to identify an item associated with the change action. A system may update an item list of the user based on the change action and the identified item.

The invention provides a universal dexterous hand action redirection method based on modular residual reinforcement learning, and the method comprises the steps: S110, receiving MANO parameters subjected to structural splitting through a finger sub-strategy network, and outputting initial action prediction; s120, simultaneously receiving the initial motion prediction and the complete MANO parameter representation as input through a residual error coordination network, correcting the local prediction of each finger under the guidance of the global hand intention, and outputting residual error correction, so as to obtain a final motion; s130, executing two-stage training: in the first stage, training each finger sub-strategy network in parallel; in the second stage, parameters of the finger sub-strategy network are frozen, and only the residual error coordination network is trained. According to the technical scheme, the real-time performance of a learning type method and the accuracy of an optimization type method are both achieved, data dependence is reduced, the generalization ability is improved, and meanwhile the equipment universality and action flexibility are both considered.

The invention relates to the technical field of three-dimensional scene modeling, in particular to a dynamic event scene reproduction method and system and electronic equipment, and the method comprises the steps: collecting scene multi-modal data, and generating a three-dimensional static scene model; generating a static object supplementary description model and a dynamic object motion prediction model by using a large language model; fusing the text semantic features and the image visual features, and integrating with a dynamic object motion prediction model to generate a total behavior prediction model; motion planning and real-time simulation are carried out on a virtual role, and a virtual role model with intelligent behavior ability is output; combining the three-dimensional static scene model, the static object supplement description model, the total behavior prediction model and the virtual role model to generate a complete dynamic event scene; according to the invention, through the innovative architecture of dynamic and static separation modeling and large language model driving, the bottleneck of static property and manual dependence of the traditional scene modeling technology is broken through, and efficient and real dynamic scene automatic reproduction is realized.

The invention provides an automatic task-independent data acquisition method for a body-equipped agent, a training method and device for an action prediction model, equipment and a storage medium, and relates to the technical field of body-equipped agents. The task-independent data acquisition method comprises the following steps: defining a 3D position feasible range of an end effector as a bounded working space cube; training the reinforcement learning network through a near-end strategy optimization algorithm; randomly sampling a target point as a target point 3D position in the workspace cube, generating a corresponding joint position of the mechanical arm by using a reinforcement learning network, and constructing position mapping; the method comprises the following steps: driving an end effector of the mechanical arm to perform random sampling in an actual space, recording a sampling image obtained when the end effector reaches a 3D sampling position in the random sampling process of the end effector, and taking the sampling image and a corresponding joint position as task irrelevant data of the mechanical arm. The data collection process is simpler and is easy to expand, and cross-task multiplexing of data is supported.

The invention provides a robot action prediction method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting multiple pieces of visual information of a robot and task execution instruction information into a first network layer of a pre-trained robot action prediction model for key point prediction processing, outputting key point features of the operated object grabbed by the robot; wherein different visual information comes from different mechanical arms of the robot; and performing action prediction processing on current mechanical arm state information, multiple pieces of visual information, task execution instruction information and the key point features based on a second network layer of the robot action prediction model, and outputting an action sequence of the robot. The key point features of the clamping jaw on the mechanical arm of the robot and the operated object are accurately determined through the robot action prediction model, and then action prediction processing is conducted through the key point features to improve the accuracy of robot action sequence determination.

The invention provides a robot action sequence generation method and device, equipment and a medium, and the method comprises the steps: inputting real-time multi-modal observation data and a current joint state into a pre-trained action prediction model, and generating an initial action prediction sequence containing a plurality of future time steps; constructing a space-time heterogeneous guide mask matrix by utilizing the current tail end linear speed and the execution state of a tail end operation part in the target robot; and correcting the initial action prediction sequence by using the space-time heterogeneous guide mask matrix and the historical action sequence queue to obtain a target action sequence, and sending the target action sequence to a controller of the target robot. By means of the method and device, the problems that in the track splicing process of a traditional method, a mechanical arm shakes, and an end effector is switched in an uncertain state are effectively solved, and the stability and response precision of target robot control are improved.

The invention provides a visual language action model training method and device, equipment and a storage medium, and the method comprises the steps: extracting a corresponding two-dimensional visual feature and a four-dimensional visual feature based on an environment image, carrying out the multi-modal feature fusion of the two-dimensional visual feature, the four-dimensional visual feature, the language feature of a natural language instruction, and the state feature of state data, and carrying out the multi-modal feature fusion of the two-dimensional visual feature and the four-dimensional visual feature. Motion prediction and visual feature reconstruction are carried out based on multi-modal fusion features, model parameter updating is carried out by using generated multi-modal output data, and the model has the capability of extracting spatio-temporal dynamic information under the conditions of not increasing additional sensors and ensuring that the overall parameter quantity of the model is small. The time and space reasoning performance of the visual language action model in the dynamic environment can be improved, so that the actuator can be accurately controlled to execute the natural language instruction in the dynamic environment in the model reasoning stage.

The invention discloses a time sequence dynamic gesture recognition method and system. The time sequence dynamic gesture recognition method mainly comprises the steps of S1, hand target detection; step S2, hand target tracking; s3, outputting category information; s4, setting an information container; and S5, gesture motion prediction. According to the method, the problem of time sequence prediction of gesture actions is simplified, meanwhile, the problem of easy ID exchange during target tracking is solved, YOLO is mainly used for detecting a hand and positioning key point information of a hand skeleton, a neural network is realized based on a 1D operator, and gesture action prediction is carried out on time sequence key point information. According to the dynamic gesture recognition method based on combination of the multiple models, lightweight design is carried out on the YOLOv8 model, real-time recognition is achieved on equipment with limited resources (for example, the real-time recognition can reach 25FPS on RK3566), and the problem that ID information is likely to be exchanged when hands are waved crosswise is solved through dynamic threshold control parameters of the target tracking module.

The invention discloses an off-line reinforcement learning action exploration agent method based on expected reward regularization, and belongs to the field of reinforcement learning. The problem that reliable track splicing and strategy generalization are difficult to realize in a complex task by the existing method is solved. The method comprises the following steps: constructing state loss and action loss based on sequence modeling, and carrying out iterative training of states and actions; designing an RTG loss function based on a weighted square error; a double-Q learning framework is adopted to maintain two independent Q functions, conservative constraints of conservative Q learning are applied in the Q function updating process, and action exploration optimization is achieved in combination with Boltzmann distribution; combining state loss, action loss, RTG regularization loss and Q value loss to form a joint optimization objective function; performing noise disturbance sampling on the plurality of RTG candidate values to generate diversified action prediction; and value evaluation is performed on the candidate actions based on a double-conservative Q function, and the action with the highest Q value is selected for execution. The method is mainly used in the intelligent agent exploration field.

The invention provides an on-line motion prediction method based on space-time consistency information decoupling and integration, and the method comprises the steps: obtaining a video stream, and extracting an RGB image and an optical flow field image from the video stream; and inputting the RGB image and the optical flow field into a trained ST-Mama model to obtain a prediction action of a first frame of the video stream. According to the method, the Mama is introduced into the online action detection task, so that the long sequence action detection capability is improved, the computing resource consumption during reasoning is greatly reduced, and the OAD requirement with high real-time performance is met. Moreover, a plurality of scanning strategies are introduced into the decoupling modules for feature supplementation and enhancement, so that the perception capability of the model to a complex process can be effectively improved. Besides, the semantic expression of the integrated module is enhanced by means of historical frame information, and progressive optimization of action features is completed through double Mama branches, so that the recognition ability of the model to the current state and the memory ability of the model to the context are enhanced.

A database system accesses historical data, including characteristics of employees before their departure from past employers. It then generates a normalized training set based on this data and geographic details of each entity. Using this set, the system trains a neural network to forecast employee exits from current employers. The system applies this model to predict the departure dates for specific employees from a target company. The system updates the training set with both predicted and actual departure dates, improving the neural network through a second stage of retraining.

The invention relates to the technical field of artificial intelligence, provides a robot control method and device based on physical constraint embedding, equipment and a medium, is applied to financial and medical health care service scenes, and can realize unified representation of multi-modal data through feature extraction. Vectorizing the physical knowledge graph, and converting the abstract physical rule into a data structure which can be identified by a machine; constructing a scene physical graph, generating physically enhanced visual features according to the language feature vector and the scene physical graph, and performing feature splicing to realize deep fusion of physical knowledge and visual and language modals; target features including dynamic physical information are generated based on a variational auto-encoder, and the real-time sensing capability of dynamic changes of a physical environment is improved; and inputting the target features into a prediction network to obtain an action prediction sequence, and converting the action prediction sequence into a control instruction for the robot, thereby realizing accurate control of the robot based on physical constraint embedding.

The invention discloses a robot virtual-real cooperative training decision optimization system and method based on digital twinning, and the method comprises the following steps: collecting state data and disturbance data of an entity robot in a real environment, and carrying out the preprocessing of the data to generate standardized input; joint coding and state perturbation mapping are carried out on the standardized data, and a feature vector sequence embedded in a hyperspherical manifold space is generated; inputting to a virtual twin control body based on a hypersurface neural element structure, executing disturbance direction sensitive activation, and outputting an activated state vector sequence; virtual and entity control action prediction sequences are generated respectively, an embedded space difference vector is calculated, and control body parameters are updated based on a consistency optimization criterion; after convergence, the control body executes reasoning to generate a target control action sequence, the entity robot is driven to complete action execution, and control strategy optimization is achieved. According to the method, high-precision migration and rapid convergence of a robot control strategy are realized, and the execution stability in a complex disturbance environment is improved.

The invention provides a robot imitation learning system and method, electronic equipment and a medium, and the method comprises the steps: a multi-mode sensing module is used for collecting the sensing data of a robot operation environment; the feature coding module is used for carrying out feature coding on the perceptual data to obtain corresponding visual feature vectors, tactile feature vectors, language feature vectors and joint feature vectors, and aligning all the feature vectors to a unified time sequence; the cross-modal alignment and fusion module is used for generating fused multi-modal collaborative representation through the fusion network; and the action prediction and control module is used for decoding the multi-mode cooperative representation, generating an action instruction of the robot, and performing closed-loop control and error recovery on execution of the action instruction based on real-time feedback tactile data and joint sensor data. According to the invention, more stable closed-loop control of the robot is realized, and the fine operation capability of the robot is improved.

The invention discloses a robot control system and robot control method based on continuous action, and the robot control system comprises a data collection module which collects pressure distribution data through a sensor array; the classification module is used for generating a posture label according to the pressure distribution data in the preset time through a convolutional neural network; the action prediction module is used for receiving the posture label and outputting a feedback action according to the posture label; the execution module is used for converting the feedback action into a control parameter of the robot, so that the robot executes the feedback action; wherein the classification module comprises a sequence construction module, and the sequence construction module is used for generating a continuous behavior sequence from a plurality of posture tags; and the action prediction module is also used for outputting a corresponding feedback action sequence according to the continuous behavior sequence. According to the method, the posture labels in the preset time are constructed into the continuous behavior sequence according to the time sequence, so that the robot can respond to the continuous touch behaviors of the user in a period of time, and the interaction continuity and naturalness are enhanced.

The invention discloses a water boiling control method and system based on historical data analysis. The method comprises the following steps: acquiring action sensing data of a plurality of historical time points through a sensor arranged at the bottom of a water boiling kettle; based on the action sensing data corresponding to all the historical time points, training to obtain an action prediction model; on the basis of the action prediction model, predicting predicted user action characteristics at the current time point according to the action sensing data at the previous time point; and in response to the user action detection signal at the current time point, determining a water boiling control instruction corresponding to the water boiling kettle according to the predicted user action characteristics. Therefore, accurate water boiling instruction generation based on action time sequence analysis can be realized, and the accuracy of intelligent operation of the water boiling kettle and the user experience are improved.

The invention provides a light-weight pedestrian motion prediction method based on a mask, and is suitable for the technical field of human-computer interaction. According to the method, human body 3D skeleton point data is processed through space and time masks, key features are extracted by using a local perceptron constructed by a lightweight multilayer perceptron module and a cross-frame fusion device, the features are fused by using 1 * 1 convolution and splicing technologies, then a frequency-space domain pedestrian prediction action sequence is generated through a prediction module, and the frequency-space domain pedestrian prediction action sequence is obtained. And converting into a time-space domain sequence through an inverse discrete cosine converter. The method has the advantages of light model, quick response and suitability for real-time interaction. In addition, the strategy of gradually increasing the number of training frames improves the accuracy and stability of prediction.

The embodiments of this specification provide a method and system for regulating energy consumption. The method for regulating energy consumption includes obtaining first information and second information; determining a room status based on the first information; determining a resident's behavioral tendency based on the second information; generating an energy consumption control strategy based on the room status and the resident's behavioral tendency, and controlling the facilities in the room based on the energy consumption control strategy. The system for regulating energy consumption includes: an information acquisition module for obtaining first information and second information; a state determination module for determining a room status based on the first information; an action prediction module for determining a resident's behavioral tendency based on the second information; and a control module for generating a control strategy based on the room status and the resident's behavioral tendency, and controlling the facilities in the room based on the control strategy.