Action generation model training method, action generation method, video generation method, virtual object action control method, and related device
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2026-01-05
- Publication Date
- 2026-08-13
Smart Images
Figure CN2026070384_13082026_PF_FP_ABST
Abstract
Description
Motion generation model training, motion generation, video generation, virtual object motion control methods and related equipment Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and related equipment for training an action generation model, generating actions, generating videos, controlling virtual object actions. Background Technology
[0002] With the development of technologies such as Natural Language Processing (NLP), Computer Vision (CV), and Artificial Intelligence (AI), an increasing number of action generation models have emerged. These models can generate action sequences of various target objects, such as digital humans, robotic dogs, or movie characters, based on input text descriptions. In practical applications, action generation models often require large-scale, high-quality datasets for training to learn the relationship between text descriptions and action sequences. However, existing datasets are limited in size and cannot cover all possible language expressions and actions. The resulting action generation models often lack realism, perform poorly when faced with new or rare text descriptions, and have insufficient generalization ability. Summary of the Invention
[0003] This disclosure provides a method and related equipment for training an action generation model, generating actions, generating videos, and controlling virtual object actions, thereby effectively improving the model performance of the action generation model.
[0004] This disclosure provides a method for training an action generation model, comprising: acquiring a first sample set, wherein the first sample in the first sample set includes a positive sample pair and multiple different negative sample pairs, the positive sample pair including sample text descriptions and their corresponding standard action videos, and the negative sample pairs including sample text descriptions and their corresponding non-standard action videos; training a video retrieval network in the action generation model using the first sample set based on a contrastive learning approach to obtain a trained video retrieval network; acquiring a second sample set, wherein the second sample in the second sample set includes sample text descriptions and their corresponding predicted action videos and standard action sequences, wherein the predicted action videos are retrieved from an action video library by the trained video retrieval network based on the sample text descriptions; and training the action generation network in the action generation model using the second sample set to obtain a trained action generation network.
[0005] This disclosure also provides an action generation method, comprising: obtaining a text description, the text description being used to describe a target object performing a target action; inputting the text description into a video retrieval network in an action generation model, so as to retrieve a target action video matching the text description from an action video library through the video retrieval network; inputting the text description and the target action video into an action generation network in the action generation model, so as to output a target action sequence through the action generation network based on the target action video and the text description, wherein the target action sequence is used to control the target object to perform at least one target action.
[0006] This disclosure also provides a virtual object action control method, comprising: obtaining a text description, the text description being used to describe the virtual object performing a target action; inputting the text description into a video retrieval network in an action generation model to retrieve a target action video matching the text description from an action video library via the video retrieval network; inputting the text description and the target action video into an action generation network in the action generation model to output a target action sequence based on the target action video and the text description via the action generation network; and controlling the virtual object to perform at least one target action based on the target action sequence.
[0007] This disclosure also provides a video generation method, comprising: obtaining a text description, the text description being used to describe a target object performing a target action; inputting the text description into a video retrieval network in an action generation model to retrieve a target action video matching the text description from an action video library via the video retrieval network; inputting the text description and the target action video into an action generation network in the action generation model to output a target action sequence based on the target action video and the text description via the action generation network; and generating a video based on the target action sequence, in which the target object performs at least one target action.
[0008] This disclosure also provides an electronic device, including: a memory and a processor; the memory being configured to store a computer program; and the processor being coupled to the memory and configured to execute the computer program for performing steps in an action generation model training method, an action generation method, a video generation method, or a virtual object action control method.
[0009] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement steps in an action generation model training method, an action generation method, a video generation method, or a virtual object action control method.
[0010] The embodiments of the present disclosure further provide a computer program product, comprising computer programs / instructions, which, when executed by a processor, enable the processor to implement the steps in the action generation model training method, the action generation method, the video generation method, or the virtual object action control method.
[0011] In the embodiments of the present disclosure, a new action generation model including a video retrieval network and an action generation network is provided. In the model training stage, the video retrieval network is trained based on a contrast learning manner. The trained video retrieval network can accurately retrieve action videos corresponding to the text description from an action video library. The action generation network takes the text description and the action video as prior knowledge based on the latent association between the action video and the action sequence, better learns the relationship between the text description and the action sequence, and guides to generate an action sequence more consistent with the action behavior of the target object. Therefore, the action generation model trained in this way has good model performance, the action generated by the trained action generation model has high fidelity, and the performance is better when facing new or rare text descriptions, and the generalization ability is better. BRIEF DESCRIPTION OF DRAWINGS
[0012] The accompanying drawings, which are included to provide a further understanding of the present disclosure and constitute a part of the present disclosure, illustrate the exemplary embodiments of the present disclosure and are used to explain the present disclosure, but do not limit the present disclosure. In the drawings:
[0013] FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0014] FIG. 2 is a flowchart of an action generation model training method provided by an embodiment of the present disclosure;
[0015] FIG. 3 is a training principle diagram of an exemplary action generation model in a model training stage;
[0016] FIG. 4 is a contrast learning principle diagram for the video retrieval network in a fine-tuning stage;
[0017] FIG. 5 is a contrast learning principle diagram for the video retrieval network in a pre-training stage;
[0018] FIG. 6 is a training principle diagram for the action generation network in an SFT stage;
[0019] FIG. 7 is a training principle diagram for the action generation network in a pre-training stage;
[0020] FIG. 8 is a reinforcement learning principle diagram for the action generation network in a DPO stage;
[0021] FIG. 9 is a flowchart of an action generation method provided by an embodiment of the present disclosure;
[0022] Figure 10 is a schematic diagram of the working principle of an exemplary action generation model in the inference stage;
[0023] Figure 11 is a flowchart of a virtual object motion control method provided in an embodiment of this disclosure;
[0024] Figure 12 is a flowchart of a video generation method provided in an embodiment of this disclosure;
[0025] Figure 13 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.
[0027] In the embodiments of this disclosure, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the access relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. In the textual description of this disclosure, the character " / " generally indicates that the preceding and following associated objects have an "or" relationship. Furthermore, in the embodiments of this disclosure, "first," "second," "third," etc., are only used to distinguish the content of different objects and have no other special meaning.
[0028] It should be noted that, in the cases involving user information in the embodiments of this disclosure, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this disclosure (including but not limited to language models or large models) comply with relevant laws and standards.
[0029] The technical solutions of this disclosure and how they solve the aforementioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The technical solutions provided by each embodiment of this disclosure are described in detail below with reference to the accompanying drawings.
[0030] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of this disclosure. In the digital human scenario, a motion generation model can be used to automatically generate natural and smooth motion sequences based on text descriptions, and the digital human can be controlled to perform corresponding actions based on these motion sequences. Referring to ① in Figure 1, the user inputs a text description on the terminal device 10, such as "digital human performing classical dance." The terminal device 10 sends the text description to the server 20. Referring to ② in Figure 1, after receiving the text description, the server 20 uses the motion generation model to generate a corresponding motion sequence based on the text description, and binds the motion sequence to the 3D (three-dimensional) model of the digital human. The server 20 uses a graphics engine to render the bound 3D model, generating a digital human video. Referring to ③ in Figure 1, the server 20 returns the digital human video to the terminal device 10. The terminal device 10 plays the digital human video, such as a video of a digital human performing classical dance, so that the user can watch a video containing the digital human on the terminal device.
[0031] It should be noted that the application scenario shown in Figure 1 is merely an exemplary application scenario, and the embodiments of this disclosure do not limit the application scenario. The embodiments of this disclosure do not limit the devices included in Figure 1, nor do they limit the positional relationships between the devices in Figure 1.
[0032] Figure 2 is a flowchart of an action generation model training method provided in an embodiment of this disclosure. Referring to Figure 2, the method may include the following steps:
[0033] 101. Obtain the first sample set. The first sample in the first sample set includes a positive sample pair and multiple different negative sample pairs. The positive sample pair includes the sample text description and its corresponding standard action video. The negative sample pair includes the sample text description and its corresponding non-standard action video.
[0034] 102. Using the first sample set, the video retrieval network in the action generation model is trained using a contrastive learning approach to obtain the trained video retrieval network.
[0035] In practical applications, during the model training phase, a large number of sample text descriptions and their corresponding standard action videos are collected. These sample text descriptions, which are the text descriptions collected during model training, are typically natural language text. The sample text can describe the actions performed by various target objects, such as, but not limited to, "This person is climbing a mountain," "A digital human is dancing," "A robot dog is jumping," etc.
[0036] In this context, a standard action video refers to an action video that matches the sample text description. The actions performed by the target object in the standard action video conform to the standard actions described in the sample text. Examples of standard action videos include, but are not limited to, "human climbing a mountain video," "digital human dancing video," and "robot dog jumping video." An action video can be understood as video data recording the process of one or more objects (such as humans, animals, or robots) performing actions. For each sample text description, in addition to the accurate standard action video, a non-standard action video for that sample text description is also required. Standard and non-standard action videos are relative to the same sample text description; in a non-standard action video, the actions performed by the target object do not conform to the standard actions described in the sample text.
[0037] During the model training phase, a first sample set can be prepared for training the video retrieval network based on a large number of collected sample text descriptions and their corresponding standard action videos. The first sample set includes multiple first samples, each consisting of one positive sample pair and multiple different negative sample pairs. The positive sample pair includes the sample text description and its corresponding standard action video, while the negative sample pair includes the sample text description and its corresponding non-standard action video. For example, "This person is climbing a mountain" is denoted as text description t1, "Digital human dancing" as text description t2, and "Robot dog jumping" as text description t3. The standard mountain climbing video corresponding to "This person is climbing a mountain" is denoted as v1, the standard dancing video corresponding to "Digital human dancing" is denoted as v2, and the standard jumping video corresponding to "Robot dog jumping" is denoted as v3. The three positive sample pairs are denoted as (t1, v1), (t2, v2), and (t3, v3) respectively; the multiple negative samples corresponding to the positive sample pair (t1, v1) are (t1, v2) and (t1, v3); the multiple negative samples corresponding to the positive sample pair (t2, v2) are (t2, v1) and (t2, v3); and the multiple negative samples corresponding to the positive sample pair (t3, v3) are (t3, v1) and (t3, v2).
[0038] Referring to Figure 3, the action generation model may include a Video Retrieval Network. This network can be any existing machine learning model with video retrieval capabilities, focusing on retrieving videos relevant to a given query from a large amount of video data. In this embodiment, the video retrieval network can retrieve corresponding action videos from an action video library based on a text description (which can be understood as a query). An action video library refers to a resource platform or database that centrally stores and manages a large number of action videos. The content in the action video library is diverse, potentially covering various types of sports, such as physical activities (running, jumping, swimming), dance forms (street dance, ballet), daily activities (walking, shaking hands), and professional skills (surgical procedures, yoga poses), etc.
[0039] In this embodiment, the video retrieval network in the action generation model is trained using a first sample set based on contrastive learning, resulting in a trained video retrieval network. Contrastive learning enhances the capabilities of the video retrieval network, enabling it to better understand the input text description and accurately retrieve the corresponding action video from the action video library.
[0040] In practical applications, the video retrieval network can be trained multiple times using a first sample set to obtain a fully trained network. For each training round, multiple first samples can be obtained from the first sample set. For any first sample set containing positive sample pairs and multiple negative sample pairs, the sample text description and standard action video from the positive sample pair are input into the video retrieval network to be trained. The network extracts the text feature vector from the sample text description and the action feature vector from the target object in the standard action video. The action feature vector represents the action performed by the target object. The network calculates the similarity between the text feature vector from the sample text description and the action feature vector from the target object in the standard action video, and uses this similarity as the similarity of the positive sample pair. Similarly, the sample text description and non-standard action video from the negative sample pair are input into the network to be trained. The network extracts the text feature vector from the sample text description and the action feature vector from the target object in the non-standard action video, and calculates the similarity between the text feature vector from the sample text description and the action feature vector from the target object in the non-standard action video, and uses this similarity as the similarity of the negative sample pair. The video retrieval network to be trained calculates the adversarial loss value corresponding to the current round of model training based on the similarity of each positive sample pair and each negative sample pair. To minimize the adversarial loss value corresponding to the current round of model training, the network parameters (also known as model parameters) of the video retrieval network are adjusted so that the distance between the text description and its corresponding standard action video in the feature space is closer (i.e., the similarity of positive sample pairs is higher), while the distance between the text description and other unrelated videos (i.e., non-standard action videos) in the feature space is further apart (i.e., the similarity of negative sample pairs is lower). The video retrieval network with adjusted network parameters is used as the next video retrieval network to be trained, and multiple first samples are used to perform the next model training until the end-of-training condition is met. The end-of-training condition can be that the model has been trained a specified number of times (e.g., 100 times), or that the adversarial loss value corresponding to the current round of model training has reached its minimum value. After the training task is completed, the trained video retrieval network can be used to retrieve the corresponding action video from the action video library based on the text description.
[0041] In this embodiment, any contrastive loss function can be used to determine the adversarial loss value. The contrastive loss function is used to make positive sample pairs (similar sample pairs) close in the representation space, while negative sample pairs (dissimilar sample pairs) are far apart in the representation space. Any contrastive loss function includes, but is not limited to, Info Noise-contrastive estimation (Info NCE), Noise-contrastive estimation (NCE), etc. For example, the Info NCE contrastive loss function is given by the following formula (1):
[0042] When using formula (1) to calculate the adversarial loss value L corresponding to this round of model training, z i z represents the text feature vector describing the text of the i-th sample in the first sample; i + z represents the motion feature vector of the target object in the standard motion video among the positive sample pairs included in the i-th first sample; j Let represent the motion feature vector of the target object in the non-standard motion video within the j-th negative sample pair included in the i-th first sample; τ represents the temperature parameter; N is a positive integer representing the number of first samples in this round of model training; and K represents the number of negative sample pairs in the i-th first sample. sim() represents the similarity calculation function (usually the cosine similarity function), sim(z i ,z i + ) represents the similarity between positive sample pairs; sim(z) i ,z j ) represents the similarity of negative sample pairs; exp() represents the exponential function, and log() represents the logarithmic function.
[0043] In this embodiment, the network structure of the video retrieval network is not limited. For example, the network structure of the video retrieval network includes, but is not limited to: Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), and Transformer architecture. The Transformer architecture is designed to address the problems of traditional recurrent neural networks and long short-term memory networks when processing long sequence data, particularly the slow training speed and difficulty in parallelization.
[0044] Optionally, to improve the retrieval performance of the video retrieval network and thus the model performance of the action generation model, as shown in Figure 3, the video retrieval network may include: a predicate semantic extraction module, an argument semantic extraction module, a first keypoint detection module, an action encoding module, a first video feature extraction module, and a similarity calculation module. Correspondingly, the implementation of training the video retrieval network in the action generation model using a first sample set based on a contrastive learning approach can include steps S1 to S4, as detailed below:
[0045] S1. For any first sample in the first sample set, extract the first semantic feature vector of the predicate in the sample text description included in the first sample through the predicate semantic extraction module; and extract the second semantic feature vector of the argument in the sample text description through the argument semantic extraction module.
[0046] Specifically, the predicate is the core part of a sentence, usually a verb or adjective, describing an action, state, or event. For example, in the sentence "Xiaoming ate an apple," "ate" is the predicate. Similarly, in the sentence "This person is climbing a mountain," "climb" is the predicate. The argument is the specific object or participant involved in the predicate, answering questions about the predicate such as "who," "what," "where," and "when." For example, in the sentence "Xiaoming ate an apple," both "Xiaoming" and "apple" are arguments. Again, in the sentence "This person is climbing a mountain," both "person" and "mountain" are arguments.
[0047] In this embodiment, the predicate semantic extraction module can extract predicates and their related semantic information from the text description. The predicate semantic extraction module can be any neural network module with predicate semantic information extraction capabilities, such as an encoder with predicate semantic information extraction capabilities. In this embodiment, the sample text description from the first sample is input into the predicate semantic extraction module. The predicate semantic extraction module identifies the predicates in the sample text description and extracts the semantic information of the predicates. The semantic information of the predicates can be represented in the form of feature vectors. Here, the semantic information of the predicates extracted from the sample text description is called the first semantic feature vector of the predicate.
[0048] In this embodiment, the argument semantic extraction module can extract arguments and their related semantic information from the text description. The argument semantic extraction module can be any neural network module with argument semantic information extraction capabilities, such as an encoder with argument semantic information extraction capabilities. In this embodiment, the sample text description from the first sample is input into the argument semantic extraction module. The argument semantic extraction module identifies the arguments in the sample text description and extracts the semantic information of the arguments. The semantic information of the arguments can be represented in the form of feature vectors. Here, the semantic information of the arguments extracted from the sample text description is called the second semantic feature vector of the arguments.
[0049] S2. The first key point detection module performs key point detection on the action video included in the first sample to obtain the first key point information of the target object in the action video included in the first sample, and the action encoding module performs feature encoding on the first key point information to obtain the first action feature vector; and the first video feature extraction module extracts features from the action video included in the first sample to obtain the first video feature vector.
[0050] In this embodiment, the first keypoint detection module can be any neural network module with keypoint detection functionality. The first keypoint detection module performs keypoint detection on the action videos included in the first sample to obtain the first keypoint information of the target object in the action videos included in the first sample. The action videos included in the first sample can be standard action videos in positive sample pairs or non-standard action videos in negative sample pairs. The target object refers to the object performing the action; for example, the target object in a "human climbing a mountain video" is a human, the target object in a "digital human dancing video" is a digital human, and the target object in a "robot dog jumping video" is a robot dog. The keypoint information corresponding to different target objects is different. Taking the human body as an example, the keypoint information of the human body includes, but is not limited to: the position coordinates and posture information of the head keypoints, the position coordinates and posture information of the torso keypoints, the position coordinates and posture information of the upper limb keypoints, and the position coordinates and posture information of the lower limb keypoints, etc.
[0051] In this embodiment, the action encoding module can be any neural network module with the function of feature encoding of key point information, such as an encoder with the function of feature encoding of key point information. The action encoding module performs feature encoding on the first key point information to obtain a first action feature vector, which can represent the action information of the target object.
[0052] In this embodiment, the first video feature extraction module can be any neural network module with video feature extraction capabilities. The first video feature extraction module extracts features from the action videos included in the first sample to obtain a first video feature vector.
[0053] S3. The similarity calculation module calculates the first similarity based on the first semantic feature vector and the first action feature vector; and calculates the second similarity based on the second semantic feature vector and the first video feature vector; and obtains the final similarity between the action video in the first sample and the sample text description based on the first similarity and the second similarity.
[0054] In this embodiment, the similarity calculation module can be understood as a functional module with similarity calculation capabilities. The similarity calculation module can use various similarity measurement methods, such as cosine similarity and Euclidean distance, to calculate the similarity between feature vectors. Here, the similarity between the first semantic feature vector of the predicate and the first action feature vector is calculated as the first similarity between the action video and the sample text description. The similarity between the second semantic feature vector of the argument and the first video feature vector is calculated as the second similarity between the action video and the sample text description. Various operations, such as weighted summation, averaging, or accumulation, are performed on the first and second similarities to obtain the final similarity between the action video and the sample text description.
[0055] It is worth noting that the first sample can include one positive sample pair and multiple negative sample pairs. Therefore, the first sample can include one standard action video and multiple non-standard action videos, meaning the first sample can include multiple action videos. For each action video in the multiple action videos, steps S2 and S3 need to be executed. That is, after executing steps S2 and S3, the final similarity between the standard action video and the sample text description in the first sample can be obtained, as can the final similarity between the non-standard action videos and the sample text description in the first sample.
[0056] S4. The final similarity between the standard action video and the sample text description in the first sample is taken as the final similarity of the positive sample pair, and the final similarity between the non-standard action video and the sample text description in the first sample is taken as the final similarity of the negative sample pair; and the first contrastive learning loss is determined based on the final similarity of each positive sample pair and the final similarity of each negative sample pair, and the network parameters of the video retrieval network are adjusted based on the first contrastive learning loss.
[0057] Referring to Figure 4, the first contrastive learning loss can be calculated based on the final similarity between the standard action video and the sample text description, and the final similarity between the non-standard action video in the first sample and the sample text description. For example, the Info NCE contrastive loss function is as follows (2):
[0058] When using formula (2) to calculate the adversarial loss value L corresponding to this round of model training, z i 1 z represents the first semantic feature vector of the predicate in the text description of the i-th first sample; i 2 z represents the second semantic feature vector of arguments in the text description of the i-th first sample; i 1+ z represents the first motion feature vector of the target object in the standard motion video among the positive samples included in the i-th first sample; i 2+ z represents the first video feature vector of the standard action video in the positive sample pair included in the i-th first sample. j 1 z represents the first motion feature vector of the target object in the non-standard motion video of the j-th negative sample pair included in the i-th first sample; j 2 It represents the first video feature vector of the non-standard action video in the j-th negative sample pair included in the i-th first sample.
[0059] sim(z i 1 ,z i 1+ )+sim(z i 2 ,z i 2+ ) represents the final similarity of positive sample pairs in the i-th first sample; sim(z) i 1 ,z j 1 )+sim(z i 2 ,z j 2 ) represents the final similarity of the j-th negative sample pair in the i-th first sample.
[0060] In some optional embodiments, the training phase of the entire video retrieval network sequentially includes a pre-training phase and a fine-tuning phase. During the pre-training phase, the predicate semantic extraction module and the action encoding module can be trained, or the argument semantic extraction module and the first video feature extraction module can be trained. Thus, during the fine-tuning phase, the video retrieval network can be fine-tuned using a first contrastive learning loss. Through the pre-training and fine-tuning phases, the video retrieval capability of the video retrieval network can be improved.
[0061] Based on the above, before extracting the first semantic feature vector of the predicate in the sample text description through the predicate semantic extraction module, during the pre-training stage, for any first sample in the first sample set, the third semantic feature vector of the predicate in the sample text description included in the first sample can also be extracted through the predicate semantic extraction module; the first keypoint detection module performs keypoint detection on the action videos included in the first sample to obtain the second keypoint information of the target object in the action videos included in the first sample, and the action encoding module performs feature encoding on the second keypoint information to obtain the second action feature vector of the target object in the action videos included in the first sample; the similarity of positive sample pairs in the predicate dimension is determined based on the third semantic feature vector and the second action feature vector of the target object in the standard action videos included in the first sample, and the similarity of negative sample pairs in the predicate dimension is determined based on the third semantic feature vector and the second action feature vector of the target object in the non-standard action videos included in the first sample. The second contrastive learning loss is determined based on the similarity of each positive sample pair in the predicate dimension and the similarity of each negative sample pair in the predicate dimension, and the network parameters of the predicate semantic extraction module and the action encoding module are adjusted based on the second contrastive learning loss. And / or, during the pre-training phase, the fourth semantic feature vector of arguments in the text description of the first sample can be extracted through the argument semantic extraction module; and the second video feature vector of the action video included in the first sample can be obtained by extracting features from the action video included in the first sample through the first video feature extraction module; the similarity of positive sample pairs in the argument dimension is determined based on the fourth semantic feature vector and the second video feature vector of the standard action video included in the first sample, and the similarity of negative sample pairs in the argument dimension is determined based on the fourth semantic feature vector and the second video feature vector of the non-standard action video included in the first sample; the third contrastive learning loss is determined based on the similarity of each positive sample pair in the argument dimension and the similarity of each negative sample pair in the argument dimension, and the network parameters of the argument semantic extraction module and the first video feature extraction module are adjusted based on the third contrastive learning loss.
[0062] Referring to Figure 5, during the pre-training phase, the second contrastive learning loss can be determined based on the third semantic feature vector of the predicate in the sample text description, the second action feature vector of the target object in the standard action video, and the second action feature vector in the non-standard action video. Taking the calculation of the second contrastive learning loss using the contrastive loss function shown in formula (1) as an example, when using formula (1) to calculate the second contrastive learning loss L corresponding to this round of model training, z i z represents the third semantic feature vector of the predicate in the i-th first sample; i + z represents the second motion feature vector of the target object in the standard motion video among the positive sample pairs included in the i-th first sample; j It represents the second motion feature vector of the target object in the non-standard motion video in the j-th negative sample pair included in the i-th first sample.
[0063] Referring to Figure 5, during the pre-training phase, the third contrastive learning loss can be determined based on the fourth semantic feature vector of arguments in the sample text description, the second video feature vector of the standard action video, and the second video feature vector of the non-standard action video. Taking the calculation of the third contrastive learning loss using the contrastive loss function shown in formula (1) as an example, when using formula (1) to calculate the third contrastive learning loss L corresponding to this round of model training, z i z represents the fourth semantic feature vector of the arguments in the i-th first sample; i + z represents the second video feature vector of the standard action video in the positive sample pair included in the i-th first sample; j It represents the second video feature vector of the non-standard action video in the j-th negative sample pair included in the i-th first sample.
[0064] It is worth noting that the video retrieval network not only focuses on the predicate semantic information and argument semantic information of the text description, but also on the video features and action features of the action video. In this way, the trained video retrieval network can pay more attention to the action posture of the target object and retrieve action videos that are more relevant to the action behavior of the target object from the action video library.
[0065] It is worth noting that the video retrieval network decomposes text descriptions into argument granularity and predicate granularity, and action videos into action granularity and video feature granularity. It uses a contrastive learning approach to match text features at the predicate granularity with action features at the action granularity, and to match text features at the argument granularity with video features at the video feature granularity. Furthermore, it introduces a learnable similarity calculation module for feature fusion. The video retrieval network trained in this way performs video retrieval at multiple levels, thereby improving the accuracy of video retrieval.
[0066] 103. Obtain the second sample set. The second sample in the second sample set includes sample text descriptions and their corresponding predicted action videos and standard action sequences. The predicted action videos are retrieved from the action video library by the trained video retrieval network based on the sample text descriptions.
[0067] 104. Use the second sample set to train the action generation network in the action generation model to obtain the trained action generation network.
[0068] In practical applications, after training the video retrieval network in the action generation model using the first sample set, the action generation network can be further trained. To train the action generation network, a second sample set is needed. This second sample set includes multiple second samples, each containing a sample text description, its corresponding predicted action video, and a standard action sequence. The sample text description is input into the video retrieval network, which then retrieves the action video corresponding to the sample text description from the action video library. The retrieved action video corresponding to the sample text description is referred to as the predicted action video. The standard action sequence includes action data for at least the standard actions corresponding to the sample text description; the standard actions are those that conform to the sample text description.
[0069] In practical applications, a second sample set can be used to perform multiple rounds of model training. In each round of model training, multiple second samples are obtained from the second sample set. The sample text descriptions and their corresponding predicted action videos from the second samples are input into the action generation network to be trained, and the predicted action sequences output by the action generation network are obtained. The loss value of the second samples is calculated based on the standard action sequence corresponding to the sample text descriptions and the predicted action sequence. The loss values of each second sample in this round of model training are weighted, summed, averaged, or accumulated to obtain the total loss value of this round of model training. The network parameters of the action generation network are adjusted with the goal of minimizing the total loss value of this round of model training. The action generation network with adjusted network parameters is used as the action generation network to be trained in the next round, and the next round of model training is performed using multiple second samples until the end-of-training condition is met. The end-of-training condition can be that the number of model training iterations reaches a specified number (e.g., 100 times), or that the total loss value corresponding to this round of model training has reached the minimum value.
[0070] When calculating the loss value for the second sample, any loss function can be used, including but not limited to: mean squared error loss function, L2 loss function (also known as Euclidean distance), L1 loss function (Manhattan distance), cross-entropy loss function, softmax loss function, and Focal loss function, which is mainly used to address the imbalance between easy and difficult samples. Specifically, the standard action sequence corresponding to the sample text description in the second sample is used as the true value input into the loss function, and the predicted action sequence corresponding to the sample text description in the second sample is used as the predicted value input into the loss function. The loss function calculates the loss value based on the input true value and predicted value.
[0071] In this embodiment, the network structure of the action generation network is not limited. For example, the network structure of the action generation network includes, but is not limited to: recurrent neural networks, convolutional neural networks, long short-term memory networks, and the Transformer architecture.
[0072] Further optionally, to improve the performance of the action generation model, referring to Figure 3, the action generation network may include: a second video feature module, a second keypoint detection module, a first mapping module, a second mapping module, and a language model; correspondingly, the action generation network in the action generation model is trained using the second sample set to obtain the trained action generation network. The implementation method may include steps S10 to S20, as follows:
[0073] S10. For any second sample in the second sample set, extract the third video feature vector of the predicted action video in the second sample through the second video feature module, and use the first mapping module to map the dimension of the third video feature vector; perform key point detection on the predicted action video through the second key point detection module to obtain the third key point information of the target object in the predicted action video, and use the second mapping module to map the dimension of the third key point information.
[0074] S20. Input the sample text description, the mapped third video feature vector, and the mapped third key point information from the second sample into the language model, and obtain the predicted action sequence corresponding to the sample text description output by the language model; adjust the network parameters of the action generation network according to the standard action sequence and the predicted action sequence in the second sample.
[0075] Specifically, the second video feature module can be any neural network module with video feature extraction capabilities. The second keypoint detection module can be any neural network module with keypoint detection capabilities. The first or second mapping module can be any neural network module with feature dimension mapping capabilities. The first or second mapping module is used to ensure that the data dimension of the input language model meets the requirements of the language model, and the first or second mapping module can perform dimensionality reduction or dimensionality increase mapping processes. Language models include, but are not limited to: Large Language Model (LLM), Autoregressive Language Model (ALM), and Generative Pre-trained Transformer (GPT).
[0076] Referring to Figure 6, the sample text descriptions and their corresponding predicted action videos from the second sample are input into the action generation network to be trained, and the predicted action sequences output by the action generation network are obtained. The loss value of the second sample is calculated based on the standard action sequence corresponding to the sample text descriptions in the second sample and the predicted action sequence. The loss values of each second sample in this round of model training are weighted, summed, averaged, or accumulated to obtain the total loss value of this round of model training. The network parameters of the action generation network are adjusted to minimize the total loss value of this round of model training. The action generation network with adjusted network parameters is used as the next action generation network to be trained, and the next model training is performed using multiple second samples until the end-of-training condition is met.
[0077] It is worth noting that the action generation network, based on 2D (2D) action videos retrieved by the video retrieval network, leverages the powerful generalization ability of the language model to integrate textual modal information and video modal information as prior knowledge to guide the generation of 3D action sequences. This can greatly alleviate the challenge of generating action sequences from open vocabulary. Open vocabulary can be understood as words that the action generation model has not learned or are rarely learned during the training phase.
[0078] It is understandable that using the second sample set to train the action generation network in the action generation model is a form of Supervised Fine-Tuning (SFT). During the SFT stage of the action generation network, the network parameters of each network model can be adjusted. Alternatively, the network parameters of the second video feature extraction module can be kept constant, while the network parameters of the other network models in the action generation network can be adjusted.
[0079] In some optional embodiments, the action generation network can be pre-trained before the SFT stage to improve its model performance. Therefore, before extracting the third video feature vector of the predicted action video from the second sample using the second video feature module, during the pre-training stage of the action generation network, the second video feature module extracts the fourth video feature vector of the predicted action video, and the first mapping module maps the dimension of the fourth video feature vector. The second keypoint detection module performs keypoint detection on the predicted action video to obtain the fourth keypoint information of the target object in the predicted action video, and the second mapping module maps the dimension of the fourth keypoint information. The mapped fourth video feature vector and the mapped fourth keypoint information are input into the language model, and the predicted text description output by the language model is obtained. The network parameters of the first and second mapping modules are adjusted based on the sample text description and the predicted text description.
[0080] Referring to Figure 7, the predicted action video from the second sample is input into the action generation network to be trained, and the predicted text description output by the speech model in the action generation network is obtained. The loss value of the second sample is calculated based on the feature vectors of the sample text descriptions and the feature vectors of the predicted text descriptions. The total loss value of the action generation network is obtained by weighted summation, averaging, or accumulation of the loss values of each second sample in this round of model training. The network parameters of the action generation network are adjusted to minimize the total loss value of this round of model training. The action generation network with adjusted network parameters is used as the next action generation network to be trained, and the next model training is performed using multiple second samples until the end-of-training condition is met.
[0081] During the pre-training phase of the action generation network, only the network parameters of the first and second mapping modules can be adjusted, while the network parameters of other network modules in the action generation network remain unchanged. Alternatively, the network parameters of each individual network model within the action generation network can be adjusted without restriction.
[0082] In some optional embodiments, after completing the SFT stage of the action generation network, the language model can be further optimized using Direct Preference Optimization (DPO) to enhance its learning of video features and text descriptions. This allows the language model to learn to adjust its output action sequences based on user preferences, resulting in more satisfying results and improved user experience. Based on this, after adjusting the network parameters of the action generation network according to the standard and predicted action sequences in the second sample, the process can further include: inputting the sample text descriptions and their predicted action videos multiple times into the action generation network to obtain multiple candidate action sequences output by the network; scoring the candidate action sequences using a reward model to obtain a score result; selecting a preferred predicted action sequence and a non-preferred predicted action sequence from the multiple candidate action sequences based on the score result; and optimizing the language model using Direct Preference Optimization based on the preferred and non-preferred action sequences.
[0083] In practical applications, referring to Figure 8, during the DPO stage, multiple sample text descriptions can be accurately generated. Each sample text description is input into the action generation network multiple times, and the action generation network outputs multiple action sequences corresponding to each sample text description. Here, the action sequences output by the action generation network are called candidate action sequences. A reward model is designed to score the candidate action sequences, obtaining the scoring results. In practical applications, there are no restrictions on the scoring method of the reward model. Further optionally, a dual-alignment reward model can be constructed to guide the alignment between text modalities and action modalities and the alignment between predicted action sequences and standard action sequences. This enables the reward model to accurately score the output of the language model, thereby achieving action-centric dual-modal alignment DPO. Based on this, the reward model is used to score the candidate action sequences, and the scoring results are obtained as follows: the reward model calculates the first distance information between the candidate action sequence and the standard action sequence, and calculates the second distance information between the candidate action sequence and the sample text description. The candidate action sequences are then scored based on the first and second distance information to obtain the scoring results.
[0084] Specifically, for each candidate action sequence among multiple candidate action sequences in each sample text description, the reward model can use various algorithms such as Euclidean distance, Dynamic Time Warping (DTW), and Cosine Similarity to calculate the first distance information between the candidate action sequence and the standard action sequence, and the second distance information between the candidate action sequence and the sample text description. Specifically, the second distance information between the candidate action sequence and the sample text description is calculated based on the feature vectors corresponding to the candidate action sequence and the sample text description. It can be understood that the first distance information reflects the similarity (or difference) between the candidate action sequence and the standard action sequence, while the second distance information reflects the similarity between the candidate action sequence and the sample text description. The reward model can perform various operations on the first and second distance information, such as weighted summation, averaging, or accumulation, to obtain a score for the candidate action sequence. Based on the score, the candidate action sequence with the higher score can be selected as the preferred prediction action sequence, and the candidate action sequence with the lower score can be selected as the non-preferred prediction action sequence. In contrast, a preferred action sequence can be understood as the expected sequence of predicted actions output by the language model, while a non-preferred action sequence can be understood as the sequence of predicted actions not expected to be output by the language model. For example, the candidate action sequence with the highest score is selected as the preferred action sequence, and the candidate action sequence with the lowest score is selected as the non-preferred action sequence. Another example is randomly selecting one from candidate action sequences with scores above a preset threshold as the preferred action sequence, and randomly selecting one from candidate action sequences with scores below a preset threshold as the non-preferred action sequence. For instance, the candidate action sequence with the highest score is selected as the preferred action sequence, and a random selection is made from candidate action sequences with scores below a preset threshold as the non-preferred action sequence. Of course, the selection of preferred and non-preferred action sequences is not limited to the examples above.
[0085] In this embodiment, referring to Figure 8, multiple sample text descriptions are collected, each corresponding to a preference prediction action sequence and a non-preference prediction action sequence. The language model can be optimized based on the preference prediction action sequence and the non-preference prediction action sequence using the DPO method. For example, DPO can use the loss function shown in formula (3).
[0086] In formula (3), Let σ represent the loss function of DPO, β represent the sigmoid activation function, and β represent the hyperparameter, typically ranging from 0.1 to 0.5. Let x represent the sample text description, and y represent the loss function of DPO. w y represents the preference for predicting action sequences.l Let V represent the non-biased predicted action sequence, and let π represent the action video retrieved from the action video library that corresponds to the sample text description. θ (y w |x, V) represents the probability distribution of the policy model's prediction preference for predicting action sequences given the input action video and sample text description, π θ (y l |x,V) represents the probability distribution of the policy model predicting a non-preferred action sequence given the input action video and sample text description, π ref (y l |x,V) represents the probability distribution of the reference model's prediction of a non-biased action sequence given the input action video and sample text description. This represents the expectation over the entire preference dataset, which includes multiple sample text descriptions of their respective preference prediction action sequences and non-preference prediction action sequences. In the DPO phase, the language model currently being optimized can be used as the policy model, while the language model not being optimized and used to provide a benchmark can be used as the reference model.
[0087] In practical applications, the goal can be to minimize the loss value of DPO to optimize the language model, making the language model output more consistent with the expected predicted action sequence. In this way, the DPO method can be used to enhance the language model's learning of video features and text descriptions.
[0088] The technical solution provided in this disclosure offers a novel action generation model comprising a video retrieval network and an action generation network. During the model training phase, the video retrieval network is trained using a contrastive learning approach. The trained video retrieval network can accurately retrieve action videos corresponding to text descriptions from an action video library. The action generation network, based on the potential correlation between action videos and action sequences, uses text descriptions and action videos as prior knowledge to better learn the relationship between text descriptions and action sequences, guiding the generation of action sequences that more closely match the target object's actions. Therefore, the action generation model trained in this way exhibits good performance, generates highly realistic actions, demonstrates superior performance when faced with new or rare text descriptions, and exhibits good generalization ability.
[0089] It is worth noting that even with the limited size of existing datasets, with the assistance of an action video library that provides rich action information, we can first use a video retrieval network to retrieve the action video corresponding to the text description from the action video library. Using the text description and the retrieved action video as prior knowledge for generating action sequences, we can generate more realistic action sequences. This can effectively solve the problem that "the existing dataset is limited in size and cannot cover all possible language expressions and actions", greatly alleviate the difficulty of generating open-vocabulary actions, and improve the generalization ability of action generation models.
[0090] Figure 9 is a flowchart of an action generation method provided in an embodiment of this disclosure. Referring to Figure 9, the method may include the following steps:
[0091] 201. Obtain the text description. The text description is used to describe the target object performing the target action.
[0092] 202. Input the text description into the video retrieval network in the action generation model to retrieve the target action video that matches the text description from the action video library.
[0093] 203. Input the text description and the target action video into the action generation network in the action generation model, so that the action generation network outputs a target action sequence based on the target action video and the text description, wherein the target action sequence is used to control the target object to perform at least one target action.
[0094] Specifically, in the action generation stage, the text description is input into the action generation model. The action generation model first uses a video retrieval network to retrieve target action videos that match the text description from the action video library. Then, the action generation network generates actions based on the target action videos and the text description to obtain a target action sequence that controls the target object to perform at least one target action.
[0095] In different application scenarios, the target object, target action, target action video, and target action sequence are different. For example, the text description is "This person is climbing a mountain," the target object is a person, the target action video is a video of climbing a mountain, the target action is the action of climbing a mountain, and the target action sequence is data representing various actions of a person climbing a mountain.
[0096] Taking the human body as an example, the target action sequence can include the position coordinates and pose data of the human body's key points, as well as shape and pose parameters conforming to the SMPL (Skinned Multi-Person Linear Model, Shape and Motion Learning Prior) model definition. Shape parameters control the human body's shape, such as height and build. Pose parameters control the human body's posture, such as the position coordinates and rotation angles of the joints. Taking an animal as an example, the target action sequence can include the animal's key point coordinates and pose data, as well as its shape and pose parameters. Of course, the motion data for the target actions in the target action sequence will differ depending on the specific target object, depending on the actual situation.
[0097] In practical applications, action video libraries providing a large number of action videos are readily available, offering a wider range of action information and covering diverse movements. Action generation models directly retrieve 2D action videos from these libraries, leveraging the potential correlation between 2D and 3D action sequences. Using text descriptions and action videos as prior knowledge, they better learn the relationship between text descriptions and action sequences, guiding the generation of action sequences that more closely match the target object's behavior. This effectively overcomes the limitation of existing datasets in training action generation models.
[0098] In practical applications, there are no restrictions on the training method and network structure of the action generation model. Further optionally, to improve the model performance of the action generation model, as shown in Figure 10, the action generation model may include a video retrieval network. The video retrieval network may include: a predicate semantic extraction module, an argument semantic extraction module, a first keypoint detection module, an action encoding module, a first video feature extraction module, and a similarity calculation module. Correspondingly, the implementation of retrieving target action videos matching the text description from the action video library through the video retrieval network is as follows: the predicate semantic extraction module extracts the semantic feature vectors of the predicates in the text description; the argument semantic extraction module extracts the semantic feature vectors of the arguments in the text description; multiple action videos are obtained from the action video library, and the first keypoint detection module performs keypoint detection on each action video to obtain the keypoint information of the target object in each action video; and the action encoding module extracts the keypoint information of the target object in the action video. The key point information of the image is used for feature encoding to obtain the action feature vector of the target object in each action video; the first video feature extraction module extracts features from each action video to obtain the video feature vector of each action video; the similarity calculation module calculates the similarity between each action video and the text description in the predicate dimension based on the semantic feature vector of the predicate and the action feature vector of the target object in each action video; and calculates the similarity between each action video and the text description in the argument dimension based on the semantic feature vector of the argument and the video feature vector of each action video; and obtains the final similarity between each action video and the text description based on the similarity in the predicate dimension and the similarity in the argument dimension; based on the final similarity between each action video and the text description, the target action video that matches the text description is selected from multiple action videos.
[0099] In practical applications, for multiple action videos obtained from an action video library, after calculating the final similarity between each action video and the text description, the action video with the highest final similarity can be selected as the target action video to match the text description. Alternatively, an action video can be randomly selected from the top N action videos with the highest final similarity, where N is a positive integer. Another option is to randomly select an action video from among the action videos with a final similarity greater than a flexibly set similarity threshold. Of course, the methods for selecting the target action video to match the text description are not limited to the examples above.
[0100] Optionally, to improve the performance of the action generation model, referring to Figure 10, the action generation model may further include an action generation network, which may include: a second video feature module, a second keypoint detection module, a first mapping module, a second mapping module, and a language model. Accordingly, the implementation of outputting the target action sequence based on the target action video and text description through the action generation network is as follows: the second video feature module extracts the target video feature vector of the target action video, and the first mapping module maps the dimension of the target video feature vector; the second keypoint detection module performs keypoint detection on the target action video to obtain the target keypoint information of the target object in the target action video, and the second mapping module maps the dimension of the target keypoint information; the text description, the mapped target video feature vector, and the mapped target keypoint information are input into the language model, and the target action sequence corresponding to the text description output by the language model is obtained.
[0101] For detailed implementation methods of each step in this embodiment, please refer to the relevant descriptions in the foregoing embodiments; they will not be elaborated upon here.
[0102] The technical solution provided in this disclosure involves an action generation model that uses a video retrieval network to directly retrieve action videos matching the text description from an action video library that provides a large number of action videos. The action generation model also utilizes the action generation network to learn the relationship between the text description and the action sequence based on the potential association between the action video and the action sequence, using the text description and the action video as prior knowledge. This guides the generation of action sequences that are more in line with the action behavior of the target object. As a result, the action generation model generates highly realistic actions, exhibits better performance when faced with new or rare text descriptions, and has good generalization ability.
[0103] Figure 11 is a flowchart of a virtual object motion control method provided in an embodiment of this disclosure. Referring to Figure 10, the method may include the following steps:
[0104] 301. Obtain the text description, which describes the target action performed by the virtual object;
[0105] 302. Input the text description into the video retrieval network in the action generation model, so as to retrieve the target action video that matches the text description from the action video library through the video retrieval network;
[0106] 303. Input the text description and target action video into the action generation network in the action generation model, so that the action generation network can output the target action sequence based on the target action video and text description;
[0107] 304. Control the virtual object to execute at least one target action according to the target action sequence.
[0108] Specifically, virtual objects can be target objects in various virtual scenes, such as, but not limited to, digital humans, virtual pets, virtual robots, virtual anchors, virtual customer service representatives, and so on. By using motion generation models to generate more realistic motion sequences within virtual scenes, the realism of controlling virtual objects to perform target actions can be greatly enhanced, significantly increasing user immersion and engagement, and providing users with a richer and more authentic virtual experience.
[0109] For detailed implementation methods and technical effects of each step in this embodiment, please refer to the relevant descriptions in the foregoing embodiments; they will not be elaborated upon here.
[0110] Figure 12 is a flowchart of a video generation method provided in an embodiment of this disclosure. Referring to Figure 12, the method may include the following steps:
[0111] 401. Obtain the text description. The text description is used to describe the target object performing the target action.
[0112] 402. Input the text description into the video retrieval network in the action generation model to retrieve target action videos that match the text description from the action video library through the video retrieval network.
[0113] 403. Input the text description and target action video into the action generation network in the action generation model, so that the action generation network can output the target action sequence based on the target action video and text description.
[0114] 404. Generate a video based on the target action sequence, in which the target object performs at least one target action.
[0115] Specifically, there is a demand for text-based videos in various scenarios such as film production, game development, and short video creation. Motion generation models are used to generate more realistic motion sequences based on text descriptions. These motion sequences are then bound to a 3D model of the target object, and the bound 3D model is rendered to generate a corresponding video. In the video, the target object performs at least one target action. Target objects can be, for example, virtual movie protagonists, virtual game characters, virtual anchors, etc., and target actions can be, for example, a virtual movie protagonist dancing, a virtual game character performing an attack, or a virtual anchor shouting, etc.
[0116] In video production, using motion generation models to generate more realistic motion sequences allows for control over the realism of target objects performing target actions in the generated video, resulting in higher video quality. This enables creators to more efficiently transform their ideas into vivid visual stories, thus enhancing the user experience.
[0117] For detailed implementation methods and technical effects of each step in this embodiment, please refer to the relevant descriptions in the foregoing embodiments; they will not be elaborated upon here.
[0118] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; or the execution subject of steps 101 and 102 can be device A, and the execution subject of steps 103 to 104 can be device B; and so on.
[0119] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.
[0120] Figure 13 is a schematic diagram of an electronic device provided in an embodiment of this disclosure. As shown in Figure 13, the electronic device includes: a memory 11 and a processor 12;
[0121] Memory 11 is configured to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, data structures, contact data, phone book data, messages, pictures, videos, etc.
[0122] The processor 12, coupled to the memory 11, is configured to execute computer programs in the memory 11 for: performing steps in an action generation model training method, an action generation method, a video generation method, or a virtual object motion control method.
[0123] Optionally, as shown in Figure 13, the electronic device may also include other components such as a communication component 13, a display 14, a power supply component 15, and an audio component 16. Figure 13 only schematically shows some components and does not imply that the electronic device only includes the components shown in Figure 13. Furthermore, the components within the dashed boxes in Figure 13 are optional, not mandatory, and their specific inclusion depends on the product form of the electronic device. The electronic device of this embodiment can be implemented as a desktop computer, laptop computer, smartphone, or IoT (Internet of Things) device, or as a server-side device such as a conventional server, cloud server, or server array. If the electronic device of this embodiment is implemented as a desktop computer, laptop computer, smartphone, or other terminal device, it may include the components within the dashed boxes in Figure 13; if the electronic device of this embodiment is implemented as a conventional server, cloud server, or server array, it may not include the components within the dashed boxes in Figure 13.
[0124] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0125] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.
[0126] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.
[0127] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.
[0128] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker configured to output audio signals.
[0129] Accordingly, this disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.
[0130] Accordingly, this disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above-described method embodiments. It should be understood that each step or combination of steps in the above-described method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.
[0131] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0132] The above are merely embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.
Claims
A motion generation model training method comprises: obtaining a first sample set, wherein each first sample in the first sample set comprises a positive sample pair and multiple different negative sample pairs, the positive sample pair comprises a sample text description and a corresponding standard motion video, and the negative sample pair comprises the sample text description and a corresponding non-standard motion video; training a video retrieval network in a motion generation model based on a contrast learning manner using the first sample set to obtain a trained video retrieval network; obtaining a second sample set, wherein each second sample in the second sample set comprises the sample text description, a corresponding predicted motion video and a standard motion sequence, wherein the predicted motion video is obtained by the trained video retrieval network from a motion video library based on the sample text description; training a motion generation network in the motion generation model using the second sample set to obtain a trained motion generation network. The method of claim 1, wherein, The video retrieval network comprises a predicate semantic extraction module, an argument semantic extraction module, a first key point detection module, a motion encoding module, a first video feature extraction module and a similarity calculation module. Correspondingly, the training of the video retrieval network in the motion generation model based on the contrast learning manner using the first sample set to obtain the trained video retrieval network comprises: for any first sample in the first sample set, extracting a first semantic feature vector of a predicate in the sample text description included in the first sample by the predicate semantic extraction module, and extracting a second semantic feature vector of an argument in the sample text description by the argument semantic extraction module; detecting key points of the motion video included in the first sample by the first key point detection module to obtain first key point information of a target object in the motion video included in the first sample, and encoding the first key point information by the motion encoding module to obtain a first motion feature vector; and extracting features of the motion video included in the first sample by the first video feature extraction module to obtain a first video feature vector; calculating a first similarity according to the first semantic feature vector and the first motion feature vector by the similarity calculation module, and calculating a second similarity according to the second semantic feature vector and the first video feature vector; and obtaining a final similarity between the motion video in the first sample and the sample text description according to the first similarity and the second similarity; taking the final similarity between the standard motion video in the first sample and the sample text description as the final similarity of the positive sample pair, and taking the final similarity between the non-standard motion video in the first sample and the sample text description as the final similarity of the negative sample pair; and determining a first contrast learning loss according to the final similarity of each positive sample pair and the final similarity of each negative sample pair, and adjusting network parameters of the video retrieval network according to the first contrast learning loss. The method of claim 2, wherein, Before extracting the first semantic feature vector of the predicate in the sample text description by the predicate semantic extraction module, the method further comprises: The third semantic feature vector of a predicate in the sample text description included in the first sample is extracted by the predicate semantic extraction module; Second key point information of a target object in the action video included in the first sample is obtained by performing key point detection on the action video included in the first sample by the first key point detection module, and a second action feature vector of the target object in the action video included in the first sample is obtained by performing feature coding on the second key point information by the action coding module; The similarity of the positive sample pair in the predicate dimension is determined according to the third semantic feature vector and the second action feature vector of the target object in the standard action video included in the first sample, and the similarity of the negative sample pair in the predicate dimension is determined according to the third semantic feature vector and the second action feature vector of the target object in the non-standard action video included in the first sample; A second contrastive learning loss is determined according to the similarity of each positive sample pair in the predicate dimension and the similarity of each negative sample pair in the predicate dimension, and the network parameters of the predicate semantic extraction module and the action coding module are adjusted according to the second contrastive learning loss. The method of claim 3, wherein, Further comprising: A fourth semantic feature vector of an argument in the sample text description included in the first sample is extracted by the argument semantic extraction module, and a second video feature vector of the action video included in the first sample is obtained by performing feature extraction on the action video included in the first sample by the first video feature extraction module; The similarity of the positive sample pair in the argument dimension is determined according to the fourth semantic feature vector and the second video feature vector of the standard action video included in the first sample, and the similarity of the negative sample pair in the argument dimension is determined according to the fourth semantic feature vector and the second video feature vector of the non-standard action video included in the first sample; A third contrastive learning loss is determined according to the similarity of each positive sample pair in the argument dimension and the similarity of each negative sample pair in the argument dimension, and the network parameters of the argument semantic extraction module and the first video feature extraction module are adjusted according to the third contrastive learning loss. The method of claim 1, wherein, The action generation network comprises a second video feature module, a second key point detection module, a first mapping module, a second mapping module and a language model. Accordingly, the action generation network in the action generation model is trained by using the second sample set to obtain a trained action generation network, comprising: For any second sample in the second sample set, a third video feature vector of the predicted action video in the second sample is extracted by the second video feature module, and the dimension of the third video feature vector is mapped by using the first mapping module. The third key point information of the target object in the predicted action video is obtained by performing key point detection on the predicted action video by the second key point detection module, and the dimension of the third key point information is mapped by using the second mapping module; inputting the sample text description in the second sample, the mapped third video feature vector and the mapped third key point information into the language model, and obtaining a predicted action sequence corresponding to the sample text description output by the language model; adjusting network parameters of the action generation network according to the standard action sequence in the second sample and the predicted action sequence. The method of claim 5, wherein, Before extracting the third video feature vector of the predicted action video in the second sample through the second video feature module, the method further includes: extracting a fourth video feature vector of the predicted action video through the second video feature module, and performing mapping processing on dimensions of the fourth video feature vector by using the first mapping module; performing key point detection on the predicted action video by using the second key point detection module to obtain fourth key point information of a target object in the predicted action video, and performing mapping processing on dimensions of the fourth key point information by using the second mapping module; inputting the mapped fourth video feature vector and the mapped fourth key point information into the language model, and obtaining a predicted text description output by the language model; adjusting network parameters of the first mapping module and the second mapping module according to the sample text description and the predicted text description. The method of claim 5, wherein, After adjusting network parameters of the action generation network according to the standard action sequence in the second sample and the predicted action sequence, the method further includes: inputting the sample text description and the predicted action video thereof into the action generation network for multiple times to obtain multiple candidate action sequences output by the action generation network for multiple times; scoring the candidate action sequences by using a reward model to obtain a scoring result of the candidate action sequences; and selecting a preferred predicted action sequence and a non-preferred predicted action sequence from the multiple candidate action sequences according to the scoring result; optimizing the language model according to the preferred predicted action sequence and the non-preferred predicted action sequence in a direct preference optimization manner. The method of claim 7, wherein, The scoring of the candidate action sequences by using the reward model to obtain a scoring result of the candidate action sequences includes: calculating first distance information between the candidate action sequence and the standard action sequence, and second distance information between the candidate action sequence and the sample text description by using the reward model, and scoring the candidate action sequence according to the first distance information and the second distance information to obtain the scoring result of the candidate action sequence. An action generation method includes: obtaining a text description, the text description being used to describe a target object performing a target action; inputting the text description into a video retrieval network in an action generation model to retrieve a target action video matching the text description from an action video library through the video retrieval network; inputting the text description and the target action video into an action generation network in the action generation model to output a target action sequence according to the target action video and the text description through the action generation network, wherein the target action sequence is used to control the target object to perform at least one target action. The method of claim 9, wherein, The video retrieval network comprises a predicate semantic extraction module, an argument semantic extraction module, a first key point detection module, an action encoding module, a first video feature extraction module and a similarity calculation module. Correspondingly, retrieving a target action video matching the text description from an action video library through the video retrieval network comprises: extracting a semantic feature vector of a predicate in the text description through the predicate semantic extraction module, and extracting a semantic feature vector of an argument in the text description through the argument semantic extraction module; obtaining a plurality of action videos from the action video library, and performing key point detection on each action video through the first key point detection module to obtain key point information of a target object in each action video, and performing feature encoding on the key point information of the target object in the action video through the action encoding module to obtain an action feature vector of the target object in each action video; and performing feature extraction on each action video through the first video feature extraction module to obtain a video feature vector of each action video; calculating, through the similarity calculation module, a similarity between each action video and the text description in a predicate dimension according to the semantic feature vector of the predicate and the action feature vector of the target object in each action video, and calculating a similarity between each action video and the text description in an argument dimension according to the semantic feature vector of the argument and the video feature vector of each action video; and obtaining a final similarity between each action video and the text description according to the similarity in the predicate dimension and the similarity in the argument dimension; selecting a target action video matching the text description from the plurality of action videos according to the final similarity between each action video and the text description. The method of claim 9, wherein, The action generation network comprises a second video feature module, a second key point detection module, a first mapping module, a second mapping module and a language model. Correspondingly, outputting a target action sequence according to the target action video and the text description through the action generation network comprises: extracting a target video feature vector of the target action video through the second video feature module, and performing mapping processing on the dimension of the target video feature vector through the first mapping module; performing key point detection on the target action video through the second key point detection module to obtain target key point information of a target object in the target action video, and performing mapping processing on the dimension of the target key point information through the second mapping module; inputting the text description, the mapped target video feature vector and the mapped target key point information into the language model, and obtaining a target action sequence corresponding to the text description output by the language model. A virtual object action control method comprises: obtaining a text description, the text description being used to describe a target action performed by a virtual object; inputting the text description into a video retrieval network in an action generation model to retrieve a target action video matching the text description from an action video library through the video retrieval network; input the text description and the target action video into an action generation network in the action generation model, so as to output a target action sequence according to the target action video and the text description through the action generation network; control the virtual object to perform at least one target action according to the target action sequence. A video generation method comprises: obtaining a text description, the text description being used to describe a target object performing a target action; inputting the text description into a video retrieval network in an action generation model, so as to retrieve a target action video matching the text description from an action video library through the video retrieval network; inputting the text description and the target action video into an action generation network in the action generation model, so as to output a target action sequence according to the target action video and the text description through the action generation network; generating a video in which the target object performs at least one target action according to the target action sequence. An electronic device, comprising: a memory and a processor; the memory is configured to store a computer program; the processor is coupled to the memory and is configured to execute the computer program to perform the following steps: obtain a first sample set, a first sample in the first sample set comprising one positive sample pair and a plurality of different negative sample pairs, the positive sample pair comprising a sample text description and a corresponding standard action video, and the negative sample pair comprising a sample text description and a corresponding non-standard action video; train a video retrieval network in an action generation model based on a contrast learning manner using the first sample set, to obtain a trained video retrieval network; obtain a second sample set, a second sample in the second sample set comprising the sample text description, a corresponding predicted action video and a standard action sequence, wherein the predicted action video is obtained by the trained video retrieval network based on the sample text description from an action video library; train an action generation network in the action generation model using the second sample set, to obtain a trained action generation network. The electronic device of claim 14, wherein the processor coupled to the memory is further configured to execute the computer program to perform the following steps: obtain a text description, the text description being used to describe a virtual object performing a target action; input the text description into a video retrieval network in an action generation model, so as to retrieve a target action video matching the text description from an action video library through the video retrieval network; input the text description and the target action video into an action generation network in the action generation model, so as to output a target action sequence according to the target action video and the text description through the action generation network; control the virtual object to perform at least a target action according to the target action sequence. The electronic device of claim 14, wherein the processor coupled to the memory is further configured execute the computer program to perform the following steps: obtain a text description, the text description is used to describe a target object performing a target action; input the text description into a video retrieval network in the action generation model to retrieve a target action video matching the text description from an action video library through the video retrieval network; input the text description and the target action video into an action generation network in the action generation model to output a target action sequence according to the target action video and the text description through the action generation network; generate a video according to the target action sequence, in which the target object performs at least one target action. A computer readable storage medium storing a computer program, when the computer program is executed by a processor, causes the processor to be able to implement the following steps: obtain a first sample set, a first sample in the first sample set comprising a positive sample pair and a plurality of different negative sample pairs, the positive sample pair comprising a sample text description and a corresponding standard action video, and the negative sample pair comprising the sample text description and a corresponding non-standard action video; train a video retrieval network in an action generation model based on a contrast learning manner using the first sample set to obtain a trained video retrieval network; obtain a second sample set, a second sample in the second sample set comprising the sample text description, a corresponding predicted action video and a standard action sequence, wherein, the predicted action video is retrieved from an action video library based on the sample text description by the trained video retrieval network; train an action generation network in the action generation model using the second sample set to obtain a trained action generation network. The computer readable storage medium according to claim 17, when the computer program is executed by a processor, causes the processor to be further able to implement the following steps: obtain a text description, the text description being used to describe a target object performing a target action; input the text description into a video retrieval network in the action generation model to retrieve a target action video matching the text description from an action video library through the video retrieval network; inputting the text description and the target action video into an action generation network in the action generation model to output a target action sequence according to the target action video and the text description through the action generation network, wherein, the target action sequence is used to control the target object to perform at least one target action. The computer readable storage medium according to claim 17, when the computer program is executed by a processor, causes the processor to be further able to implement the following steps: obtain a text description, the text description being used to describe a virtual object performing a target action; input the text description into a video retrieval network in the action generation model to retrieve a target action video matching the text description from an action video library through the video retrieval network; input the text description and the target action video into an action generation network in the action generation model to output a target action sequence according to the target action video and the text description through the action generation network; control the virtual object to perform at least one target action according to the target action sequence. The computer readable storage medium according to claim 17, when the computer program is executed by the processor, causes the processor to be further able to implement the following steps: obtain a text, the text being used to describe a target object performing a target action; inputting the text description into a video retrieval network in the action generation model to retrieve a target action video matching the text description from an action video library through the video retrieval network; inputting the text description and the target action video into an action generation network in the action generation model to output a target action sequence according to the target action video and the text description through the action generation network; generating a video in which the target object performs at least one target action according to the target action sequence.