Face animation driving method and apparatus, and readable storage medium and electronic device
Patent Information
- Application Number
- PCT/CN2025/147814
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-27
- Filing Date
- 2025-12-31
- Publication Date
- 2026-09-03
Smart Images
Figure CN2025147814_03092026_PF_FP_ABST
Abstract
Description
A facial motion driving method, apparatus, readable storage medium, and electronic device.
[0001] This application claims priority to Chinese Patent Application No. 202510229313.2, filed on February 27, 2025, entitled "A Facial Movement Driving Method, Apparatus, Readable Storage Medium and Electronic Device", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application belongs to the field of facial motion driving technology, and in particular relates to a facial motion driving method, device, computer-readable storage medium and electronic device. Background Technology
[0003] In the existing technology, there are relatively mature methods for driving facial movements through sound. However, these existing methods mainly drive facial movements based on the content of the sound. Although the facial movements are consistent with the content of the sound, they still appear monotonous and stiff compared to real human facial movements, and lack realism. Summary of the Invention
[0004] In view of this, embodiments of this application provide a facial motion driving method, apparatus, computer-readable storage medium, and electronic device to solve the problem of poor realism in existing facial motion driving methods.
[0005] A first aspect of this application provides a facial motion driving method, which may include:
[0006] Acquire the target sound used to drive facial movements;
[0007] The target sound is subjected to emotional feature extraction processing to obtain the emotional features corresponding to the target sound;
[0008] Based on the emotional characteristics, the target voice is processed using a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice;
[0009] Control the preset virtual digital human to perform the three-dimensional facial movements.
[0010] In one specific implementation of the first aspect, the step of processing the target voice based on the emotional features using a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice may include:
[0011] The target sound is subjected to sound feature extraction processing to obtain the sound features corresponding to the target sound;
[0012] The previous model output is processed by action encoding to obtain the first processed data;
[0013] The first processed data is subjected to periodic position encoding to obtain the second processed data;
[0014] The second processed data is subjected to biased causal multi-head self-attention processing to obtain the third processed data;
[0015] The sound features and the third processed data are subjected to biased cross-modal multi-head self-attention processing to obtain the fourth processed data.
[0016] The emotional features and the fourth processed data are fused together to obtain the fifth processed data;
[0017] The fifth processed data is subjected to forward feedback processing to obtain the sixth processed data;
[0018] The sixth processed data is subjected to motion decoding processing to obtain the three-dimensional facial motion corresponding to the target sound.
[0019] In one specific implementation of the first aspect, before performing sound feature extraction processing on the target sound, the following may be included:
[0020] Obtain a sound feature extraction model corresponding to the first language;
[0021] The sound feature extraction model is fine-tuned using a second language corpus to obtain the fine-tuned sound feature extraction model.
[0022] Accordingly, the step of performing sound feature extraction processing on the target sound to obtain sound features corresponding to the target sound may include:
[0023] The target sound is processed by extracting sound features using the finely tuned sound feature extraction model to obtain the sound features corresponding to the target sound.
[0024] In one specific implementation of the first aspect, before processing the target sound using a preset three-dimensional facial motion model, the method may further include:
[0025] Determine a first loss function for the three-dimensional facial motion model; wherein the first loss function is a loss function that applies truth constraints to the three-dimensional facial motion model;
[0026] Determine a second loss function for the three-dimensional facial motion model; wherein the second loss function is a loss function for applying inter-frame constraints to the three-dimensional facial motion model;
[0027] The first loss function and the second loss function are fused to obtain the fusion loss function of the three-dimensional facial motion model;
[0028] The initial 3D facial motion model is trained based on the fusion loss function to obtain the trained 3D facial motion model.
[0029] In one specific implementation of the first aspect, determining the first loss function of the three-dimensional facial motion model may include:
[0030] Determine the output error of the three-dimensional facial motion model; wherein, the output error is the absolute value of the difference between the expected output and the actual output;
[0031] The airfoil function of the output error is determined as the first loss function of the three-dimensional facial motion model.
[0032] In one specific implementation of the first aspect, the step of determining the second loss function of the three-dimensional facial motion model may include:
[0033] Determine the first inter-frame difference of the three-dimensional facial motion model; wherein, the first inter-frame difference is the difference between two consecutive expected outputs;
[0034] Determine the second inter-frame difference of the three-dimensional facial motion model; wherein, the second inter-frame difference is the difference between two consecutive actual outputs;
[0035] The second loss function of the three-dimensional facial motion model is determined based on the difference between the first inter-frame difference and the second inter-frame difference.
[0036] In one specific implementation of the first aspect, before performing emotional feature extraction processing on the target sound, the following may be included:
[0037] Obtain a sentiment feature extraction model corresponding to the first language;
[0038] The sentiment feature extraction model is adjusted by low-rank adaptation using a second language corpus to obtain the sentiment feature extraction model after low-rank adaptation.
[0039] Accordingly, the step of performing emotional feature extraction processing on the target sound to obtain emotional features corresponding to the target sound may include:
[0040] The emotional feature extraction model, adjusted by low-rank adaptation, is used to extract emotional features from the target sound, thereby obtaining the emotional features corresponding to the target sound.
[0041] A second aspect of this application provides a facial motion driving device, which may include:
[0042] The sound acquisition module is used to acquire the target sound used to drive facial movements;
[0043] The emotion feature extraction module is used to perform emotion feature extraction processing on the target sound to obtain the emotion features corresponding to the target sound;
[0044] The facial motion model processing module is used to process the target voice based on the emotional features using a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice.
[0045] The facial motion execution control module is used to control a preset virtual digital human to perform the three-dimensional facial motions.
[0046] In one specific implementation of the second aspect, the facial motion model processing module may include:
[0047] The sound feature extraction unit is used to perform sound feature extraction processing on the target sound to obtain sound features corresponding to the target sound;
[0048] An action coding unit is used to perform action coding processing on the previous model output to obtain the first processed data;
[0049] A position encoding unit is used to perform periodic position encoding processing on the first processed data to obtain the second processed data;
[0050] The first attention processing unit is used to perform biased causal multi-head self-attention processing on the second processed data to obtain the third processed data.
[0051] The second attention processing unit is used to perform biased cross-modal multi-head self-attention processing on the sound features and the third processed data to obtain the fourth processed data.
[0052] The data fusion unit is used to fuse the emotional features and the fourth processed data to obtain the fifth processed data;
[0053] A forward feedback unit is used to perform forward feedback processing on the fifth processed data to obtain the sixth processed data;
[0054] The motion decoding unit is used to perform motion decoding processing on the sixth processing data to obtain the three-dimensional facial motion corresponding to the target sound.
[0055] In one specific implementation of the second aspect, the facial motion driving device may further include:
[0056] The model fine-tuning module is used to obtain a sound feature extraction model corresponding to the first language; and to fine-tune the sound feature extraction model using corpus of the second language to obtain the fine-tuned sound feature extraction model.
[0057] Accordingly, the sound feature extraction unit can be specifically used to: perform sound feature extraction processing on the target sound using the finely tuned sound feature extraction model to obtain the sound features corresponding to the target sound.
[0058] In one specific implementation of the second aspect, the facial motion driving device may further include:
[0059] The first loss function determination module is used to determine the first loss function of the three-dimensional facial motion model; wherein, the first loss function is a loss function that applies truth constraints to the three-dimensional facial motion model;
[0060] The second loss function determination module is used to determine the second loss function of the three-dimensional facial motion model; wherein, the second loss function is a loss function for applying inter-frame constraints to the three-dimensional facial motion model;
[0061] The fusion loss function determination module is used to fuse the first loss function and the second loss function to obtain the fusion loss function of the three-dimensional facial motion model;
[0062] The model training module is used to train the initial three-dimensional facial motion model based on the fusion loss function to obtain the trained three-dimensional facial motion model.
[0063] In one specific implementation of the second aspect, the first loss function determination module may be specifically used to: determine the output error of the three-dimensional facial motion model; wherein the output error is the absolute value of the difference between the expected output and the actual output; and determine the airfoil function of the output error as the first loss function of the three-dimensional facial motion model.
[0064] In one specific implementation of the second aspect, the second loss function determination module may be specifically used to: determine the first inter-frame difference of the three-dimensional facial motion model; wherein the first inter-frame difference is the difference between two consecutive expected outputs; determine the second inter-frame difference of the three-dimensional facial motion model; wherein the second inter-frame difference is the difference between two consecutive actual outputs; and determine the second loss function of the three-dimensional facial motion model based on the difference between the first inter-frame difference and the second inter-frame difference.
[0065] In one specific implementation of the second aspect, the facial motion driving device may further include:
[0066] A low-rank adaptation adjustment module is used to obtain an emotion feature extraction model corresponding to the first language; and to perform low-rank adaptation adjustment on the emotion feature extraction model using corpus of the second language to obtain the low-rank adapted emotion feature extraction model.
[0067] Accordingly, the emotion feature extraction module can be specifically used to: perform emotion feature extraction processing on the target sound through the emotion feature extraction model adjusted by low-rank adaptation, so as to obtain the emotion feature corresponding to the target sound.
[0068] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described facial motion driving methods.
[0069] A fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described facial motion driving methods.
[0070] The fifth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the steps of any of the above-described facial motion driving methods.
[0071] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment obtains a target voice for driving facial movements; performs emotional feature extraction processing on the target voice to obtain emotional features corresponding to the target voice; based on the emotional features, processes the target voice through a preset three-dimensional facial movement model to obtain three-dimensional facial movements corresponding to the target voice; and controls a preset virtual digital human to perform the three-dimensional facial movements. Through this application embodiment, emotional features in the voice can be extracted, and during the facial movement driving process, corresponding three-dimensional facial movements can be generated based on the emotional features, making the facial movements as emotional as real human facial movements, and more realistic and natural. Attached Figure Description
[0072] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0073] Figure 1 is a flowchart of an embodiment of a facial motion driving method according to this application;
[0074] Figure 2 is a schematic diagram of human 3D facial movements based on hybrid deformation expression;
[0075] Figure 3 is a schematic diagram of the structure of the three-dimensional facial motion model;
[0076] Figure 4 is a schematic flowchart of determining the loss function and training the model;
[0077] Figure 5 is a structural diagram of an embodiment of a facial motion driving device according to this application;
[0078] Figure 6 is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0079] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0080] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0081] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0082] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0083] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."
[0084] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0085] In the existing technology, there are relatively mature methods for driving facial movements through sound. However, these existing methods mainly drive facial movements based on the content of the sound. Although the facial movements are consistent with the content of the sound, they still appear monotonous and stiff compared to real human facial movements, and lack realism.
[0086] In view of this, embodiments of this application provide a facial motion driving method, apparatus, computer-readable storage medium, and electronic device to solve the problem of poor realism in existing facial motion driving methods.
[0087] In the embodiments of this application, emotional features in the voice can be extracted, and corresponding three-dimensional facial movements can be generated based on the emotional features during the facial movement driving process, so that the facial movements can be as emotional as real human facial movements, making them more realistic and natural.
[0088] The execution subject of this application embodiment can be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptops, handheld computers, robots, and servers.
[0089] Please refer to Figure 1. One embodiment of a facial motion driving method in this application may include:
[0090] Step S101: Obtain the target sound used to drive facial movements.
[0091] The target sound is the sound that the electronic device will play. During the playback of the target sound, the pre-set virtual digital human needs to make corresponding facial movements in sync, so that its mouth shape and expression match the target sound, thereby producing the effect of the virtual digital human speaking like a human.
[0092] Step S102: Extract emotional features from the target sound to obtain the emotional features corresponding to the target sound.
[0093] In the embodiments of the present application, any sentiment feature extraction model in the prior art can be used to perform sentiment feature extraction processing, which may include but is not limited to the BERT model and other models, which are not specifically limited in the embodiments of the present application.
[0094] Considering that although the BERT model can be used as a sentiment feature extraction model after training, it requires a large amount of data to achieve a relatively good generalization effect, so the generalization performance of the model trained on public datasets is poor. In addition to extracting English content, the English wav2vec model can also extract certain sentiment features. Based on this phenomenon, a sentiment feature extraction model with strong generalization can be trained through full fine-tuning based on the English wav2vec model, which is denoted as wav2vec_emo_english.
[0095] In a specific implementation of the embodiments of the present application, sentiment feature extraction processing can be directly performed on a target sound through wav2vec_emo_english, so as to obtain the sentiment feature corresponding to the target sound.
[0096] In another specific implementation of the embodiments of the present application, considering that although wav2vec_emo_english has good sentiment generalization for other non-English languages (such as Chinese), its spatial aggregation is not absolutely ideal, so feature compensation can be further introduced to fix the deficiency of spatial aggregation. In the case where the language (denoted as the first language) corresponding to the existing sentiment feature extraction model (such as wav2vec_emo_english) is inconsistent with the language (denoted as the second language) corresponding to the target sound, the sentiment feature extraction model corresponding to the first language can be obtained, and the second language corpus is used to perform Low-Rank Adaptation (LoRA) adjustment on the sentiment feature extraction model, so as to obtain the sentiment feature extraction model after LoRA adjustment.
[0097] LoRA adjustment simulates the parameter change amount through low-rank decomposition, so as to realize indirect training of a large model with extremely small parameter amount. Through the multiplication of the two preceding and following matrices A and B, the first matrix A is responsible for dimensionality reduction, the second matrix B is responsible for dimensionality increase, the middle layer dimension is r, the dimension of the trainable layer and the dimension of the pre-trained model layer are both d, the dimension d is first reduced to r through the fully connected layer, and then mapped back to the d dimension from r through the fully connected layer, where r << d, r is the rank of the matrix, so the matrix calculation changes from d*d to d*r + r*d, and the parameter amount is greatly reduced.
[0098] For existing sentiment feature extraction models, LoRA adjustments can be made to the query vector (Q), key vector (K), and value vector (V) involved in their self-attention mechanism. Taking Q as an example, the left branch is the original frozen network, and the right branch introduces W_A and W_B parameters for learning. The fully connected layer parameters of W_A are (d, r), and the fully connected layer parameters of W_B are (r, d). The parameters of the right branch are learnable and are denoted as Q. 右 Then we can obtain the updated query vector (denoted as Q'): Q' = Q + Q 右 The adjustment process for K and V is similar to that for Q, resulting in the updated key vector (denoted as K') and value vector (denoted as V'), which will not be elaborated further here. In application, Q', K', and V' are used to replace the original Q, K, and V.
[0099] When performing emotional feature extraction on target audio, the LoRA-adjusted emotional feature extraction model can be used to extract emotional features from the target audio, thereby obtaining the emotional features corresponding to the target audio.
[0100] Step S103: Based on emotional features, the target voice is processed through a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice.
[0101] Among these, 3D facial animation can be expressed as human 3D facial animation based on BlendShape (BS). The richness and diversity of human facial movements stem from the displacement of certain points on the face in space. For example, when some points at the corners of the mouth move diagonally upwards, visually, it creates the appearance of a smiling face. Based on this principle, a human facial animation system can be designed by extracting and creating basic facial movements. These basic facial movements can be viewed as facial deformations in different emotional states. With these basic facial movements, various composite facial movements can be created by adjusting their weight coefficients. The magnitude of the weight coefficients determines the contribution of each basic facial movement to the composite facial movement.
[0102] Figure 2 shows a schematic diagram of human 3D facial movements expressed based on BlendShape. The number of basic facial movements is denoted as N, where N is a positive integer. Its specific value can be flexibly set according to actual conditions; for example, it can be set to 52 or other values. This embodiment does not specifically limit this. By linearly weighting these basic facial movements under different combinations of weighting coefficients, various composite facial movements can be obtained.
[0103] While existing technologies already contain numerous models for generating facial animations based on sound, these models primarily rely on vertex representations or 3DMMs for their 3D facial animation representation. In this embodiment, BlendShape can be used to represent 3D facial animations, resulting in the 3D facial animation model described in this embodiment.
[0104] A 3D facial motion model can employ an autoregressive approach, which initializes the model with a closed mouth and continuously predicts the current state based on previous predictions and current speech features. As an example, Figure 3 illustrates a possible model structure for a 3D facial motion model. As shown, a 3D facial motion model may include, but is not limited to, a voice feature extraction module, a BS Encoder, a Periodic Positional Encoding module, a Biased Causal Multi-Head Self-Attention module, a Biased Cross-Modal Multi-Head Self-Attention module, a Biased Cross-Emo Multi-Head Self-Attention module, a Feed Forward module, and a BS Decoder module.
[0105] Compared to existing technologies, the core difference in this model structure lies in the addition of a biased cross-modal multi-head self-attention module. This module can fuse sentiment features with the output of the biased cross-modal multi-head self-attention module through a self-attention mechanism, as shown in the following equation:
[0106] in, K is the Q-value corresponding to the output of the biased cross-modal multi-head self-attention module. E and V E For the K and V values corresponding to the emotional characteristics, B A For alignment bias, Att is the self-attention mechanism processing function, and d k For K E The number of dimensions.
[0107] Based on the model structure shown in Figure 3, the process of processing the target sound using a 3D facial motion model can include: extracting sound features from the target sound to obtain sound features corresponding to the target sound; performing motion encoding on the previous model output to obtain first processed data; performing periodic position encoding on the first processed data to obtain second processed data; performing biased causal multi-head self-attention processing on the second processed data to obtain third processed data; performing biased cross-modal multi-head self-attention processing on the sound features and the third processed data to obtain fourth processed data; fusing the emotional features and the fourth processed data to obtain fifth processed data; performing forward feedback processing on the fifth processed data to obtain sixth processed data; and performing motion decoding on the sixth processed data to obtain the 3D facial motion corresponding to the target sound. The specific processing steps can refer to existing models for generating corresponding facial motions based on sound, and will not be elaborated further in this embodiment.
[0108] It should be noted that in the embodiments of this application, the input of the action encoding module (i.e., the previous facial action) and the output of the action decoding module (i.e., the currently predicted facial action) are both human 3D facial actions expressed based on BlendShape. In the human 3D facial actions expressed based on BlendShape, since the weight coefficients of the basic facial actions are all positive, a sigmoid activation function can be added after the action decoding process to ensure that the weight coefficients are all positive.
[0109] In the sound feature extraction module, any sound feature extraction model in the prior art can be used to perform sound feature extraction processing, including but not limited to the wav2vec model and other models. This application embodiment does not make specific limitations on this.
[0110] If the language (first language) corresponding to the existing sound feature extraction model is the same as the language (second language) corresponding to the target sound, the sound feature extraction process of the target sound can be directly performed through the existing sound feature extraction model to obtain the sound features corresponding to the target sound.
[0111] When the first and second languages are different, an existing sound feature extraction model can be fine-tuned using corpus data from the second language. This fine-tuned model improves its adaptability to the second language. For example, if the existing sound feature extraction model corresponds to English, and the target sound corresponds to Chinese, then Chinese corpus data can be used to fine-tune the existing model, resulting in a more refined version that better suits Chinese. During sound feature extraction, the finely tuned model can then be used to extract sound features from the target sound, yielding the corresponding sound features.
[0112] The three-dimensional facial motion model can be pre-trained using a corresponding dataset. The loss function used during training can be flexibly set according to the actual situation, and this application does not impose specific limitations on it.
[0113] In one specific implementation of this application, the loss function can be determined and the model trained according to the process shown in Figure 4:
[0114] Step S401: Determine the first loss function for the 3D facial motion model.
[0115] Here, the first loss function is the loss function used to constrain the ground truth of the 3D facial motion model. Specifically, the output error of the 3D facial motion model can be determined, and the airfoil function of the output error can be used as the first loss function of the 3D facial motion model, as shown in the following equation:
[0116] Where y is the expected output of the 3D facial motion model, i.e., the ground truth. This is the actual output of the 3D facial motion model. The output error is the absolute value of the difference between the expected output and the actual output. WingLoss is the airfoil function. WingLoss responds much better to subtle differences than other functions, and can optimize the small errors of facial movement changes more finely. L1 is the first loss function of the 3D facial movement model.
[0117] Step S402: Determine the second loss function for the three-dimensional facial motion model.
[0118] The second loss function is used to apply inter-frame constraints to the 3D facial motion model. Specifically, the first inter-frame difference and the second inter-frame difference of the 3D facial motion model can be determined separately, and the second loss function of the 3D facial motion model can be determined based on the difference between the first inter-frame difference and the second inter-frame difference, as shown in the following formula:
[0119] Among them, (y i -y i-1 The first frame difference is the difference between the expected outputs of two consecutive frames. This is the second frame difference, which is the difference between the actual output of two consecutive frames. L1 represents the difference between the first and second frame differences, and L2 is the second loss function for the 3D facial motion model.
[0120] Step S403: Fuse the first loss function and the second loss function to obtain the fusion loss function of the three-dimensional facial motion model.
[0121] The specific fusion method can be flexibly set according to the actual situation, and this application embodiment does not impose specific limitations on it. As an example, for the sake of simplicity, the sum of the first loss function and the second loss function can be directly used as the fusion loss function of the 3D facial motion model.
[0122] Step S404: Train the initial 3D facial motion model based on the fusion loss function to obtain the trained 3D facial motion model.
[0123] In this embodiment, the model can be trained using a sufficient number of training samples. Each training sample includes a set of sound samples and corresponding 3D facial motion labels. By using the sound samples of the training samples as input and the corresponding 3D facial motion labels as the expected output, the 3D facial motion model can be trained to obtain the trained 3D facial motion model.
[0124] During training, for each training sample, the 3D facial motion model can process the audio sample of that training sample to obtain the actual output of that training sample. Then, a fusion loss function can be used to calculate the training loss value based on the expected output and the actual output of that training sample. After calculating the training loss value, the model parameters of the 3D facial motion model can be adjusted according to the training loss value.
[0125] In this embodiment, assuming the initial model parameters of the 3D facial motion model are W1, the training loss value is backpropagated to modify the model parameters W1, resulting in modified model parameters W2. After modifying the parameters, the next training process is executed. In this training process, the training loss value is recalculated, and this training loss value is backpropagated to modify the model parameters W2, resulting in modified model parameters W3, and so on. This process is repeated continuously, and the model parameters can be modified in each training process until a preset training condition is met. The training condition can be that the number of training iterations reaches a preset threshold, which can be set according to the actual situation, for example, it can be set to thousands, tens of thousands, hundreds of thousands, or even larger values. The training condition can also be that the 3D facial motion model converges. Since it is possible that the number of training iterations has not reached the threshold, but the 3D facial motion model has converged, it may lead to unnecessary duplication of work; or the 3D facial motion model may never converge, which may lead to an infinite loop and the training process cannot be terminated. Based on the above two situations, the training condition can also be that the number of training iterations reaches the threshold or the 3D facial motion model converges. Once the training conditions are met, a trained 3D facial motion model can be obtained.
[0126] Through the above process, the sound samples of the training samples and the corresponding three-dimensional facial action labels are used as the learning objects of the three-dimensional facial action model. After the training process, the three-dimensional facial action model can establish a mapping relationship between the sound samples and the three-dimensional facial action labels. Thus, when faced with a new sound, the corresponding three-dimensional facial action can also be obtained according to this mapping relationship.
[0127] After training the 3D facial motion model, it can be used to process the target sound, thereby obtaining the 3D facial motion corresponding to the target sound.
[0128] Step S104: Control the preset virtual digital human to perform three-dimensional facial movements.
[0129] A digital human (or meta human) is a digitally created avatar that closely resembles a human. After obtaining the 3D facial movements corresponding to a target voice, the virtual digital human can be controlled to generate corresponding blendshape expressions, allowing it to display 3D facial movements that are as close as possible to real human facial movements.
[0130] In summary, this application embodiment obtains a target voice for driving facial movements; extracts emotional features from the target voice to obtain emotional features corresponding to the target voice; based on the emotional features, processes the target voice using a preset 3D facial movement model to obtain 3D facial movements corresponding to the target voice; and controls a preset virtual digital human to perform the 3D facial movements. Through this application embodiment, emotional features in the voice can be extracted, and during the facial movement driving process, corresponding 3D facial movements can be generated based on these emotional features, making the facial movements as emotional as real human facial movements, and more realistic and natural.
[0131] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0132] Corresponding to the facial motion driving method described in the above embodiments, Figure 5 shows a structural diagram of an embodiment of a facial motion driving device provided in this application.
[0133] In this embodiment, a facial motion driving device may include:
[0134] The sound acquisition module 501 is used to acquire the target sound used to drive facial movements;
[0135] The emotion feature extraction module 502 is used to perform emotion feature extraction processing on the target sound to obtain the emotion features corresponding to the target sound;
[0136] The facial motion model processing module 503 is used to process the target voice based on the emotional features using a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice.
[0137] The facial motion execution control module 504 is used to control a preset virtual digital human to perform the three-dimensional facial motions.
[0138] In one specific implementation of this application embodiment, the facial motion model processing module may include:
[0139] The sound feature extraction unit is used to perform sound feature extraction processing on the target sound to obtain sound features corresponding to the target sound;
[0140] An action coding unit is used to perform action coding processing on the previous model output to obtain the first processed data;
[0141] A position encoding unit is used to perform periodic position encoding processing on the first processed data to obtain the second processed data;
[0142] The first attention processing unit is used to perform biased causal multi-head self-attention processing on the second processed data to obtain the third processed data.
[0143] The second attention processing unit is used to perform biased cross-modal multi-head self-attention processing on the sound features and the third processed data to obtain the fourth processed data.
[0144] The data fusion unit is used to fuse the emotional features and the fourth processed data to obtain the fifth processed data;
[0145] A forward feedback unit is used to perform forward feedback processing on the fifth processed data to obtain the sixth processed data;
[0146] The motion decoding unit is used to perform motion decoding processing on the sixth processing data to obtain the three-dimensional facial motion corresponding to the target sound.
[0147] In one specific implementation of this application embodiment, the facial motion driving device may further include:
[0148] The model fine-tuning module is used to obtain a sound feature extraction model corresponding to the first language; and to fine-tune the sound feature extraction model using corpus of the second language to obtain the fine-tuned sound feature extraction model.
[0149] Accordingly, the sound feature extraction unit can be specifically used to: perform sound feature extraction processing on the target sound using the finely tuned sound feature extraction model to obtain the sound features corresponding to the target sound.
[0150] In one specific implementation of this application embodiment, the facial motion driving device may further include:
[0151] The first loss function determination module is used to determine the first loss function of the three-dimensional facial motion model; wherein, the first loss function is a loss function that applies truth constraints to the three-dimensional facial motion model;
[0152] The second loss function determination module is used to determine the second loss function of the three-dimensional facial motion model; wherein, the second loss function is a loss function for applying inter-frame constraints to the three-dimensional facial motion model;
[0153] The fusion loss function determination module is used to fuse the first loss function and the second loss function to obtain the fusion loss function of the three-dimensional facial motion model;
[0154] The model training module is used to train the initial three-dimensional facial motion model based on the fusion loss function to obtain the trained three-dimensional facial motion model.
[0155] In one specific implementation of this application embodiment, the first loss function determination module can be specifically used to: determine the output error of the three-dimensional facial motion model; wherein the output error is the absolute value of the difference between the expected output and the actual output; and determine the airfoil function of the output error as the first loss function of the three-dimensional facial motion model.
[0156] In one specific implementation of this application embodiment, the second loss function determination module may be specifically used to: determine the first inter-frame difference of the three-dimensional facial motion model; wherein the first inter-frame difference is the difference between two consecutive expected outputs; determine the second inter-frame difference of the three-dimensional facial motion model; wherein the second inter-frame difference is the difference between two consecutive actual outputs; and determine the second loss function of the three-dimensional facial motion model based on the difference between the first inter-frame difference and the second inter-frame difference.
[0157] In one specific implementation of this application embodiment, the facial motion driving device may further include:
[0158] A low-rank adaptation adjustment module is used to obtain an emotion feature extraction model corresponding to the first language; and to perform low-rank adaptation adjustment on the emotion feature extraction model using corpus of the second language to obtain the low-rank adapted emotion feature extraction model.
[0159] Accordingly, the emotion feature extraction module can be specifically used to: perform emotion feature extraction processing on the target sound through the emotion feature extraction model adjusted by low-rank adaptation, so as to obtain the emotion feature corresponding to the target sound.
[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0162] Figure 6 shows a schematic block diagram of an electronic device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.
[0163] As shown in FIG6, the electronic device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the above-described facial motion driving method embodiments, such as steps S101 to S104 shown in FIG1. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, such as the functions of modules 501 to 504 shown in FIG5.
[0164] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the electronic device 6.
[0165] The electronic device 6 may include, but is not limited to, computing devices such as mobile phones, tablets, desktop computers, laptops, handheld computers, robots, and servers. Those skilled in the art will understand that Figure 6 is merely an example of the electronic device 6 and does not constitute a limitation on the electronic device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device 6 may also include input / output devices, network access devices, buses, etc.
[0166] The processor 60 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.
[0167] The memory 61 can be an internal storage unit of the electronic device 6, such as a hard disk or memory. The memory 61 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 61 can include both internal and external storage units of the electronic device 6. The memory 61 is used to store the computer program and other programs and data required by the electronic device 6. The memory 61 can also be used to temporarily store data that has been output or will be output.
[0168] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0169] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0170] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0171] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0174] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.
[0175] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A facial motion driving method, characterized in that, include: Acquire the target sound used to drive facial movements; The target sound is subjected to emotional feature extraction processing to obtain the emotional features corresponding to the target sound; Based on the emotional characteristics, the target voice is processed using a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice; Control the preset virtual digital human to perform the three-dimensional facial movements.
2. The facial motion driving method according to claim 1, characterized in that, The process of processing the target voice based on the emotional features using a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice includes: The target sound is subjected to sound feature extraction processing to obtain the sound features corresponding to the target sound; The previous model output is processed by action encoding to obtain the first processed data; The first processed data is subjected to periodic position encoding to obtain the second processed data; The second processed data is subjected to biased causal multi-head self-attention processing to obtain the third processed data; The sound features and the third processed data are subjected to biased cross-modal multi-head self-attention processing to obtain the fourth processed data. The emotional features and the fourth processed data are fused together to obtain the fifth processed data; The fifth processed data is subjected to forward feedback processing to obtain the sixth processed data; The sixth processed data is subjected to motion decoding processing to obtain the three-dimensional facial motion corresponding to the target sound.
3. The facial motion driving method according to claim 2, characterized in that, Before performing sound feature extraction processing on the target sound, the process also includes: Obtain a sound feature extraction model corresponding to the first language; The sound feature extraction model is fine-tuned using a second language corpus to obtain the fine-tuned sound feature extraction model. Accordingly, the step of performing sound feature extraction processing on the target sound to obtain sound features corresponding to the target sound includes: The target sound is processed by extracting sound features using the finely tuned sound feature extraction model to obtain the sound features corresponding to the target sound.
4. The facial motion driving method according to claim 1, characterized in that, Before processing the target sound using a preset 3D facial motion model, the process also includes: Determine a first loss function for the three-dimensional facial motion model; wherein the first loss function is a loss function that applies truth constraints to the three-dimensional facial motion model; Determine a second loss function for the three-dimensional facial motion model; wherein the second loss function is a loss function for applying inter-frame constraints to the three-dimensional facial motion model; The first loss function and the second loss function are fused to obtain the fusion loss function of the three-dimensional facial motion model; The initial 3D facial motion model is trained based on the fusion loss function to obtain the trained 3D facial motion model.
5. The facial motion driving method according to claim 4, characterized in that, The determination of the first loss function for the three-dimensional facial motion model includes: Determine the output error of the three-dimensional facial motion model; wherein, the output error is the absolute value of the difference between the expected output and the actual output; The airfoil function of the output error is determined as the first loss function of the three-dimensional facial motion model.
6. The facial motion driving method according to claim 4, characterized in that, The determination of the second loss function for the three-dimensional facial motion model includes: Determine the first inter-frame difference of the three-dimensional facial motion model; wherein, the first inter-frame difference is the difference between two consecutive expected outputs; Determine the second inter-frame difference of the three-dimensional facial motion model; wherein, the second inter-frame difference is the difference between two consecutive actual outputs; The second loss function of the three-dimensional facial motion model is determined based on the difference between the first inter-frame difference and the second inter-frame difference.
7. The facial motion driving method according to any one of claims 1 to 6, characterized in that, Before performing emotional feature extraction processing on the target sound, the process also includes: Obtain a sentiment feature extraction model corresponding to the first language; The sentiment feature extraction model is adjusted by low-rank adaptation using a second language corpus to obtain the sentiment feature extraction model after low-rank adaptation. Accordingly, the step of performing emotional feature extraction processing on the target sound to obtain emotional features corresponding to the target sound includes: The emotional feature extraction model, adjusted by low-rank adaptation, is used to extract emotional features from the target sound, thereby obtaining the emotional features corresponding to the target sound.
8. A facial motion driving device, characterized in that, include: The sound acquisition module is used to acquire the target sound used to drive facial movements; The emotion feature extraction module is used to perform emotion feature extraction processing on the target sound to obtain the emotion features corresponding to the target sound; The facial motion model processing module is used to process the target voice based on the emotional features using a preset three-dimensional facial motion model to obtain a three-dimensional facial motion corresponding to the target voice. The facial motion execution control module is used to control a preset virtual digital human to perform the three-dimensional facial motions.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the facial motion driving method as described in any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the facial motion driving method as described in any one of claims 1 to 7.