Audio-driven facial reconstruction method and related device

Abstract

An audio-driven facial reconstruction method and facial reconstruction apparatus, an electronic device, a storage medium and a program product. The facial reconstruction method is executed by an electronic device, and comprises: acquiring input audio (201); on the basis of the input audio and an input facial image of a target object, obtaining a target facial contour of the target object driven by the input audio, the target facial contour comprising a contour related to a mouth shape (202); on the basis of the target facial contour of the target object, a reference facial image of the target object and a reference facial contour of the reference facial image, determining optical flow information, the optical flow information being used for representing offset information between a reconstructed facial image and the reference facial image of the target object (203); and, on the basis of the reference facial image of the target object and the optical flow information, performing facial feature reconstruction on the input facial image, so as to obtain a reconstructed facial image (204).

Description

Audio-driven face reconstruction methods and related equipment

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on January 24, 2025, with application number 202510122004.5, entitled "Audio-Driven Face Reconstruction Method and Related Device", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and more specifically, to an audio-driven face reconstruction method and related equipment. Background Technology

[0004] In audio-driven scenarios, it's necessary to use audio to drive the lip movements of characters in existing videos, ensuring that the resulting driven video displays the lip movements of the characters as they pronounce the audio. However, related technologies suffer from issues with unsatisfactory lip-syncing effects after audio-driven processing. Summary of the Invention

[0005] In view of the above problems, this application proposes an audio-driven face reconstruction method and related device to improve the above problems.

[0006] According to one aspect of the embodiments of this application, an audio-driven face reconstruction method is provided, comprising: acquiring input audio; obtaining a target face contour of the target object under the driving action of the input audio, the target face contour including a lip-related contour, based on the input audio and an input face image of the target object; determining optical flow information based on the target face contour of the target object, a reference face image of the target object, and a reference face contour of the reference face image, the optical flow information being used to characterize the offset information between the reconstructed face image and the reference face image of the target object; and reconstructing facial features from the input face image based on the reference face image of the target object and the optical flow information to obtain the reconstructed face image.

[0007] According to one aspect of the embodiments of this application, an audio-driven face reconstruction apparatus is provided, comprising: an acquisition module for acquiring input audio; a first driving module for obtaining a target face contour of the target object under the driving action of the input audio, the target face contour including a lip-related contour, based on the input audio and an input face image of the target object; an optical flow information determination module for determining optical flow information based on the target face contour of the target object, a reference face image of the target object, and a reference face contour of the reference face image, the optical flow information being used to characterize the offset information between the reconstructed face image and the reference face image of the target object; and a reconstruction module for reconstructing facial features from the input face image based on the reference face image of the target object and the optical flow information to obtain the reconstructed face image.

[0008] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0009] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer-readable instructions thereon, which, when executed by a processor, implement the method described above.

[0010] According to one aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the method described above.

[0011] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0013] Figure 1A shows a schematic diagram of an application scenario where the technical solutions of the embodiments of this application can be applied.

[0014] Figure 1B is a schematic diagram illustrating a speech video frame sequence driven by input audio according to an embodiment of this application.

[0015] Figure 2 is a flowchart illustrating an audio-driven face reconstruction method according to an embodiment of this application.

[0016] Figure 3A is a flowchart illustrating an audio-driven face reconstruction method according to another embodiment of this application.

[0017] Figure 3B is a flowchart illustrating step 230 according to an embodiment of this application.

[0018] Figure 4 is a schematic diagram illustrating lip-shape keypoint driving according to an embodiment of this application.

[0019] Figure 5 is a flowchart illustrating step 240 according to another embodiment of this application.

[0020] Figure 6 is a flowchart illustrating optical flow estimation according to an embodiment of this application.

[0021] Figure 7 is a flowchart illustrating face reconstruction of the t-th frame of a speech video frame according to an embodiment of this application.

[0022] Figure 8 is a flowchart illustrating step 740 according to an embodiment of this application.

[0023] Figure 9 is a schematic diagram of face reconstruction using a face reconstruction network according to an embodiment of this application.

[0024] Figure 10 is a flowchart illustrating the training of an optical flow network and a face reconstruction network according to an embodiment of this application.

[0025] Figure 11 is a block diagram of an audio-driven face reconstruction apparatus according to an embodiment of this application.

[0026] Figure 12 shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0029] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0030] In this document, "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0031] Figure 1A illustrates a schematic diagram of an application scenario where the technical solutions of the embodiments of this application can be applied. As shown in Figure 1A, the application scenario includes a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected via a wired or wireless network communication connection.

[0032] Terminal 110 can upload the input audio and the input facial image of the target object to server 120. Server 120 can, according to the scheme of this application, drive the input facial image of the target object through the input audio to obtain the reconstructed facial image of the target object when it speaks according to the input audio.

[0033] Specifically, server 120 can first obtain the target facial contour of the target object under the driving force of the input audio based on the input audio and the input facial image of the target object. The target facial contour includes the contour related to the mouth shape. Then, based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image, optical flow information is determined. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object. Subsequently, by combining the reference facial image of the target object and the optical flow information, facial feature reconstruction is performed on the input facial image to obtain the reconstructed facial image.

[0034] In some embodiments, the input facial image of the target object can be a speech video of the target object. The speech video frame sequence of the target object can be determined based on the speech video of the target object. In this case, the server 120 can follow the process shown in FIG1B to drive the speech video frames in the speech video frame sequence by input audio. As shown in FIG1B, after selecting multiple key video frames reflecting the speaking lip-sync style of the target object from the speech video frame sequence, the server can drive the lip-sync key points of the face key point maps corresponding to each speech video frame in the speech video frame sequence according to the audio features of the input audio and the face key point maps (also known as face contours) corresponding to the multiple key video frames, to obtain a driven face key point map sequence. Subsequently, optical flow estimation is performed based on each driven face key point map in the driven face key point map sequence, the reference video frame selected in the speech video frame sequence, and the face key point map of the reference video frame to obtain an optical flow information sequence. Finally, face reconstruction is performed on the speech video frame sequence based on the reference video frame and the optical flow information sequence to obtain a target speech video frame sequence simulating the target object speaking according to the input audio.

[0035] Terminal 110 can be a smartphone, tablet, laptop, desktop computer, smart TV, in-vehicle terminal, virtual interactive device, smart wearable device, etc. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0036] Furthermore, the method of this application is not limited to being executed by server 120, but can also be executed by terminal 110, or implemented by interaction between server 120 and terminal 110.

[0037] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0038] Figure 2 is a flowchart illustrating an audio-driven face reconstruction method according to an embodiment of this application. This method can be executed by an electronic device with processing capabilities, such as a server or terminal, and is not specifically limited thereto. Referring to Figure 2, the method includes at least steps 201 to 204, which are described in detail below:

[0039] Step 201: Obtain the input audio.

[0040] Step 202: Based on the input audio and the input face image of the target object, obtain the target face contour of the target object under the driving action of the input audio. The target face contour includes the contour related to the mouth shape.

[0041] The target object can be a real person, a virtual character (such as a digital human), or a virtual role; no specific limitations are imposed here. The input facial image of the target object represents the facial region of the target object.

[0042] Input audio refers to the audio used to drive the input facial image of the target object. This input audio can be the audio of the target object, the audio of other objects, or audio obtained by converting text content into speech. The duration of the input audio can be the duration of a single vocal unit.

[0043] In some embodiments, the input facial image is a video of the target object speaking in a first language; the input audio is audio in a second language; the first language and the second language are different.

[0044] In some embodiments, audio features of the input audio can be extracted, and facial key points can be extracted from the input face image to obtain the facial contour of the input face image, which includes multiple facial key points extracted from the input face image. Then, using the audio features of the input audio and the facial contour of the input face image, lip-sync key point-driven lip-sync key point extraction is performed to obtain the target face contour of the target object under the influence of the input audio; that is, the lip shape of the target object presented in the target face contour is the lip shape of the target object when it pronounces the audio.

[0045] Audio feature extraction models can be used to extract features from input audio, yielding audio features. These models can be constructed using one or more neural networks. In some embodiments, the audio feature extraction model can be a whisper model or a wav2vec model. In some embodiments, MFCC (Mel Frequency Cepstral Coefficients) features of the input audio can also be extracted as audio features.

[0046] For trainable models such as Whisper and wav2vec, the training process can utilize a large amount of audio data and corresponding text labels. Taking the wav2vec model as an example, it employs self-supervised learning, first segmenting the audio into frames, and then using contrastive learning to allow the model to learn the feature representations of the audio. During training, the model's goal is to maximize the similarity between positive sample pairs (different representations of the same audio) and minimize the similarity between negative sample pairs (different audio representations). During training, stochastic gradient descent (SGD) or its improved algorithms (such as Adam) are used to update the model parameters, setting appropriate learning rates and training epochs until the model converges.

[0047] Facial key points are the key locations or features that represent facial characteristics. Examples of facial key points include features of the eyebrows, eyes, nose, mouth, and facial contours.

[0048] In some embodiments, facial keypoints can be extracted from the input face image to obtain a facial keypoint map corresponding to the input face image, which serves as the facial contour corresponding to the input face image. In other embodiments, facial keypoints can be extracted from the input face image to obtain multiple facial keypoints. Then, related facial keypoints among the multiple facial keypoints are connected by curves to obtain a facial keypoint map that reflects the facial contour, which serves as the facial contour corresponding to the input face image. Related facial keypoints can be multiple facial keypoints located in the same facial feature region. For example, multiple facial keypoints located in the nose region are related multiple facial keypoints, multiple facial keypoints located in the left eye region are related multiple facial keypoints, and multiple facial keypoints located in the lip region are related multiple facial keypoints.

[0049] In some embodiments, facial landmark extraction models can be used to extract facial landmarks from an input facial image. Different facial landmark extraction models may extract different facial landmarks. For example, a facial landmark extraction model that extracts 256 facial landmarks or a facial landmark extraction model that extracts 68 facial landmarks can be used to extract facial landmarks.

[0050] These facial landmark extraction models are typically built on convolutional neural networks (CNNs). Training requires a large dataset of facial images with labeled landmarks. Taking a CNN-based model as an example, labeled facial images are input into the model, and the model outputs the predicted landmark locations. A loss function (such as mean squared error loss) is used to measure the difference between the predicted and actual labeled landmarks. The gradient of the loss function with respect to the model parameters is calculated using backpropagation, and then optimization algorithms (such as stochastic gradient descent or Adam) are used to update the model parameters. During training, data augmentation techniques (such as rotation, flipping, scaling, etc.) can be used to increase data diversity and improve the model's generalization ability. Multiple rounds of training are conducted until the model's performance on the validation set stabilizes.

[0051] In some embodiments, the facial keypoint extraction model can be a model for extracting the two-dimensional coordinates of facial keypoints or a model for extracting the three-dimensional coordinates of facial keypoints. If the facial keypoint extraction model is a model for extracting the three-dimensional coordinates of facial keypoints, after extracting the three-dimensional coordinates of each facial keypoint through the facial keypoint extraction model, the three-dimensional coordinates of the facial keypoints can be transformed into a two-dimensional plane to obtain the facial keypoint map of the input facial image. Alternatively, after transforming the three-dimensional coordinates of the facial keypoints into a two-dimensional plane, multiple related facial keypoints can be connected to obtain the facial keypoint map of the input facial image.

[0052] In some embodiments, step 202 includes the following steps A1-A3:

[0053] Step A1: Obtain the facial pose key points of the input face image of the target object. The facial pose key points are facial key points that are independent of the lip shape.

[0054] Facial key points can be extracted from the input face image to obtain the corresponding facial key point map. Then, the mouth shape key points in the corresponding facial key point map of the input face image are occluded to obtain the occluded key point map. The facial key points presented in the occluded key point map (i.e. the unoccluded facial key points) are used as the facial pose key points of the input face image.

[0055] Step A2: Obtain the lip shape key points related to the lip shape of the target object.

[0056] Lip key points refer to facial key points related to the shape of the mouth. Examples of lip key points include multiple lip key points located on the lips and cheek key points near the lips.

[0057] In some embodiments, multiple speech images reflecting the lip-syncing style of the target object are determined, and then lip-syncing key points are extracted from each of the multiple speech images to obtain the lip-syncing key points in each speech image.

[0058] The lip-syncing style of a target audience member can be understood as their habitual lip movements during speech. This can be conveyed through changes in lip shape during speech. Therefore, multiple images showing different lip sizes can be used to reflect the target audience member's lip-syncing style. It's understandable that a larger number of images accurately expresses the target audience member's lip-syncing style, but this also increases the subsequent processing workload. Therefore, in specific embodiments, a trade-off between accuracy and processing volume can be struck, determining the number of selected images based on needs.

[0059] In some embodiments, multiple speaking images with different lip sizes can be selected from a sequence of speaking video frames of the target subject (e.g., the speaking video frame sequence from which the input face image is derived, or other speaking video frame sequences). Alternatively, multiple speaking images can be randomly selected from the speaking video frame sequence. Since it is a random selection, the probability of selecting speaking images with exactly the same or very similar lip sizes is low. Therefore, randomly selected multiple speaking images can reflect the speaking lip style of the target subject.

[0060] For example, if we call the pixels representing the lips in a speech image "lip pixels," then the closed pixel region formed by the lip pixels as boundaries in a video frame is the pixel region representing the mouth shape, which is called the mouth shape pixel region. Based on this, we can count the number of pixels in the mouth shape pixel region in each video frame of the speech video frame sequence. The larger the number of pixels in the mouth shape pixel region, the larger the mouth shape. Therefore, based on the number of pixels in the mouth shape pixel region of each video frame, we can select multiple video frames with different numbers of pixels in the mouth shape pixel region from the speech video frame sequence as multiple speech images.

[0061] In some embodiments, step A3 may include: selecting multiple speech images of the target object with different lip shapes; and extracting lip shape key points related to the lip shape based on the multiple speech images with different lip shapes. That is, extracting lip shape key points related to the lip shape from each speech image to obtain a lip shape key point map corresponding to each speech image. This lip shape key point map presents the pixel positions of the lip shape key points in each speech image, correspondingly reflecting the lip shape of the target object presented in each speech image.

[0062] In some embodiments, facial key points can be extracted from each speech image in a manner similar to that described above, resulting in a facial key point map corresponding to each speech image. This facial key point map presents the lip shape key points of the target object in the speech image that are related to the lip shape.

[0063] Step A3: Based on the input audio, lip shape key points, and facial pose key points, obtain the target facial contour of the target object.

[0064] In some embodiments, after determining the lip shape key points of the target object in the facial key point map corresponding to the input face image, the lip shape key points in the facial key point map corresponding to the input face image can be occluded to obtain an occluded key point map. In this way, the facial key points presented in the occluded key point map are the facial pose key points in the input face image.

[0065] The input audio, occlusion keypoint map, and facial keypoint maps (or lip keypoint maps) corresponding to multiple speech images are input into the keypoint reconstruction network. The keypoint reconstruction network reconstructs the lip keypoints of the occluded area in the occlusion keypoint map to obtain the target facial contour of the target object.

[0066] When reconstructing the network at keypoints, appropriate hyperparameters need to be set. For example, the learning rate can be set to 0.001, and the Adam optimizer can be used for parameter updates. The number of training epochs can be set to 50-100 epochs depending on the dataset size and model convergence. The batch size can be set to 16-32 to balance training speed and memory usage. Additionally, an early stopping strategy can be employed: when the loss on the validation set no longer decreases within a certain number of epochs, training can be stopped to prevent overfitting.

[0067] In the target facial contour of the target object, apart from the reconstructed lip-shape keypoints, the positions of other facial keypoints can be the same as the positions of the corresponding facial keypoints in the occluded keypoint map. The lip shape presented by the target facial contour is the lip shape of the pronunciation according to the input audio.

[0068] A keypoint reconstruction network is a neural network model used for audio-driven generation of lip-sync keypoints. In some embodiments, the keypoint reconstruction network may include a first encoder network and a first decoder network; for example, the keypoint reconstruction network may be a Transformer network. The first encoder network fuses and encodes the audio features of the input audio, facial keypoint maps (or lip-sync keypoint maps) corresponding to multiple speech images, and occlusion keypoint maps to obtain a fused feature map. Then, the fused feature map is input into the first decoder network, which decodes the fused feature map and outputs the target facial contour of the target object.

[0069] In the above embodiments, multiple speech images with different lip shapes are introduced to correspond to facial key point maps, which assist in driving the lip shape key point of the facial key point map corresponding to the input face image. This is to inject the lip shape style of the target object, because multiple speech images with different lip shapes can reflect the lip shape style of the target object, and the facial key point maps corresponding to multiple speech images with different lip shapes can also reflect the lip shape style of the target object. By injecting the lip shape style of the target object, it is ensured that the obtained target face contour of the target object not only corresponds to the lip shape of the speech content of the input audio, but also conforms to the lip shape style of the target object.

[0070] Step 203: Determine optical flow information based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object.

[0071] It is understandable that the target face contour can be approximated as the face key point map of the face image to be reconstructed. The optical flow between the reconstructed face image and the reference face image of the target object can reflect the optical flow of the face key point map of the reconstructed face image relative to the reference face image of the target object. Therefore, in this application, the optical flow between the reference face contour of the reference face image and the target face contour, as well as the pixels of the reference face image itself, can be used to estimate the optical flow information between the reconstructed face image and the reference face image of the target object in reverse.

[0072] In some embodiments, step 203 includes the following steps B1 to B3:

[0073] Step B1: Select i reference video frames from the target object's speech video frame sequence as reference facial images; i is an integer greater than 1.

[0074] The speech video frame sequence of the target object can be a speech video frame sequence from which the input face image is derived, or a speech video frame sequence that presents the speech process of the target object. Each speech video frame in the speech video frame sequence of the target object presents the facial region of the target object.

[0075] Step B2: Obtain the reference facial contour of the i-th reference video frame. Facial keypoint extraction can be performed in a similar manner to that described above to obtain the reference facial contour for each reference video frame.

[0076] In some embodiments, step B1 may be performed by at least one of the following two selection methods: selecting at least one speech video frame with the most tooth pixels in the speech video frame sequence of the target object; or selecting at least one speech video frame in the video frame window of the speech video frame sequence that contains the input face image when the input face image is from the speech video frame sequence.

[0077] A video frame window containing the input face image refers to a window with a certain width centered on the input face image. The width of the window can be represented by the number of video frames it covers. In some embodiments, at least one speech video frame can be randomly selected from the video frame window containing the input face image, or at least one speech video frame can be selected at equal intervals within the video frame window containing the input face image, serving as the i-th reference video frame mentioned above.

[0078] In other embodiments, at least one speech video frame with the most tooth pixels may be selected from the speech video frame sequence from which the face image is derived; and at least one speech video frame may be selected from the video frame window containing the input face image in the speech video frame sequence as i-frame reference video frame (i.e. reference face image).

[0079] Step B3: The optical flow network determines the optical flow information based on the target facial contour of the target object, the i-frame reference video frame, and the reference facial contour of the i-frame reference video frame.

[0080] Optical flow network refers to a neural network model used for optical flow estimation. In some embodiments, the optical flow network may be U-net (a U-shaped network based on convolutional neural networks), FlowNet (Learning Optical Flow with Convolutional Networks, an optical flow estimation network based on convolutional neural networks), a neural network model based on a transformer network, or other neural network models with an encoder-decoder structure, without being specifically limited here.

[0081] Before training the optical flow network, the input data can be preprocessed. For the input facial landmark maps and reference video frames, they should be uniformly resized (e.g., 256×256) and normalized, scaling pixel values ​​to the [0,1] range. During training, a stochastic gradient descent optimization strategy is used, with a learning rate set to 0.0001, and a learning rate decay mechanism is employed, reducing the learning rate to 0.9 times its original value every 10 epochs. Simultaneously, to prevent overfitting, a Dropout layer can be added to the network with a dropout rate set to 0.2.

[0082] The encoder network in the optical flow network is referred to as the second encoding network, and the decoder network in the optical flow network is referred to as the second decoding network. In some embodiments, the target facial contour of the target object, the i-frame reference video frame, and the reference facial contour of the i-frame reference video frame can be input into the second encoding network, which performs fusion encoding to obtain the first fused encoding feature; then, the second decoding network decodes the first fused encoding feature to obtain the optical flow information.

[0083] Optical flow information may include one or more optical flow maps, the number of which is the same as the number of selected reference video frames. An optical flow map reflects the offset information between the reconstructed face image to be generated and a reference face image (reference video frame). In some embodiments, the optical flow information may also include weights corresponding to each optical flow map. The weight of an optical flow map can be understood as its confidence level; that is, the higher the weight, the higher the confidence and accuracy of the optical flow map.

[0084] In some embodiments, step B3 includes: when the input face image comes from a speech video frame sequence, obtaining the facial contour of at least one adjacent face image adjacent to the input face image; inputting the target face contour, the facial contour of at least one adjacent face image, the i-frame reference video frame, and the reference face contour of the i-frame reference video frame into an optical flow network to obtain optical flow information.

[0085] The facial contour of at least one adjacent facial image adjacent to the input facial image is also used to determine optical flow information to ensure the temporal coherence of multiple optical flow maps.

[0086] Step 204: Based on the reference face image and optical flow information of the target object, perform facial feature reconstruction on the input face image to obtain the reconstructed face image.

[0087] In some embodiments, the process may involve facial feature reconstruction of facial regions in the input face image. The reconstructed face image includes facial pixel regions representing the face of the target object. The lip shape of the target object presented in the reconstructed face image is the same as the lip shape of the target object when pronouncing the input audio, and conforms to the speaking style of the target object. Furthermore, since the facial regions in the input face image are reconstructed, the background region of the reconstructed face image can be the same as the background region in the input face image.

[0088] In some embodiments, step 204 includes the following steps C1-C2:

[0089] Step C1: Occlude the lip-related region of the input face image to obtain a lip-occluded image.

[0090] A lip-masked image is an image obtained by masking the lip-related regions of an input face image (such as a speaking video frame). Except for the masked areas, the other pixel regions in this image are the same as those in the input face image. The masked pixel regions are used to indicate the pixel regions in the input face image that need to be reconstructed.

[0091] Lip-related occlusion refers to obscuring the pixel region in the input face image that represents the mouth shape. Therefore, the occluded image contains the same pixel regions as the input face image, except for the occluded area. It is understood that during lip-occlusion, the occluded pixel region includes at least the pixel region representing the mouth shape, and may also include the surrounding pixel regions.

[0092] In some embodiments, the lower half of the target object's cheek in the input face image can be occluded to obtain a lip-masking image, or the occluded area can be a pixel region that is based on the lower half of the target object's cheek and expanded outwards.

[0093] It's understandable that the occluded pixel areas in the lip-masking image are the pixel areas that need to be reconstructed in the subsequent face reconstruction process. From this perspective, the lip-masking image is used to indicate the pixel areas in the input face image that need to be reconstructed, while other unoccluded pixel areas do not need to be reconstructed in the subsequent process.

[0094] In step C2, the face reconstruction network performs face reconstruction based on the reference face image of the target object, optical flow information, and lip occlusion image to obtain the reconstructed face image.

[0095] In some embodiments, face reconstruction can be performed using a face reconstruction network, which is a neural network model that can generate images.

[0096] In training face reconstruction networks, in addition to using the loss functions mentioned above, regularization methods can be employed to improve the model's generalization ability. For example, L2 regularization can be added to the convolutional layers of the first image encoder, second image encoder, and image decoder, with a regularization coefficient set to 0.0001. This limits the size of the model parameters and prevents overfitting. Simultaneously, batch normalization can be used during training to accelerate model convergence and improve training stability.

[0097] The face reconstruction network can be reconstructed as follows: extract the image features of the reference face image of the target object, the optical flow information features of the optical flow information, and the image features of the lip occlusion image, fuse the three features to obtain the third fused feature, and decode the third fused feature to obtain the reconstructed face image.

[0098] In some embodiments, the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder; step C2 includes the following ①-④:

[0099] ① The reference facial image of the target object is encoded by the first image encoder to obtain the reference visual features.

[0100] The reference visual features are features obtained by encoding the reference face image of the target object by the first image encoder. They include sub-reference visual features extracted by the first image encoder for each reference video frame. In some embodiments, they may also include the k-th layer sub-reference visual features output by the first encoding layer of the first image encoder for each reference video frame.

[0101] ② Optical flow mapping is performed based on optical flow information and reference visual features to obtain optical flow fusion features.

[0102] Optical flow mapping refers to image transformation based on optical flow. In this application, the sub-reference visual features of each reference video frame relative to the reconstructed face image are processed using the sub-reference optical flow information in the optical flow information to obtain the optical flow mapping features corresponding to each reference video frame. In some embodiments, the sub-reference visual features of each reference video frame relative to the reconstructed face image in the optical flow information can be optically flow mapped to obtain the optical flow mapping features corresponding to each reference video frame. Then, the optical flow mapping features corresponding to all reference video frames are weighted to obtain the optical flow fusion features.

[0103] Optical flow fusion features are features obtained by optical flow mapping based on optical flow information and reference visual features. In this application, optical flow information includes the optical flow map and weights of the i-th reference video frame, and reference visual features include the k-th layer sub-reference visual features output by the k-th layer of the first encoding layer in the first image encoder for each reference video frame. Optical flow fusion features include the k-th layer sub-optical flow fusion features, which are obtained by weighting the k-th layer optical flow mapping features corresponding to each reference video frame.

[0104] In some embodiments, the optical flow information includes the optical flow map and weights of the i-th reference video frame; the reference visual features include the k-th layer sub-reference visual features output by the k-th layer first coding layer of the first image encoder for each reference video frame; the optical flow fusion features include the k-th layer sub-optical flow fusion features; 1≤k≤M; M is the number of first coding layers in the first image encoder; the step of performing optical flow mapping based on the optical flow information and the reference visual features to obtain the optical flow fusion features includes: performing optical flow mapping between the k-th layer sub-reference visual features of the w-th reference video frame and the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame; 1≤w≤i; w is a positive integer; and weighting the k-th layer optical flow mapping features corresponding to the i-th reference video frames according to the weights of the optical flow maps of each reference video frame to obtain the k-th layer sub-optical flow fusion features.

[0105] For example, for an i-th reference video frame, if the k-th layer of the first encoding layer in the first image encoder is the k-th layer sub-reference visual feature output from the w-th reference frame, then... In the i-th reference video frame, the optical flow graph of the w-th reference video frame relative to the reconstructed face image is f. w The weight of the w-th reference video frame relative to the optical flow graph of the reconstructed face image is α. w The sub-optical flow fusion feature Z of the k-th layer can be determined according to the following formula 1. k :

[0106] in, This represents the optical flow mapping feature of the k-th layer corresponding to the w-th reference video frame.

[0107] ③ The second image encoder performs fusion encoding based on the image features of the lip-occlusion image and the optical flow fusion features to obtain the target visual features.

[0108] The target visual features are obtained by fusing and encoding the image features and optical flow fusion features of the lip-occlusion image by the second image encoder. In some embodiments, the features are obtained by splicing and fusing the k-th layer sub-optical flow fusion features with the k-th layer encoded features, and finally used by the image decoder to decode and reconstruct the face image.

[0109] In some embodiments, the image features of the lip-occluded image and the optical flow fusion features are stitched together and then input into a second image encoder, which performs fusion encoding to obtain the target visual features.

[0110] In other embodiments, the second image encoder includes M second coding layers; in this embodiment, the step of obtaining target visual features by fusing and encoding the image features and optical flow fusion features of the lip-occluded image by the second image encoder includes: concatenating the k-th layer sub-optical flow fusion features with the k-th layer coding features to obtain the k-th layer concatenated features; the k-th layer coding features are output by the k-th layer second coding layer in the second image encoder; 1≤k≤M; when k=1, the 1-th layer coding features are the image features of the lip-occluded image; if k<M, the k+1-th layer second coding layer in the second image encoder performs fusing and encoding on the k-th layer concatenated features to obtain the k+1-th layer coding features; incrementing k by 1 and returning to execute the step of concatenating the k-th layer sub-optical flow fusion features with the k-th layer coding features to obtain the k-th layer concatenated features; if k=M, the M-th layer concatenated features are used as the target visual features.

[0111] In other words, the sub-optical flow fusion feature of the k-th layer is first concatenated with the feature output from the second coding layer of the k-th layer in the second image encoder, and then input into the (k+1)-th second coding layer in the second image encoder for fusion encoding. In this way, the features output from different first coding layers in the first image encoder can be fully utilized, rather than just the features output from the last first coding layer in the first image encoder, so as to ensure the accuracy of the target visual features and thus ensure the quality of the reconstructed face image obtained by subsequent decoding.

[0112] ④ The visual features of the target are decoded by the image decoder to obtain the reconstructed face image.

[0113] First, input audio and an input facial image of the target object are used to obtain the target facial contour of the target object driven by the input audio. Since the target facial contour includes a contour related to lip shape, the lip shape presented in the target facial contour is equivalent to the lip shape presented by the target object when pronouncing the input audio. Then, using the target facial contour of the target object, a reference facial image of the target object, and a reference facial contour of the reference facial image, the offset information (i.e., optical flow information) between the reconstructed facial image to be generated and the reference facial image of the target object is estimated in reverse. Using the reference facial image of the target object and the optical flow information, facial feature reconstruction is performed on the input facial image to obtain the reconstructed facial image. In this application, using the target facial contour as a constraint in the reconstruction process ensures that the lip shape presented in the obtained reconstructed facial image is consistent with the lip shape presented in the target facial contour, that is, the lip shape presented by the target object when pronouncing the input audio. Therefore, the problem of poor lip shape after audio driving in related technologies can be effectively solved, ensuring the quality of the reconstructed facial image.

[0114] The i-frame reference video frame selected for the input face image can provide rich oral cavity detail information, such as tooth details, as well as background information. This ensures that the reconstructed face image can accurately present the tooth details of the target object when pronouncing the input audio. Furthermore, the background pixel region in the i-frame reference video frame can provide accurate background information for the reconstruction process, especially when part of the background is also occluded in the lip-sync image, thus ensuring the quality of the obtained reconstructed video frame.

[0115] The following explanation will focus on the approach of driving the target object's speech video frame sequence through the first target video to obtain a simulated target object speaking according to the first input audio, thus illustrating the solution of this application. As shown in Figure 3A, the method may include the following steps 210-250:

[0116] Step 210: Obtain the first input audio and the video frame sequence of the target object's speech.

[0117] The target subject's speech video frame sequence shows the target subject's facial region in each frame. The target subject's speech video frame sequence is the video frame sequence to be driven. The target subject's speech video frame sequence can be a sequence of multiple video frames extracted sequentially and at intervals from the target subject's speech video. Alternatively, the target subject's speech video can be used as the speech video frame sequence. The target subject's speech video can be captured facing the target subject during the first target video's speech.

[0118] In some embodiments, the speech video frame sequence is extracted from a speech video of a target object speaking in a first language; the first input audio is the audio of the speech content corresponding to the speech video in a second language; the first language and the second language are different. The speech content corresponding to the speech video refers to the speech content of the target object in the speech video. The first language could be, for example, Mandarin Chinese and the second language could be English; or, for example, Mandarin Chinese and the second language could be German, without specific limitations. In this case, it is equivalent to using the first input audio and the speech video frame sequence of the target object to generate a first target video frame sequence that simulates the target object speaking the speech content in the first input audio in the second language, thereby automatically translating the speech video frame sequence of the target object.

[0119] Understandably, assuming the number of speech video frames in the speech video frame sequence is N, the first input audio can be divided into N audio units. The x-th audio unit among the N audio units is used to drive the x-th speech video frame in the speech video frame sequence. Therefore, the N audio units can be regarded as the input audio mentioned above, so as to drive the corresponding speech video frame. Each speech video frame in the speech video frame sequence can be regarded as the input face image to be reconstructed mentioned above. Step 220: Select multiple key video frames in the speech video frame sequence. These multiple key video frames are used to reflect the speaking lip-sync style of the target object.

[0120] Key video frames refer to video frames selected from a sequence of spoken video frames that reflect the lip-syncing style of a target subject. The lip-syncing style of a target subject can also be understood as their habitual mouth movements, which can be reflected by changes in their mouth shapes during speech. Therefore, multiple video frames showing different mouth shapes can be used to reflect the lip-syncing style of a target subject. It is understood that the more key video frames there are, the more accurately the lip-syncing style of the target subject can be expressed, but this also increases the subsequent processing workload. Therefore, in specific embodiments, a trade-off can be struck between accuracy and processing volume, and the number of key video frames selected can be determined according to needs.

[0121] In some embodiments, multiple speech video frames with different lip sizes can be selected from the speech video frame sequence as multiple key video frames. For example, multiple video frames can be randomly selected from the speech video frame sequence as key video frames. Since the selection is random, the probability of selecting video frames with completely identical or very similar lip sizes is small. Therefore, the randomly selected multiple video frames can reflect the speaking lip style of the target subject.

[0122] For example, if we call the pixels representing lips in a video frame "lip pixels," then the closed pixel region formed by the lip pixels as boundaries within the video frame is the pixel region representing the mouth shape, and is called the mouth shape pixel region. Based on this, we can count the number of pixels in the mouth shape pixel region in each video frame of the speech video frame sequence. The larger the number of pixels in the mouth shape pixel region, the larger the mouth shape. Therefore, based on the number of pixels in the mouth shape pixel region of each video frame, we can select multiple video frames with different numbers of pixels in the mouth shape pixel region from the speech video frame sequence as key video frames.

[0123] Step 230: Based on the audio features of the first input audio and the facial key point maps of multiple key video frames, perform lip-shape key point driving on the facial key point maps of each speech video frame in the speech video frame sequence to obtain a driven facial key point map sequence.

[0124] The first input audio-driven facial keypoint map sequence includes multiple sequentially arranged driven facial keypoint maps. The t-th driven facial keypoint map in the sequence is obtained by driving the lip-shape keypoints of the facial keypoint map corresponding to the t-th video frame in the speech video frame sequence. The number of video frames in the speech video frame sequence is the same as the number of driven facial keypoint maps in the sequence. The driven facial keypoint map sequence is used to present the facial keypoint maps of the target object during the simulated speech of the target object according to the first input audio. The lip shapes presented by each driven facial keypoint map in the sequence not only match the first input audio but also conform to the speaking lip-shape style of the target object.

[0125] Lip-sync keypoint driving is applied to the facial keypoint maps corresponding to each speech video frame in the speech video frame sequence to simulate the facial keypoint map of the target object when speaking according to the first input audio. In other words, the driving facial keypoint map sequence is used to present the facial keypoint map of the target object during the process of simulating the target object speaking according to the first input audio.

[0126] The t-th driving facial landmark map corresponding to the t-th speech video frame in the speech video frame sequence presents the facial landmarks of the target object when it pronounces according to the t-th audio segment in the first input audio. The first input audio can be sequentially divided into N audio segments of equal duration according to the number of video frames N in the speech video frame sequence. The t-th audio segment in the first input audio is used to drive the t-th speech video frame in the speech video frame sequence. Furthermore, the other audio segments in the first input audio besides the t-th audio segment can serve as auxiliary information to help determine the lip-sync landmark driving result of the t-th speech video frame in the speech video frame sequence.

[0127] In step 230, facial keypoint maps corresponding to multiple key video frames are introduced to assist in driving the lip-shape keypoints of the facial keypoint maps corresponding to each speech video frame in the speech video frame sequence. This is to inject the target object's speech lip-shape style, because multiple key video frames can reflect the target object's speech lip-shape style, and the facial keypoint maps corresponding to multiple key video frames can also reflect the target object's speech lip-shape style. By injecting the target object's speech lip-shape style, it is ensured that the obtained driven facial keypoint map sequence not only corresponds to the lip-shape of the speech content at the corresponding moment in the first input audio, but also conforms to the target object's speech lip-shape style. In other words, on the one hand, the lip-shape presented by the t-th driven facial keypoint map is the lip-shape when the t-th audio segment is pronounced; on the other hand, the introduction of facial keypoint maps corresponding to multiple key video frames ensures that the lip-shape presented by the generated t-th driven facial keypoint map conforms to the target object's speech lip-shape style.

[0128] In some embodiments, as shown in FIG3B, step 230 includes: step 310, obtaining the t-th facial key point image of the t-th frame of the speech video frame sequence; t is a positive integer ranging from 1 to N; N is the number of speech video frames in the speech video frame sequence. Step 320, performing lip shape key point removal processing on the t-th facial key point image to obtain the t-th candidate key point image.

[0129] The t-th candidate keyframe image refers to the facial keyframe image after removing lip-related keyframes from the t-th facial keyframe image. After removing lip-related keyframes from the t-th facial keyframe image, the facial keyframes unrelated to lip-related features are retained, such as keyframes located in the bridge of the nose area, the eye area, and the forehead area. Lip-related keyframes in the t-th facial keyframe image can also be understood as keyframes representing facial poses other than lip-related features.

[0130] Retain the facial key points in the t-th facial key point image that are not related to the lip shape. This ensures that after the lip shape key point is driven in the subsequent facial key point image, the position of the facial key points that are not related to the lip shape is basically the same as the position in the t-th facial key point image. That is, the facial poses other than the lip shape are basically the same as the facial poses other than the lip shape in the t-th facial key point image.

[0131] It is understandable that the missing facial key points in the t-th candidate key point image compared to the t-th facial key point image are the facial key points that need to be regenerated.

[0132] Step 330: The key point reconstruction network reconstructs the lip shape key points of the t-th candidate key point map based on the audio features of the first input audio and the facial key point maps of multiple key video frames, to obtain the t-th driving facial key point map corresponding to the t-th speaking video frame.

[0133] A keypoint reconstruction network is a neural network model used to drive the generation of lip-sync keypoints in audio. In some embodiments, the keypoint reconstruction network may include a first encoder network and a first decoder network; for example, the keypoint reconstruction network may be a Transformer network. The first encoder network fuses and encodes the audio features of the first input audio, facial keypoint maps of multiple key video frames, and the t-th candidate keypoint map to obtain a fused feature map. Then, the fused feature map is input into the first decoder network, which decodes the fused feature map and outputs the t-th driving facial keypoint map corresponding to the t-th speech video frame.

[0134] In some embodiments, the sub-audio features of the t-th audio segment can be obtained from the audio features of the first input audio. The first encoder network then fuses and encodes the sub-audio features of the t-th audio segment, the facial key point maps corresponding to multiple key video frames, and the t-th candidate key point map to obtain a fused feature map. Subsequently, the fused feature map is input into the first decoder network, which decodes the fused feature map and outputs the t-th driving facial key point map corresponding to the t-th speech video frame.

[0135] Similarly, the relevant facial key points in the t-th driving facial key point map can be connected to obtain a driving facial key point map that reflects the facial contour.

[0136] Figure 4 is a schematic diagram of lip-shape keypoint driving according to an embodiment of this application. The audio features of the first input audio, the facial keypoint maps corresponding to multiple key video frames, and the t-th candidate keypoint map are input into the keypoint reconstruction network. Based on the audio features of the first input audio and the facial keypoint maps corresponding to multiple key video frames, lip-shape keypoints representing the lip shape of the t-th audio unit in the first input audio and matching the speaking lip-shape style of the target object are generated, i.e., reconstructed lip-shape keypoints. Then, the reconstructed lip-shape keypoints and the facial keypoints in the t-th candidate keypoint map are combined, and the related facial keypoints are connected to output the t-th driving facial keypoint map.

[0137] In some embodiments, instead of removing lip-shape key points from the t-th facial key point map corresponding to the t-th speech video frame, the t-th facial key point map corresponding to the t-th speech video frame, the facial key point maps corresponding to multiple key video frames, and the audio features of the first input audio (or the sub-audio features of the t-th audio unit in the first input audio) are input into the key point reconstruction network. The key point reconstruction network then reconstructs the lip-shape key points in the t-th facial key point map and outputs the t-th driving facial key point map.

[0138] Step 240: Optical flow estimation is performed based on each driving facial key point map in the driving facial key point map sequence, the reference video frame selected in the speech video frame sequence, and the facial key point map of the reference video frame to obtain the optical flow information sequence.

[0139] The optical flow information sequence includes optical flow information applicable to each reconstructed video frame, used to describe the pixel movement of the reference video frame relative to the reconstructed video frame, such as pixel offset and offset direction. A reconstructed video frame refers to the video frame to be generated by driving each video frame in the speech video frame sequence using the first input audio, i.e., the video frame in the target speech video frame sequence below. The optical flow information applicable to a single reconstructed video frame describes the pixel movement of the reference video frame relative to the reconstructed video frame, such as pixel offset and offset direction. The target speech video frame sequence is a sequence obtained by driving the speech video frame sequence of the target object using the first input audio. Each reconstructed video frame in this sequence is obtained by facial reconstruction of the corresponding speech video frame in the speech video frame sequence. The lip shape of the target object presented in the reconstructed video frame is the same as the lip shape of the target object when pronouncing the first input audio, and conforms to the speaking style of the target object. Its background area is the same as the background area of ​​the corresponding frame in the speech video frame sequence.

[0140] It is understandable that the t-th driving facial keypoint map in the driving facial keypoint map sequence can be approximated as the facial keypoint map of the t-th reconstructed video frame to be generated. The optical flow of the reference video frame relative to the reconstructed video frame can reflect the facial keypoint map of the reference video frame relative to the facial keypoint map of the t-th reconstructed video frame. Therefore, in this application, the optical flow of the facial keypoint map of the reference video frame relative to the facial keypoint map of the t-th reconstructed video frame, as well as the pixels of the reference video frame itself, can be used to inversely estimate the optical flow information of the reference video frame relative to the t-th reconstructed video frame.

[0141] The reference video frame selected from the speech video frame sequence can be one or more, without specific limitation. It is understood that if there are multiple reference video frames, the optical flow information applicable to the t-th reconstructed video frame includes the sub-optical flow information of each of the multiple reference video frames relative to the t-th reconstructed video frame. The sub-optical flow information of a reference video frame relative to the t-th reconstructed video frame may include the optical flow map of that reference video frame relative to the t-th reconstructed video frame. In other embodiments, the sub-optical flow information of a reference video frame relative to the t-th reconstructed video frame may also include the weights corresponding to the optical flow map.

[0142] In some embodiments, the reference video frames selected for different reconstructed video frames may be the same or different. In some embodiments, one or more video frames may be randomly selected from the speech video frame sequence as reference video frames.

[0143] In some embodiments, as shown in FIG5, step 240 includes: step 510, selecting i reference video frames for the reconstructed video frame t in the speech video frame sequence; t is a positive integer from 1 to N; N is the number of reconstructed video frames in the target speech video frame sequence; 1≤i≤N, i is a positive integer.

[0144] The value of i can be set as needed; it can be 1 or other values. In specific embodiments, to facilitate the provision of rich optical flow information, i can be an integer greater than 1.

[0145] In some embodiments, the multiple key video frames selected above can be used as the i-th reference video frame selected for the reconstructed video frame. In other embodiments, i-th video frames can be randomly selected from the speech video frame sequence as the i-th reference video frame selected for the reconstructed video frame t.

[0146] In some embodiments, to ensure that each video frame in the generated target speech video frame sequence has richer tooth details, multiple video frames with rich tooth details can be selected from the speech video frame sequence as the i-th reference video frame selected for obtaining the reconstructed video frame for frame t. For example, the i-th speech video frame with the most tooth pixels can be selected from the speech video frame sequence as the i-th reference video frame selected for obtaining the reconstructed video frame for frame t. Tooth pixels refer to the pixels representing teeth in a video frame.

[0147] In other embodiments, i > 1, step 510 includes: selecting the m-th speech video frame with the most tooth pixels in the speech video frame sequence; 1 ≤ m < i, where m is a positive integer; selecting the im-th speech video frame in the window where the t-th speech video frame is located in the speech video frame sequence; and using the m-th speech video frame and the im-th speech video frame as the i-th reference video frame selected for the reconstructed video frame of the t-th frame.

[0148] In other words, among the i-frame reference video frames selected for reconstructing the video frame for frame t, one part is the m-frame speech video frame with the most tooth pixels, and the other part is one or more speech video frames near the speech video frame for frame t in the speech video frame sequence.

[0149] The window containing the t-th frame of the speech video is a window with a certain width centered on the t-th frame. The width of the window can be represented by the number of video frames it covers. In some embodiments, either 1m frames can be randomly selected from the window containing the t-th frame, or 1m frames can be selected at equal intervals within the window containing the t-th frame.

[0150] For example, if i = 10 and m = 5, select the 5 video frames with the most tooth pixels from the speech video frame sequence to ensure the richness and clarity of tooth details in the subsequently generated video frames. Assume the window containing the t-th speech video frame extends from frame ta / 2 to frame t+a / 2, where a is an even number and a is the width of the window, for example, a = 20. Select 5 video frames at equal intervals from the window of frame ta / 2 to frame t+a / 2 from the speech video frame sequence as another 5 reference frames. By selecting video frames near the t-th speech video frame, accurate background details are provided.

[0151] Step 520: Obtain the facial key point map of the i-frame reference video frame.

[0152] Step 530: The optical flow network estimates the optical flow of the i-th reference video frame relative to the t-th reconstructed video frame based on the t-th driving facial keypoint map in the driving facial keypoint map sequence, the i-th reference video frame, and the facial keypoint map of the i-th reference video frame, and obtains the optical flow information applicable to the t-th reconstructed video frame.

[0153] In some embodiments, the t-th driving facial key point map, the i-th reference video frame, and the facial key point map of the i-th reference video frame can be input into a second coding network, which performs fusion coding to obtain a first fusion coded feature; then, the second decoding network decodes the first fusion coded feature to obtain optical flow information suitable for the reconstructed video frame t.

[0154] The optical flow information applicable to the reconstructed video frame t may include i optical flow graphs, each representing the optical flow of a reference video frame relative to the reconstructed video frame t. In some embodiments, the optical flow information applicable to the reconstructed video frame t may further include weights of the i optical flow graphs.

[0155] In other embodiments, step 530 includes the following ①-③:

[0156] ① Obtain at least one adjacent driving face keypoint map from the t-th driving face keypoint map sequence.

[0157] Multiple driving facial keypoint maps adjacent to and before the t-th driving facial keypoint map can be selected as adjacent driving facial keypoint maps determined for the t-th driving facial keypoint map. The number of adjacent driving facial keypoint maps selected before the t-th driving facial keypoint map can be the same as or different from the number of adjacent driving facial keypoint maps selected after the t-th driving facial keypoint map.

[0158] For example, if it is required to determine 2b adjacent driving facial keypoint maps for each driving facial keypoint map, the preceding b frames of driving facial keypoint maps and the following b frames of driving facial keypoint maps adjacent to the t-th driving facial keypoint map in the driving facial keypoint map sequence can be selected as the 2b adjacent driving facial keypoint maps determined for the t-th driving facial keypoint map, where b is a positive integer and b is less than N / 2.

[0159] At least one adjacent driving face keypoint map is determined for the t-th driving face keypoint map to help determine the optical flow information corresponding to the t-th driving face keypoint map, so as to ensure the temporal continuity of the optical flow information corresponding to the continuous driving face keypoint maps.

[0160] In some embodiments, when t=1, b consecutive driving facial keypoint maps following the first driving facial keypoint map can be obtained from the driving facial keypoint map sequence as the corresponding adjacent driving facial keypoint maps. When t=N, b consecutive driving facial keypoint maps preceding the Nth driving facial keypoint map can be obtained from the driving facial keypoint map sequence as the corresponding adjacent driving facial keypoint maps.

[0161] ② Input the t-th driving facial landmark map, at least one adjacent driving facial landmark map, i-frame reference video frame, and the facial landmark map of i-frame reference video frame into the optical flow network.

[0162] The second coding network in the optical flow network can fuse and encode the t-th driving facial keypoint map, at least one adjacent driving facial keypoint map, i-frame reference video frame, and the reference facial keypoint map corresponding to i-frame reference video frame respectively to obtain the first fused coding feature; then, the second decoding network decodes the first fused coding feature to obtain the optical flow information corresponding to the t-th driving facial keypoint map.

[0163] ③ Obtain the optical flow information output by the optical flow network that is suitable for reconstructing the video frame at frame t.

[0164] Figure 6 is a flowchart of optical flow estimation according to an embodiment of this application. As shown in Figure 6, in order to determine the optical flow information corresponding to the t-th driving facial keypoint map, the input of the optical flow network includes three parts: ① the tb-th to t+b-th driving facial keypoint maps in the driving facial keypoint map sequence (wherein, the b driving facial keypoint maps before the t-th driving facial keypoint map, the b driving facial keypoint maps after the t-th driving facial keypoint map, and the t-th driving facial keypoint map are 2b adjacent driving facial keypoint maps determined for the t-th driving facial keypoint map); ② 5 reference video frames selected for the reconstructed video frame t-th frame; ③ facial keypoint maps of the 5 reference video frames.

[0165] The optical flow network can perform optical flow estimation based on the three inputs mentioned above, obtaining optical flow information suitable for the reconstructed video frame t. As shown in Figure 6, the optical flow information of the reconstructed video frame t includes optical flow maps of the five reference video frames relative to the reconstructed video frame t, and the weights of the five optical flow maps.

[0166] It is worth mentioning that, in Figure 6, in order to protect portrait rights, the facial areas in the 5 reference video frames shown in Figure 6 have been occluded. In the actual application of the solution of this application, it is not necessary to occlude or blur the facial areas of the reference video frames that need to be input into the optical flow network.

[0167] Step 250: Based on the reference video frame and optical flow information sequence, perform face reconstruction on the speech video frame sequence to obtain the target speech video frame sequence simulating the target object speaking according to the first input audio.

[0168] The reconstructed video frame t in the target speech video frame sequence is obtained by reconstructing the face from the speech video frame t in the speech video frame sequence. The number of video frames in the target speech video frame sequence is the same as the number of speech video frames in the speech video frame sequence. If the speech video frame t in the speech video frame sequence is considered as the input face image mentioned above, then the reconstructed video frame t in the target speech video frame sequence can be considered as the reconstructed face image mentioned above.

[0169] The reconstructed video frame includes a facial pixel region representing the target object's face. The lip shape of the target object presented in the t-th reconstructed video frame of the target speech video frame sequence is the same as the lip shape of the target object when pronouncing the t-th audio unit in the first target video, and conforms to the target object's speaking lip-shape style. Furthermore, since the facial region in the speech video frame is reconstructed, the background region in the t-th reconstructed video frame of the target speech video frame sequence can be the same as the background region in the t-th speech video frame of the speech video frame sequence.

[0170] In some embodiments, face reconstruction can be performed using a face reconstruction network, which is a neural network model capable of generating images. The face reconstruction network can sequentially reconstruct faces from each video frame in a speech video frame sequence. For example, for the t-th speech video frame in the speech video frame sequence, face reconstruction can be performed as follows: Optical flow information suitable for the t-th reconstructed video frame is obtained from the optical flow information sequence; the face reconstruction network can extract image features from the reference video frame, optical flow information features corresponding to the t-th driving facial keypoint map, and image features from the t-th speech video frame; these three features are then fused to obtain a third fused feature; and the third fused feature is decoded to obtain the t-th reconstructed video frame.

[0171] In other embodiments, face reconstruction can be performed on the t-th speech video frame in the speech video frame sequence according to the process shown in Figure 7, including:

[0172] Step 710: Obtain optical flow information applicable to the reconstructed video frame t.

[0173] Step 720: Obtain the i-th reference video frame selected for reconstructing the video frame of frame t.

[0174] Step 730: Lip occlusion is performed on the t-th frame of the speech video to obtain the t-th frame lip occlusion image.

[0175] Lip occlusion refers to obscuring the pixel region in the t-th frame of a spoken video frame that represents the mouth shape. Therefore, the lip-occluded image of the t-th frame, except for the obscured area, contains the same pixel regions as the t-th frame of the spoken video frame. It is understood that during lip occlusion, the obscured pixel region includes at least the pixel region representing the mouth shape, and may also include the surrounding pixel regions.

[0176] In some embodiments, the lower half of the target object's cheek in the t-th frame of the speaking video can be occluded to obtain the t-th frame lip-sync image, or the occluded area can be a pixel region based on the lower half of the target object's cheek and expanded outwards.

[0177] It is understandable that the occluded pixel areas presented in the lip-sync occlusion image of frame t are the pixel areas that need to be reconstructed in the subsequent face reconstruction process. From this perspective, the lip-sync occlusion image of frame t is used to indicate the pixel areas that need to be reconstructed in the speaking video frame of frame t, while other unoccluded pixel areas do not need to be reconstructed in the subsequent process.

[0178] Step 740: The face reconstruction network performs face reconstruction based on the optical flow information applicable to the reconstructed video frame t, the i-frame reference video frame selected for the reconstructed video frame t, and the lip occlusion image of the t-th frame, to obtain the reconstructed video frame t.

[0179] The i-th reference video frame selected for the t-th driving facial keypoint map can provide rich oral cavity detail information, such as tooth details, as well as background information. This ensures that the reconstructed video frame can accurately represent the tooth details of the target object when pronouncing the corresponding audio unit. Furthermore, the background pixel regions in the i-th reference video frame provide accurate background information for the reconstruction process, especially when part of the background is occluded in the t-th lip-sync image, thus ensuring the quality of the obtained reconstructed video frame. For each speech video frame in the speech video frame sequence, the face reconstruction network can process it according to a similar process as shown in Figure 7.

[0180] In this application, lip-sync keypoints are used to drive the facial keypoint maps corresponding to each speech video frame in the target object's speech video frame sequence using the audio features of the first input audio and facial keypoint maps of multiple key video frames. This results in a driven facial keypoint map sequence, which presents the facial keypoints of the target object speaking according to the first input audio. Since multiple key video frames can reflect the target object's lip-sync style, it is equivalent to injecting the target object's lip-sync style into the lip-sync keypoint driving process. This ensures that the lip-sync presented in each driven facial keypoint map in the obtained driven facial keypoint map sequence not only matches the first input audio but also conforms to the target object's lip-sync style.

[0181] Therefore, subsequent optical flow estimation based on the driving facial key point map sequence results in an optical flow information sequence that incorporates the target subject's lip-sync style. This ensures that subsequent facial reconstruction of the speech video frame sequence based on the reference video frame and the optical flow information sequence, yielding the target speech video frame sequence, guarantees that the lip-sync of the target subject in the obtained target speech video frame sequence matches the lip-sync of the subject speaking according to the first input audio, and also conforms to the target subject's lip-sync style. The solution of this application effectively solves the problem in related technologies where images generated by audio-driven lip-sync lack the unique lip-sync style of the individual.

[0182] Considering the optical flow of the facial keypoint map of the reference video frame relative to each driving facial keypoint map in the driving facial keypoint map sequence, which reflects the optical flow of the reference video frame relative to the video frame to be generated, optical flow estimation is performed by combining the facial keypoint map of the reference video frame, the driving facial keypoint map sequence, and the reference video frame to obtain an optical flow information sequence. This optical flow information sequence reflects the optical flow of the reference video frame relative to the video frame to be generated. Subsequently, the optical flow information sequence and the reference video frame are used to reconstruct the face of the speech video frame sequence, resulting in a target speech video frame sequence simulating the target object speaking according to the first input audio. Thus, it is possible to back-estimate the optical flow information between different video frames based on the optical flow information between facial keypoints in different video frames, and then combine the estimated optical flow information with the reference video frame as a reference for lip-syncing.

[0183] In related technologies, it is generally required that the mouth shape presented in the pre-driving speech video frame sequence is a closed mouth shape, in order to avoid significant jitter in the timing of the mouth shape after driving if the pre-driving speech video frame sequence shows a large amount of talking.

[0184] In this application, lip-sync keypoints are driven first to obtain a sequence of driven facial keypoint maps. Optical flow estimation and face reconstruction are then performed based on this sequence. Each driven facial keypoint map in the sequence constrains the facial keypoints in the reconstructed video frames. By driving lip-sync keypoints first, this application prioritizes the stability of the lip movements presented in the reconstructed video frames. Furthermore, during the optical flow estimation stage, at least one adjacent driven facial keypoint map is used as auxiliary information to assist in optical flow estimation. This ensures the temporal continuity of optical flow information applicable to consecutive reconstructed video frames, thereby guaranteeing the temporal continuity of facial poses (including lip movements) presented in the reconstructed video frames and preventing jitter. Thus, even if the lip movements in the preceding speech video frame sequence are large-scale speech movements, lip movement jitter will not occur. Therefore, this application's solution does not have requirements regarding the lip movements presented in the preceding speech video frame sequence, making it more universally applicable.

[0185] Furthermore, in this application, the m-frame speech video with the most tooth pixels can be selected as part or all of the reference video frames. The m-frame speech video with the most tooth pixels can provide rich tooth detail information of the target object, which can ensure that the tooth details presented in the generated reconstructed video frame are basically the same as the tooth details of the target object presented in the speech video frame sequence before driving. In this way, while ensuring the preservation of the target object's lip-sync style, the tooth details of the target object are well preserved, ensuring the lip-sync driving quality of the speech video frame sequence.

[0186] Furthermore, in the solution of this application, the reference video frames used for auxiliary driving can be determined according to the requirements. The more reference video frames there are, the more stable the target speech video frame sequence obtained by driving will be, but the corresponding time spent on lip-syncing will also be longer. Conversely, the fewer reference video frames there are, the shorter the time spent on lip-syncing can be.

[0187] In some embodiments, the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder; as shown in FIG8, step 740 includes:

[0188] Step 810: The first image encoder encodes the i-th reference video frame selected as the reconstructed video frame for the t-th frame to obtain reference visual features.

[0189] The reference visual features include the sub-reference visual features extracted by the first image encoder for each reference video frame selected for the reconstruction of the video frame t.

[0190] Since the first image encoder includes multiple image coding layers, for ease of distinction, the image coding layer in the first image encoder is referred to as the first coding layer. In some embodiments, the visual features output by the last layer of the first coding layer in the first image encoder as the reference video frame can be used as the sub-reference visual features of the reference video frame.

[0191] In other embodiments, the visual features output by each first coding layer in the first image encoder for a reference video frame can also be used as sub-reference visual features of that reference video frame. In other words, in this case, the reference visual features include the k-th layer sub-reference visual features output by the k-th first coding layer in the first image encoder for each reference video frame; 1≤k≤M; M is the total number of first coding layers in the first image encoder.

[0192] Step 820: Perform optical flow mapping based on the optical flow information corresponding to the t-th driving facial key point map and the reference visual features to obtain optical flow fusion features.

[0193] Optical flow mapping refers to the transformation of an image based on optical flow. In some embodiments, it can be achieved by using the sub-reference optical flow information of each reference video frame relative to the reconstructed video frame t in the optical flow information applicable to the t-th frame, performing optical flow mapping on the sub-reference visual features of the corresponding reference video frames, obtaining the optical flow mapping features corresponding to each reference video frame, and then weighting the optical flow mapping features corresponding to all reference video frames corresponding to the t-th frame reconstructed video frame to obtain the optical flow fusion features.

[0194] In some embodiments, if the optical flow information corresponding to the t-th driving facial keypoint map includes the sub-reference optical flow information of the i-th reference video frame corresponding to the t-th driving facial keypoint map relative to the t-th driving facial keypoint map; the sub-reference optical flow information includes the optical flow map and the weight of the optical flow map; in this embodiment, step 820 includes: performing optical flow mapping on the k-th layer sub-reference visual features of the w-th reference video frame and the optical flow map of the w-th reference video frame relative to the t-th reconstructed video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame; 1≤w≤i; w is a positive integer; according to the weight of the optical flow map of each reference video frame relative to the t-th reconstructed video frame, the k-th layer optical flow mapping features corresponding to the i-th reference video frames are weighted to obtain the k-th layer sub-optical flow fusion features.

[0195] For example, for the i-th reference video frame selected for reconstructing the t-th video frame, if the k-th layer of the first coding layer in the first image encoder is the k-th layer sub-reference visual feature output from the w-th reference frame, then... Among the i-th reference video frames selected for the reconstructed video frame t, the optical flow diagram of the w-th reference video frame relative to the reconstructed video frame t is as follows: The weights of the optical flow graph of the w-th reference video frame relative to the t-th reconstructed video frame are: The sub-optical flow fusion characteristics of the k-th layer can be determined according to the following formula 2.

[0196] in, This represents the optical flow mapping feature of the k-th layer corresponding to the w-th reference video frame.

[0197] Step 830: The second image encoder performs fusion encoding based on the image features and optical flow fusion features of the lip-masking image in frame t to obtain the target visual features.

[0198] In some embodiments, the image features and optical flow fusion features of the lip-shaped occlusion image in frame t can be stitched together and then input into a second image encoder, which performs fusion encoding to obtain the target visual features.

[0199] In some embodiments, the second image encoder may include multiple coding layers, referred to as the second coding layer in the second image encoder. The number of second coding layers in the second image encoder is M, which is the same as the number of first coding layers in the first image encoder. Step 830 includes: concatenating the k-th layer sub-optical flow fusion feature with the k-th layer coding feature to obtain the k-th layer concatenated feature; the k-th layer coding feature is output by the k-th layer second coding layer in the second image encoder; 1≤k≤M; when k=1, the 1-th layer coding feature is the image feature of the lip occlusion image of the t-th frame; if k<M, the k-th layer concatenated feature is fused and encoded by the (k+1)-th layer second coding layer in the second image encoder to obtain the (k+1)-th layer coding feature; incrementing k by 1 and returning to execute the step of concatenating the k-th layer sub-optical flow fusion feature with the k-th layer coding feature to obtain the k-th layer concatenated feature; if k=M, the M-th layer concatenated feature is used as the target visual feature.

[0200] In other words, the k-th layer sub-optical flow fusion feature is first concatenated with the feature output from the k-th second coding layer of the second image encoder, and then input into the (k+1)-th second coding layer of the second image encoder for fusion encoding. This fully utilizes the features output from different first coding layers in the first image encoder, rather than just the features output from the last first coding layer, ensuring the accuracy of the target visual features and thus guaranteeing the quality of the reconstructed video frames obtained from subsequent decoding.

[0201] Step 840: The visual features of the target are decoded by the image decoder to obtain the reconstructed video frame t.

[0202] Understandably, by processing each video frame in the speech video frame sequence according to the above process, the corresponding reconstructed video frame can be obtained.

[0203] Figure 9 is a schematic diagram of a face reconstruction network according to an embodiment of this application. Corresponding to the embodiment in Figure 6, to determine the reconstructed video frame corresponding to the t-th frame of the speech video (i.e., the t-th frame of the reconstructed video), five reference video frames selected for the t-th frame of the reconstructed video during optical flow estimation are used for face reconstruction. As shown in Figure 9, the five reference video frames selected for the t-th frame of the reconstructed video are input into a first image encoder. Each first coding layer in the first image encoder performs feature encoding on each reference video frame, obtaining the features output by each first coding layer for each reference video frame. Assuming the feature output by the k-th layer of the first image encoder, which is the input w-th frame of the reference, is the k-th layer sub-reference visual feature, then...

[0204] Subsequently, using the optical flow maps of the five reference video frames determined above relative to the reconstructed video frame t, and the weights corresponding to the five optical flow maps, the features output by the same first coding layer in the first image encoder for each reference video frame are fused according to Formula 1 above to obtain the sub-optical flow fusion features of each layer. For example, the features output by the first coding layer of the k-th layer in the first image encoder for the five reference video frames are fused to obtain the sub-optical flow fusion features of the k-th layer.

[0205] Subsequently, the lip-sync image of frame t is encoded by the first layer (second coding layer) of the second image encoder to obtain the image features of frame t. Then, the corresponding sub-optical flow fusion features of the same layer are concatenated with the features output from the corresponding layer in the second image encoder. The concatenated features are then input into the next layer (second coding layer) of the second image encoder for fusion encoding. For example, the sub-optical flow fusion features of layer k... The features output from the second coding layer at layer k in the second image encoder are concatenated and then input into the second coding layer at layer k+1 in the second image encoder for fusion encoding.

[0206] After the features output from the last layer of the second encoding layer in the second image encoder are spliced ​​with the sub-optical flow fusion features of the corresponding layer to obtain the target visual features, the image decoder decodes the target visual features to obtain the t-th reconstructed video frame.

[0207] It is worth mentioning that, in Figure 9, in order to protect portrait rights, the facial regions in the five reference video frames shown in Figure 9, as well as the facial regions in the t-th reconstructed video frame obtained by facial reconstruction, are obscured. In actual application of the scheme of this application, it is not necessary to obscure or blur the facial regions of the reference video frames.

[0208] In some embodiments, to ensure the quality of the reconstructed video frames, the optical flow network and face reconstruction network mentioned above need to be trained in advance. The training process for the optical flow network and face reconstruction network can be shown in Figure 10, including:

[0209] Step 1010: Obtain multiple training samples; wherein each training sample includes a first sample speech video and a second sample speech video of the same reference object.

[0210] In both the first and second sample speech videos, the facial area of ​​the reference subject is shown in each frame. Similarly, the reference subject can be a real person, a virtual person, a digital person, etc. The lip movements of the reference subject are different in the same positions shown in the first and second sample speech videos. For example, the first and second sample speech videos could be recorded while the reference subject is speaking different content; the different content could be different content in the same language, or content expressing the same meaning in different languages.

[0211] In some embodiments, the number of video frames in the first sample speech video and the second sample speech video are the same in the same training sample. In some embodiments, if the number of video frames in the first sample speech video and the second sample speech video are different, some video frames in the sample speech video with more video frames can be discarded to ensure that the number of video frames in both is the same.

[0212] Step 1020: Obtain the sample-driven facial key point map sequence; the sample-driven facial key point map sequence is obtained by driving the lip shape key points of the sample facial key point maps corresponding to each video frame in the first sample speech video based on the first sample audio corresponding to the second sample speech video, or it is a sequence formed by the sample facial key point maps corresponding to each video frame in the second sample speech video.

[0213] The first sample audio corresponding to the second sample speech video refers to the audio representing the speech content of the reference object in the second sample speech video, which can be the audio of the reference object synchronously collected during the recording of the second sample speech video.

[0214] In some embodiments, if it is not necessary to train the keypoint reconstruction network in sync with the optical flow network and the face reconstruction network, the sequence of sample face keypoint maps corresponding to each video frame in the second sample speech video can be used as the sample-driven face keypoint map sequence.

[0215] In some embodiments, where it is necessary to train the keypoint reconstruction network synchronously with the optical flow network and the face reconstruction network, the keypoint reconstruction network can perform lip-sync keypoint driving on the sample facial keypoint maps corresponding to each video frame in the first sample speech video based on the first sample audio corresponding to the second sample speech video, thereby obtaining a sample-driven facial keypoint map sequence. Alternatively, in some embodiments, the trained keypoint reconstruction network can perform lip-sync keypoint driving on the sample facial keypoint maps corresponding to each video frame in the first sample speech video based on the first sample audio corresponding to the second sample speech video, thereby obtaining a sample-driven facial keypoint map sequence. This process is basically similar to the implementation process of step 230 above, and will not be described again here.

[0216] Step 1030: The optical flow network performs optical flow estimation based on the sample-driven facial key point map sequence, the sample reference frame selected from the first sample speech video, and the facial key point map of the sample reference frame, to obtain the sample optical flow information sequence.

[0217] In step 1040, the face reconstruction network performs face reconstruction on each video frame in the first sample speech video based on the sample reference frame and the sample optical flow information sequence, thereby obtaining the reconstructed speech video. The implementation processes of steps 1030 and 1040 are similar to those described above and will not be repeated here.

[0218] The reconstructed speech video is a video obtained by the face reconstruction network performing face reconstruction on each video frame in the first sample speech video based on the sample reference frame and the sample optical flow information sequence. It is used to calculate the reconstruction loss to adjust the parameters of the optical flow network and the face reconstruction network.

[0219] Step 1050: Calculate the reconstruction loss based on the reconstructed speech video and the second sample speech video.

[0220] In some embodiments, a first loss can be determined based on the pixel differences between relatively aligned video frames in the reconstructed speech video and the second sample speech video, and the first loss can be used as the reconstruction loss.

[0221] The x-th video frame in the reconstructed speech video and the x-th video frame in the second sample speech video are video frames that are aligned in position, where x is a positive integer of any value.

[0222] First, the pixel difference between two relatively aligned video frames in the reconstructed speech video and the second sample speech video can be calculated using the first loss function. Then, the pixel differences between all relatively aligned video frames are summed to obtain the first loss. The first loss function is used to calculate the pixel difference between two relatively aligned video frames in the reconstructed speech video and the second sample speech video. Specifically, it can be the absolute value loss function, the mean squared error loss function, the cosine loss function, the mean absolute value loss function, the Euclidean distance loss function, etc., without specific limitations here. The first loss can be expressed as: l pix =‖x r -x h || 1 ; (Formula 3)

[0223] x r The video of the speech indicating reconstruction, x h This refers to the second sample speech video, l pix This is the first loss.

[0224] In some embodiments, step 1050 includes: determining a first loss based on the pixel differences between video frames that are aligned in position in the reconstructed speech video and the second sample speech video; determining a second loss based on the depth features of each video frame in the reconstructed speech video and the depth features of each video frame in the second sample speech video; determining a third loss based on the discrimination results of the discriminator network on the second sample speech video and on each video frame in the reconstructed speech video; and weighting at least one of the second loss and the third loss with the first loss to obtain a reconstruction loss.

[0225] In some embodiments, a deep feature extraction network capable of accurately extracting depth features can be introduced to extract depth features from each video frame in the reconstructed speech video and from each video frame in the second sample speech video. Then, a second loss is calculated based on the depth features of two video frames that are aligned in position in the reconstructed speech video and the second sample speech video.

[0226] In some embodiments, the depth feature loss between two relatively aligned video frames in the reconstructed speech video and the second sample speech video can be calculated according to the second loss function. Then, the depth feature losses corresponding to all relatively aligned video frames in the reconstructed speech video and the second sample speech video are added together to obtain the second loss. The second loss function is a function used to calculate the depth feature loss between two relatively aligned video frames in the reconstructed speech video and the second sample speech video. Specifically, it can be an absolute value loss function, a mean squared error loss function, a cosine loss function, a mean absolute value loss function, an Euclidean distance loss function, etc., and is not specifically limited here.

[0227] In some embodiments, the deep feature extraction network may be a VGG network (Visual Geometry Group Network), but is not limited to this. The second loss can be expressed as: l per =‖φ(x r )-φ(x h )‖ 1 ; (Formula 4)

[0228] l per This represents the second loss, where φ is the deep feature extraction network, such as the VGG network, φ(x r ) indicates that the deep feature extraction network is used to reconstruct the speech video x r Depth features extracted from video frames, φ(x) h ) indicates that the deep feature extraction network is used for the second sample speech video x h Depth features extracted from video frames.

[0229] In some embodiments, the third loss can be determined according to the following formula:

[0230] Where D(x) h ) indicates that the discriminator network evaluates the second sample speech video x. h The discrimination results of each video frame; D(x) r ) indicates that the discriminator network reconstructs the speech video x r The discrimination results of each video frame, l adv This is the third loss. The discriminator network is used to determine whether the input image (video frame) is a real image or a generated image, and the discrimination result indicates whether the corresponding video frame is a real recorded video frame or an automatically generated video frame.

[0231] In some embodiments, the first loss and the second loss can be weighted to obtain the reconstruction loss. In other embodiments, the first loss and the third loss can be weighted to obtain the reconstruction loss. In yet another embodiment, the first loss, the second loss, and the third loss can be weighted to obtain the reconstruction loss. The weighting coefficients in the weighting process can all be integers, and the weighting coefficients for different items can be the same or different. For example, the reconstruction loss can be determined according to the following formula 5: l = l pix +l per +l adv ; (Formula 6)

[0232] In the above embodiments, the introduction of a second loss and / or a third loss, instead of focusing solely on the pixel differences between video frames that are aligned in the reconstructed speech video and the second sample speech video, allows the supervised optical flow network and face reconstruction network to learn the high-frequency information of the supervised second sample speech video. This can make the images (video frames) generated by the optical flow network and face reconstruction network after training smoother, and make the high-frequency information of the generated images closer to the high-frequency information of the real images.

[0233] Step 1060: Based on the reconstruction loss, adjust the parameters of at least the optical flow network and the face reconstruction network until the training termination condition is met.

[0234] In some embodiments, if it is not necessary to train the keypoint reconstruction network, optical flow network, and face reconstruction network synchronously, in step 1060, the parameters of the optical flow network and face reconstruction network can be adjusted according to the reconstruction loss using gradient descent. The specific steps are as follows: First, calculate the gradient of the reconstruction loss with respect to the network parameters, for the parameters θ of the optical flow network... flow The parameters θ of the face reconstruction network recon Calculate the gradient respectively and Where L is the reconstruction loss. Then, the parameters are updated according to the learning rate η, as shown in the formula: Repeat the above steps until the training termination condition is met.

[0235] In another embodiment, if it is necessary to train the keypoint reconstruction network, optical flow network, and face reconstruction network synchronously, and the sample-driven face keypoint map sequence is obtained by the keypoint reconstruction network driving the lip shape keypoints of the sample face keypoint maps corresponding to each video frame in the first sample speech video based on the sample audio corresponding to the second sample speech video, the parameters of the optical flow network, face reconstruction network, and keypoint reconstruction network can be adjusted according to the reconstruction loss and in the gradient descent manner.

[0236] Training can end under various conditions, such as reaching a threshold number of iterations or convergence of the reconstruction loss; these conditions are not specified here. Following the above training process ensures that the trained optical flow network accurately estimates optical flow and guarantees the quality of video frames / images obtained by the face reconstruction network.

[0237] In some embodiments, a training sample may include a first sample speech image and a second sample speech image of the same reference object. In this case, the optical flow network and the face reconstruction network can be trained at least according to the following steps 1)-6):

[0238] 1) Obtain multiple training samples; wherein each training sample includes a first sample speech image and a second sample speech image of the same reference object;

[0239] 2) Obtain the sample facial contour; the sample facial contour is obtained by driving the lip shape key points of the sample facial contour corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or it is the facial contour corresponding to the second sample speech image.

[0240] 3) The optical flow network performs optical flow estimation based on the sample face contour, the sample reference face image of the reference object, and the face contour of the sample reference face image to obtain the sample optical flow information;

[0241] 4) The face reconstruction network reconstructs the face of the first sample speech image based on the sample reference face image and sample optical flow information to obtain the sample reconstructed face image;

[0242] 5) Reconstruct the face image from the sample and the speech image from the second sample, and calculate the reconstruction loss.

[0243] Similarly, a first loss can be determined based on the pixel difference between the reconstructed face image and the second speech image; a second loss can be determined based on the depth features of the reconstructed face image and the second speech image; a third loss can be determined based on the discrimination results of the discriminator network on the second speech image and the discrimination results on the reconstructed face image; and at least one of the second and third losses can be weighted with the first loss to obtain the reconstruction loss.

[0244] 6) Based on the reconstruction loss, adjust the parameters of at least the optical flow network and the face reconstruction network until the training termination condition is met.

[0245] The implementation details of 1)-6) above are similar to the embodiment corresponding to Figure 10 above, and will not be repeated here.

[0246] In some embodiments, the keypoint reconstruction network can be trained according to ①-⑧ as follows:

[0247] ① Obtain multiple third-sample speech videos and corresponding sample speech audio for each third-sample speech video; the third-sample speech videos present the facial region of the first reference object;

[0248] ② Based on the number Y of video frames in the third sample speech video, divide the corresponding sample speech audio into Y sample audio units of equal duration. Thus, the lip shape presented in the y-th frame of the third sample speech video is the lip shape when the y-th sample audio unit in the corresponding sample speech audio is pronounced. y is less than or equal to Y, and y is a positive integer. Here, a sample audio unit is an audio unit obtained by dividing the corresponding sample speech audio of the third sample speech video into units of equal duration, and the lip shape presented in the y-th frame of the third sample speech video is the lip shape when the y-th sample audio unit in the corresponding sample speech audio is pronounced.

[0249] ③ Obtain key video frames from the third sample speech video. These key video frames are used to reflect the speaking style of the target subject.

[0250] ④ Facial key points were extracted from each video frame of the third sample speech video to obtain facial key point maps of each video frame of the third sample speech video.

[0251] ⑤ The lip shape key point removal process is performed on the facial key point map corresponding to each video frame in the third sample speech video to obtain the sample candidate key point map corresponding to each video frame in the third sample speech video.

[0252] ⑥ For the y-th video frame in the third sample speech video, the keypoint reconstruction network reconstructs the lip shape keypoints of the candidate keypoint map corresponding to the y-th video frame based on the audio features of the y-th sample audio unit in the sample speech audio and the facial keypoint maps corresponding to the key video frames of multiple samples, and obtains the prediction-driven facial keypoint map corresponding to the y-th video frame; y is a positive integer that takes values ​​sequentially from 1 to Y.

[0253] ⑦ Calculate the lip shape key point reconstruction loss based on the predicted driving facial key point map corresponding to the y-th video frame and the facial key point map of the y-th video frame.

[0254] The lip-shape keypoint reconstruction loss is calculated based on the predicted driving facial keypoint map corresponding to the y-th video frame and the facial keypoint map of the y-th video frame. It is used to adjust the parameters of the keypoint reconstruction network. It is usually obtained by calculating the positional deviation of the corresponding keypoints in the predicted driving facial keypoint map and the facial keypoint map, such as by using the mean squared error loss function.

[0255] The positional deviation of corresponding key points in the predicted facial keypoint map and the facial keypoint map of the y-th video frame can be calculated by, for example, using the mean squared error loss function, as shown in the formula. Where L key This represents the loss for lip-sync keypoint reconstruction, where n is the number of keypoints and p is the number of keypoints.i To predict the position of the i-th keypoint in the driving facial keypoint map, g i Let i be the position of the i-th key point in the facial key point graph of the y-th video frame.

[0256] ⑧ Adjust the parameters of the keypoint reconstruction network based on the lip shape keypoint reconstruction loss until the first training termination condition is met.

[0257] Through the training process described above, the keypoint reconstruction network can accurately drive lip-sync keypoints, ensuring that the lip shape presented in the reconstructed facial keypoints is the same as the lip shape corresponding to the audio applied to the drive. Moreover, it guarantees that the lip shape presented by the reconstructed facial keypoints conforms to the speaker's lip-sync style.

[0258] One approach to addressing the lack of distinctive lip-syncing styles in videos is to train a separate neural network model for each subject, learning the subject's lip-syncing style during training. However, this method requires training a separate model for each subject before it can be used to drive lip-syncing in their videos. This approach demands significant time for training models applicable to different subjects, thus lacking universality.

[0259] In this application, the approach of combining keypoint reconstruction network, optical flow network and face reconstruction network to drive lip movements through audio is applicable to different objects and does not require separate training for different objects. Moreover, it can ensure that the lip movements in the video obtained by driving are the same as the lip movements corresponding to the audio, and it can conform to the speaking lip movement style of the object in the video frame before driving, thus well preserving the speaking lip movement style of the object. Even if it is necessary to drive the speaking video of a new object, it is not necessary to retrain.

[0260] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0261] Figure 11 is a block diagram of an audio-driven face reconstruction device according to an embodiment of this application. As shown in Figure 11, the audio-driven face reconstruction device includes: an acquisition module 1110 for acquiring input audio; a first driving module 1120 for obtaining a target face contour of the target object under the driving action of the input audio based on the input audio and an input face image of the target object, wherein the target face contour includes a contour related to lip shape; an optical flow information determination module 1130 for determining optical flow information based on the target face contour of the target object, a reference face image of the target object, and a reference face contour of the reference face image, wherein the optical flow information is used to characterize the offset information between the reconstructed face image and the reference face image of the target object; and a reconstruction module 1140 for reconstructing facial features of the input face image based on the reference face image of the target object and the optical flow information to obtain a reconstructed face image.

[0262] In some embodiments, the optical flow information determination module 1130 includes: a selection unit, configured to select i reference video frames as reference face images from a sequence of speech video frames of a target object; i is an integer greater than 1; a first acquisition unit, configured to acquire the reference face contour of the i reference video frames; and an optical flow information determination unit, configured to determine optical flow information by an optical flow network based on the target face contour of the target object, the i reference video frames, and the reference face contour of the i reference video frames.

[0263] In some embodiments, in the speech video frame sequence of the target object, selecting i reference video frames as reference facial images includes at least one of the following two selection methods: selecting at least one speech video frame with the most tooth pixels in the speech video frame sequence of the target object; or, when the input facial image comes from the speech video frame sequence, selecting at least one speech video frame in the video frame window containing the input facial image in the speech video frame sequence.

[0264] In some embodiments, the optical flow information determination unit is configured to: when the input face image comes from a speech video frame sequence, acquire the facial contour of at least one adjacent face image adjacent to the input face image; input the target face contour, the facial contour of at least one adjacent face image, the i-frame reference video frame, and the reference face contour of the i-frame reference video frame into the optical flow network to obtain optical flow information.

[0265] In some embodiments, the reconstruction module 1140 includes: an occlusion unit for occluding a lip-related region of an input face image to obtain a lip-occluded image; and a face reconstruction unit for performing face reconstruction by a face reconstruction network based on a reference face image of a target object, optical flow information, and the lip-occluded image to obtain a reconstructed face image.

[0266] In some embodiments, the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder; the face reconstruction unit includes: a first encoding unit, used to encode a reference face image of the target object by the first image encoder to obtain reference visual features; an optical flow mapping unit, used to perform optical flow mapping based on optical flow information and the reference visual features to obtain optical flow fusion features; a fusion encoding unit, used to perform fusion encoding based on image features of a lip-occlusion image and optical flow fusion features by the second image encoder to obtain target visual features; and a decoding unit, used to decode the target visual features by the image decoder to obtain a reconstructed face image.

[0267] In some embodiments, the optical flow information includes the optical flow map and weights of the i-th reference video frame; the reference visual features include the k-th layer sub-reference visual features output by the k-th layer first coding layer of the first image encoder for each reference video frame; the optical flow fusion features include the k-th layer sub-optical flow fusion features; 1≤k≤M; M is the number of first coding layers in the first image encoder; the optical flow mapping unit is used to: perform optical flow mapping between the k-th layer sub-reference visual features of the w-th reference video frame and the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame; 1≤w≤i; w is a positive integer; and perform weighted processing on the k-th layer optical flow mapping features corresponding to the i-th reference video frames according to the weights of the optical flow maps of each reference video frame to obtain the k-th layer sub-optical flow fusion features.

[0268] In some embodiments, the second image encoder includes M second coding layers; a fusion coding unit, configured to: concatenate the k-th layer sub-optical flow fusion feature with the k-th layer coding feature to obtain the k-th layer concatenated feature; the k-th layer coding feature is output by the k-th layer second coding layer in the second image encoder; 1≤k≤M; when k=1, the 1-th layer coding feature is the image feature of the lip-occluded image; if k<M, the k+1-th layer second coding layer in the second image encoder performs fusion coding on the k-th layer concatenated feature to obtain the k+1-th layer coding feature; increment k by 1, and return to execute the step of concatenating the k-th layer sub-optical flow fusion feature with the k-th layer coding feature to obtain the k-th layer concatenated feature; if k=M, use the M-th layer concatenated feature as the target visual feature.

[0269] In some embodiments, the audio-driven face reconstruction device further includes: a first acquisition module for acquiring multiple training samples; wherein each training sample includes a first sample speech image and a second sample speech image of the same reference object; a second acquisition module for acquiring sample face contours; the sample face contours are obtained by performing lip-sync keypoint driving on the sample face contours corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or are the face contours corresponding to the second sample speech image; a first optical flow estimation module for performing optical flow estimation by an optical flow network based on the sample face contours, a sample reference face image of the reference object, and the face contours of the sample reference face image to obtain sample optical flow information; a first face reconstruction module for performing face reconstruction on the first sample speech image by a face reconstruction network based on the sample reference face image and the sample optical flow information to obtain a sample reconstructed face image; a reconstruction loss calculation module for calculating reconstruction loss based on the sample reconstructed face image and the second sample speech image; and an adjustment module for adjusting the parameters of at least the optical flow network and the face reconstruction network based on the reconstruction loss until the training termination condition is met.

[0270] In some embodiments, the reconstruction loss calculation module is configured to: determine a first loss based on the pixel difference between the sample reconstructed face image and the second sample speech image; determine a second loss based on the depth features of the sample reconstructed face image and the depth features of the second sample speech image; determine a third loss based on the discrimination result of the discriminator network on the second sample speech image and the discrimination result on the sample reconstructed face image; and weight at least one of the second loss and the third loss with the first loss to obtain the reconstruction loss.

[0271] In some embodiments, the first driving module 1120 includes: a facial pose key point determination unit, configured to acquire facial pose key points of an input facial image of a target object, wherein the facial pose key points are facial key points unrelated to lip shape; a lip shape key point determination unit, configured to acquire lip shape key points of the target object related to lip shape; and a target facial contour determination unit, configured to obtain the target facial contour of the target object based on the input audio, lip shape key points, and facial pose key points.

[0272] In some embodiments, the lip shape key point determination unit is configured to: select multiple speech images of a target object with different lip shapes; and extract lip shape key points related to the lip shape based on the multiple speech images with different lip shapes.

[0273] In some embodiments, the input facial image is a video of the target object speaking in a first language; the input audio is audio in a second language; the first language and the second language are different.

[0274] Figure 12 shows a schematic diagram of a computer system suitable for implementing the embodiments of this application. It should be noted that the computer system 1200 of the electronic device shown in Figure 12 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application. This electronic device may be a server, terminal, or other device, and can be used to execute the methods provided in this application.

[0275] As shown in Figure 12, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0276] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0277] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0278] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0279] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0280] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0281] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0282] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal. It can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0283] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods of any of the above embodiments.

[0284] In summary, this application provides an audio-driven face reconstruction method, apparatus, device, computer-readable storage medium, and computer program product. By acquiring input audio and combining it with an input face image of the target object, a target face contour of the target object is generated under the driving force of the input audio. This contour includes a lip-shape-related contour. This process utilizes an audio feature extraction model and a facial keypoint extraction model to accurately capture the features of the input audio and the facial keypoints of the input face image, thereby accurately reflecting the lip-shape features of the target object when pronouncing the input audio. Next, optical flow information is determined based on the target face contour, a reference face image of the target object, and the reference face contour. This optical flow information represents the offset information between the reconstructed face image and the reference face image. Through the fusion encoding and decoding operations of the optical flow network, pixel movement between different images can be accurately captured. Finally, facial features are reconstructed from the input face image using the reference face image and the optical flow information to obtain the reconstructed face image. Using the target facial contour as a reconstruction constraint ensures that the lip shape presented in the reconstructed facial image is consistent with the target facial contour, that is, the same as the lip shape when the target object pronounces the input audio. This avoids the problem of poor lip shape effect after audio driving in related technologies, improves the accuracy and realism of the reconstructed facial image, and achieves more accurate audio-driven facial reconstruction in terms of technology. It also improves resource utilization and reduces unnecessary calculations and errors.

[0285] Furthermore, when determining optical flow information, i reference video frames are selected from the target object's speech video frame sequence as reference facial images, where i is an integer greater than 1. The reference facial contour of the i reference video frames is obtained, and then the optical flow network determines the optical flow information based on the target facial contour, the i reference video frames, and the reference facial contour of the i reference video frames. Multiple reference video frames provide rich facial feature information. Different reference video frames contain facial features from different angles and expressions. The optical flow network can learn more comprehensive optical flow patterns from this information, improving the accuracy and stability of optical flow estimation, reducing errors in the reconstruction process, and enhancing the robustness of the system.

[0286] Furthermore, when selecting an i-frame reference video image as the reference face image, at least one of the following two selection methods is used: First, select at least one video frame with the most tooth pixels from the target object's speech video frame sequence. Second, if the input face image comes from the speech video frame sequence, select at least one video frame from the video frame window containing the input face image in the speech video frame sequence. Selecting a video frame with many tooth pixels provides rich tooth detail information, which is an important feature of face reconstruction, making the reconstructed face image more realistic. Selecting a video frame from the video frame window containing the input face image ensures the temporal and content-related relevance between the reference video frame and the input face image, making the reference video frame more consistent with the actual situation of the current input face image, further improving the accuracy of optical flow information and the quality of the reconstructed face image, and enhancing the detail and visual effect of the reconstructed image.

[0287] Furthermore, when the input face image comes from a sequence of speech video frames, the facial contours of at least one adjacent face image are obtained. The target face contour, the facial contours of at least one adjacent face image, the i-frame reference video frame, and the reference face contour of the i-frame reference video frame are input into the optical flow network to obtain optical flow information. The facial contours of adjacent face images reflect the continuous temporal changes of the face; introducing this information as an aid ensures the temporal coherence of the optical flow information. In actual face reconstruction, facial movement is continuous. The coherence of the optical flow information makes the reconstructed face image smoother in the temporal dimension, avoiding obvious jitter or discontinuities, improving the smoothness and naturalness of the reconstructed face image, and enhancing the user experience.

[0288] Furthermore, when reconstructing facial features from the input face image, the lip-related regions of the input face image are first occluded to obtain a lip-occluded image. Then, the face reconstruction network performs face reconstruction based on the reference face image of the target object, optical flow information, and the lip-occluded image to obtain the reconstructed face image. Occluding the lip-related regions clarifies the target region that needs to be reconstructed and reduces unnecessary interference information. The face reconstruction network can concentrate more computational resources and attention on the lip region that needs to be reconstructed, avoiding the influence of other regions, thereby improving the efficiency and accuracy of reconstruction, making the reconstructed lip shape more consistent with the requirements of the target face contour, and optimizing the allocation of computational resources.

[0289] Furthermore, when the face reconstruction network includes a first image encoder, a second image encoder, and an image decoder, the first image encoder encodes the reference face image of the target object to obtain reference visual features. Optical flow mapping is then performed between the reference visual features and optical flow information to obtain optical flow fusion features. The second image encoder performs fusion encoding based on the image features of the lip-occluded image and the optical flow fusion features to obtain the target visual features. Finally, the image decoder decodes the target visual features to obtain the reconstructed face image. The first image encoder encodes the reference face image, extracting its deep features. The optical flow mapping process combines optical flow information with the reference visual features, ensuring that the obtained optical flow fusion features contain both motion and feature information. The second image encoder fuses and encodes the image features of the lip-occluded image and the optical flow fusion features, further integrating multiple pieces of information to make the target visual features richer and more accurate. The image decoder decodes the visual features of the target and finally generates a reconstructed face image. This layered encoding and decoding method fully extracts and integrates a variety of information, improves the quality and realism of the reconstructed face image, makes the reconstructed image closer to the real face image, and enhances the information processing capability of the system.

[0290] Furthermore, the optical flow information includes the optical flow map and weights of the i-th reference video frame, the reference visual features include the k-th layer sub-reference visual features output by the k-th layer of the first encoding layer in the first image encoder for each reference video frame, and the optical flow fusion features include the k-th layer sub-optical flow fusion features. The k-th layer sub-reference visual features of the w-th reference video frame are optically mapped to the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame. Then, based on the weights of the optical flow maps of each reference video frame, the k-th layer optical flow mapping features corresponding to the i-th reference video frame are weighted to obtain the k-th layer sub-optical flow fusion features. The weights of the optical flow map reflect its reliability. The weighted processing method based on the weights reasonably fuses the features of different reference video frames according to the reliability of the optical flow map. Features corresponding to optical flow maps with high reliability are given greater weight in the fusion process, highlighting the role of these features, improving the accuracy and effectiveness of the optical flow fusion features, thereby improving the quality of the reconstructed face image and optimizing the information fusion process.

[0291] Furthermore, the second image encoder includes M second coding layers. The k-th layer sub-optical flow fusion feature is concatenated with the k-th layer coding feature to obtain the k-th layer concatenated feature. The k-th layer coding feature is output by the k-th layer second coding layer in the second image encoder. When k = 1, the 1-th layer coding feature is the image feature of the lip-occluded image. If k < M, the (k+1)-th layer second coding layer in the second image encoder performs fusion coding on the k-th layer concatenated feature to obtain the (k+1)-th layer coding feature. k is incremented by 1, and the concatenation step is returned. If k = M, the M-th layer concatenated feature is used as the target visual feature. This layer-by-layer concatenation and fusion coding method fully utilizes the features output from different layers in the first image encoder. The features extracted by different coding layers have different levels of abstraction and detailed information. Through layer-by-layer concatenation and fusion, these features at different levels are integrated, avoiding the information loss caused by using only the last layer feature. This allows the target visual features to contain richer and more accurate information, thereby improving the quality and detail of the reconstructed facial images, making the reconstructed images clearer and more realistic, and improving the efficiency of feature utilization.

[0292] Furthermore, before performing face reconstruction using the face reconstruction network, multiple training samples are acquired. Each training sample includes a first sample speech image and a second sample speech image of the same reference subject. Sample facial contours are obtained by performing lip-sync keypoint analysis on the sample facial contour corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or by using the facial contour corresponding to the second sample speech image. An optical flow network performs optical flow estimation based on the sample facial contours, a sample reference face image of the reference subject, and the facial contours of the sample reference face image to obtain sample optical flow information. The face reconstruction network then performs face reconstruction on the first sample speech image based on the sample reference face image and the sample optical flow information, obtaining a reconstructed face image. Reconstruction loss is calculated based on the reconstructed face image and the second sample speech image. The parameters of the optical flow network and the face reconstruction network are then adjusted based on the reconstruction loss until the training termination condition is met. Through training with a large number of training samples, the optical flow network and the face reconstruction network can learn the features and patterns between different samples. The reconstruction loss measures the difference between the network's output reconstructed face image and the real second sample speech image. Adjusting network parameters based on reconstruction loss enables the network to continuously optimize its performance, improve its generalization ability, and accurately perform optical flow estimation and face reconstruction when faced with different inputs, resulting in higher quality reconstructed face images and enhancing the system's adaptability and stability.

[0293] Furthermore, when calculating the reconstruction loss, a first loss is determined based on the pixel differences between the reconstructed face image and the second sample speech image; a second loss is determined based on the depth features of the reconstructed face image and the second sample speech image; and a third loss is determined based on the discriminator network's discrimination results for the second sample speech image and the reconstructed face image. At least one of the second and third losses is weighted with the first loss to obtain the reconstruction loss. Pixel differences reflect the similarity of images on their surfaces, depth features reflect the three-dimensional structural information of the image, and the discriminator network's discrimination results determine the realism of the image. Introducing depth features and the discriminator network's discrimination results allows for multi-dimensional supervision of the learning process of the optical flow network and the face reconstruction network, enabling the network to not only focus on pixel-level differences but also learn the image's depth and realism features. The reconstruction loss obtained through weighted processing provides a more comprehensive measure of the network's performance, allowing the network to continuously optimize during training, improving the quality and realism of the reconstructed face image, making the reconstructed image closer to the real image, and enhancing the effectiveness of network training.

[0294] Furthermore, when obtaining the target facial contour of the target object, the facial pose key points of the input facial image of the target object are first acquired. These facial pose key points are independent of lip shape. Then, the lip shape key points of the target object are acquired. Finally, based on the input audio, lip shape key points, and facial pose key points, the target facial contour of the target object is obtained. Acquiring facial pose key points and lip shape key points separately allows for the separation and processing of different facial features, making the reconstruction process more refined. Facial pose key points ensure the stability of the poses of other parts of the face during reconstruction, while lip shape key points accurately adjust the lip shape according to the input audio. During reconstruction, this ensures consistency between the lip shape and the input audio while maintaining the stability of other facial poses, improving the accuracy and rationality of the target facial contour, making the reconstructed facial image more natural and realistic, and optimizing the facial feature processing workflow.

[0295] Furthermore, when acquiring lip-shape key points related to the target subject's mouth shape, multiple speech images of the target subject with different mouth shapes are selected. Based on these multiple speech images with different mouth shapes, lip-shape key points related to the mouth shape are extracted. Multiple speech images with different mouth shapes can cover the changes in the target subject's mouth shape under different pronunciation situations, and the lip-shape key points extracted from them are more representative and comprehensive. These key points can more accurately reflect the lip shape features of the target subject, which helps to obtain the target's facial contour more accurately in subsequent steps. This makes the lip shape presented in the reconstructed facial image more consistent with the actual lip shape style of the target subject, improves the accuracy and personalization of the reconstructed image, and enhances the precision of lip shape feature extraction.

[0296] Furthermore, when the input facial image comes from a speech video in the target object's first language, and the input audio is in a second language, and the first and second languages ​​are different, this method can still achieve audio-driven image processing, ensuring that the lip movements in the reconstructed facial image match the lip movements of the target object as they pronounce the input audio. This demonstrates the method's good adaptability in cross-language scenarios, and its core audio-driven and facial reconstruction mechanisms are not limited by language. By processing the input audio and input facial image, the method can accurately capture the lip movement features of different languages, achieving cross-language audio-driven facial reconstruction. This expands the method's application scope, improves its versatility and practicality, provides an effective solution for facial reconstruction in multilingual environments, and enhances the system's cross-language processing capabilities.

[0297] The manual also mentions that when training audio feature extraction models (such as the Whisper model and the WAV2VEC model), self-supervised learning and contrastive learning methods are used to allow the model to learn the feature representation of audio. This training method utilizes a large amount of audio data and corresponding text labels, maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, enabling the model to extract audio features more accurately. In practical applications, more accurate audio feature extraction helps to generate target facial contours more precisely, improving the accuracy and reliability of the entire face reconstruction process and enhancing the system's ability to process audio information.

[0298] Meanwhile, when training the facial keypoint extraction model, a convolutional neural network (CNN) was built, using a large dataset of facial images with labeled keypoints for training. A loss function was used to measure the difference between predicted and ground truth keypoints. Backpropagation and optimization algorithms were employed to update model parameters, and data augmentation techniques were used to increase data diversity. These measures enabled the facial keypoint extraction model to extract facial keypoints more accurately, providing a more reliable foundation for subsequent lip-shape keypoint-driven facial reconstruction, improving the accuracy and robustness of facial feature extraction, and optimizing the front-end processing workflow for facial reconstruction.

[0299] When training a keypoint reconstruction network, setting appropriate hyperparameters, such as learning rate, number of training epochs, and batch size, and employing an early stopping strategy to prevent overfitting are crucial. Appropriate hyperparameter settings enable the network to converge faster during training, improving training efficiency. Simultaneously, the early stopping strategy prevents the network from overlearning the training data, enhancing its generalization ability. In practical applications, a well-trained keypoint reconstruction network can more accurately drive lip-sync, making the reconstructed facial contours more consistent with the input audio and the target subject's lip-sync style, thus improving the accuracy and stability of lip-sync driving.

[0300] Before training the optical flow network, the input data is preprocessed, with its size adjusted and normalized. During training, a stochastic gradient descent optimization strategy is employed, along with appropriate learning rates and decay mechanisms, and a dropout layer is added to prevent overfitting. These preprocessing steps ensure the input data better meets the network's requirements, and the optimization strategy and overfitting prevention measures enable the optical flow network to estimate optical flow more accurately, yielding more precise optical flow information. Accurate optical flow information is crucial for subsequent face reconstruction, improving the quality and accuracy of reconstructed face images and enhancing the stability and reliability of optical flow estimation.

[0301] In training the face reconstruction network, in addition to using a loss function, regularization methods (such as L2 regularization) and batch normalization techniques are also employed. Regularization methods limit the size of model parameters, preventing overfitting, while batch normalization accelerates model convergence and improves training stability. These measures enable the face reconstruction network to reconstruct faces more accurately, generating higher-quality reconstructed face images, thus improving the effectiveness and quality of face reconstruction and enhancing the network's generalization ability and training efficiency.

[0302] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0303] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0304] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0305] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An audio-driven face reconstruction method, performed by an electronic device, comprising: Get the input audio; Based on the input audio and the input facial image of the target object, the target facial contour of the target object is obtained under the driving action of the input audio, and the target facial contour includes a contour related to the mouth shape; Optical flow information is determined based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object. and Based on the reference facial image of the target object and the optical flow information, facial feature reconstruction is performed on the input facial image to obtain the reconstructed facial image.

2. The method according to claim 1, wherein determining optical flow information based on the target facial contour of the target object, a reference facial image of the target object, and a reference facial contour of the reference facial image comprises: In the video frame sequence of the target object's speech, frame i is selected as the reference face image. i is an integer greater than 1; Obtain the reference facial contour of the reference video frame i; The optical flow network determines the optical flow information based on the target facial contour of the target object, the i-frame of the reference video frame, and the reference facial contour of the i-frame of the reference video frame.

3. The method according to claim 2, wherein selecting i-th reference video frame as the reference face image from the speech video frame sequence of the target object includes at least one of the following two selection methods: In the speech video frame sequence of the target object, at least one speech video frame with the highest number of tooth pixels is selected; When the input face image comes from the speech video frame sequence, at least one speech video frame is selected from the video frame window in the speech video frame sequence that contains the input face image.

4. The method according to claim 2 or 3, wherein determining the optical flow information by the optical flow network based on the target facial contour of the target object, the i-th frame of the reference video frame, and the reference facial contour of the i-th frame of the reference video frame includes: When the input face image comes from the speech video frame sequence, obtain the facial contour of at least one adjacent face image that is adjacent to the input face image; The target facial contour, the facial contour of at least one adjacent facial image, the i-th frame of the reference video frame, and the reference facial contour of the i-th frame of the reference video frame are input into the optical flow network to obtain optical flow information.

5. The method according to any one of claims 2 to 4, wherein reconstructing facial features from the input facial image based on a reference facial image of the target object and the optical flow information to obtain the reconstructed facial image comprises: The lip-related region of the input face image is occluded to obtain a lip-occluded image; The face reconstruction network performs face reconstruction based on the reference face image of the target object, the optical flow information, and the lip occlusion image to obtain the reconstructed face image.

6. The method according to claim 5, wherein the face reconstruction network comprises a first image encoder, a second image encoder, and an image decoder; The face reconstruction network performs face reconstruction based on the reference face image of the target object, the optical flow information, and the lip occlusion image to obtain the reconstructed face image, including: The reference facial image of the target object is encoded by the first image encoder to obtain reference visual features; Optical flow mapping is performed based on the optical flow information and the reference visual features to obtain optical flow fusion features; The second image encoder performs fusion encoding based on the image features of the lip-occlusion image and the optical flow fusion features to obtain the target visual features; The image decoder decodes the visual features of the target to obtain the reconstructed face image.

7. The method according to claim 6, wherein the optical flow information includes the optical flow graph of the i-th reference video frame and the weights of the optical flow graph; the reference visual feature includes the k-th layer sub-reference visual feature output by the k-th layer first coding layer of the first image encoder for each reference video frame; the optical flow fusion feature includes the k-th layer sub-optical flow fusion feature; 1≤k≤M; M is the number of first coding layers in the first image encoder; The step of performing optical flow mapping based on the optical flow information and the reference visual features to obtain optical flow fusion features includes: Optical flow mapping is performed between the k-th layer sub-reference visual features of the w-th reference video frame and the optical flow map of the w-th reference video frame to obtain the k-th layer optical flow mapping features corresponding to the w-th reference video frame; 1≤w≤i; w is a positive integer; Based on the weights of the optical flow maps of each reference video frame, the k-th layer optical flow mapping features corresponding to the i-th frame and the reference video frame are weighted to obtain the k-th layer sub-optical flow fusion features.

8. The method according to claim 7, wherein the second image encoder comprises an M-layer second coding layer; The step of fusing and encoding the target visual features by the second image encoder based on the image features of the lip-occlusion image and the optical flow fusion features includes: The k-th layer sub-optical flow fusion feature is concatenated with the k-th layer coding feature to obtain the k-th layer concatenated feature; The k-th layer coding feature is output by the k-th second coding layer in the second image encoder; 1≤k≤M; when k=1, the 1-th layer coding feature is the image feature of the lip-occlusion image; If k < M, the k+1th layer of the second encoding layer in the second image encoder is used to fuse and encode the k-th layer splicing features to obtain the k+1th layer encoded features. Increment k by 1 and return to the step of concatenating the k-th layer sub-optical flow fusion feature with the k-th layer coding feature to obtain the k-th layer concatenated feature; If k = M, the Mth layer stitched features are used as the target visual features.

9. The method according to any one of claims 5 to 8, wherein before the face reconstruction network performs face reconstruction based on a reference face image of the target object, the optical flow information, and the lip-shaped occlusion image to obtain the reconstructed face image, the method further comprises: Multiple training samples are acquired; wherein each training sample includes a first sample speech image and a second sample speech image of the same reference object; Obtain the sample facial contour; the sample facial contour is obtained by driving the lip shape key points of the sample facial contour corresponding to the first sample speech image based on the sample audio corresponding to the second sample speech image, or it is the facial contour corresponding to the second sample speech image. The optical flow network performs optical flow estimation based on the sample face contour, the sample reference face image of the reference object, and the face contour of the sample reference face image to obtain sample optical flow information; The face reconstruction network performs face reconstruction on the first sample speech image based on the sample reference face image and the sample optical flow information to obtain the sample reconstructed face image; Reconstruct the facial image based on the given sample and the speech image based on the second sample, and calculate the reconstruction loss; Based on the reconstruction loss, at least the parameters of the optical flow network and the face reconstruction network are adjusted until the training termination condition is met.

10. The method according to claim 9, wherein the step of reconstructing the facial image based on the sample and calculating the reconstruction loss based on the second sample speech image comprises: The first loss is determined based on the pixel differences between the reconstructed facial image from the sample and the second sample speech image; The second loss is determined by reconstructing the depth features of the facial image from the sample and the depth features of the speech image from the second sample; The third loss is determined based on the discrimination results of the discriminator network on the second sample speech image and the discrimination results on the reconstructed face image of the sample; The reconstruction loss is obtained by weighting at least one of the second loss and the third loss with the first loss.

11. The method according to any one of claims 1 to 10, wherein obtaining the target facial contour of the target object under the driving action of the input audio based on the input audio and the input facial image of the target object comprises: Obtain the facial pose key points of the input facial image of the target object, wherein the facial pose key points are facial key points that are independent of the mouth shape; Obtain the key points of the mouth shape of the target object related to the mouth shape; Based on the input audio, lip shape key points, and facial pose key points, the target facial contour of the target object is obtained.

12. The method according to claim 11, wherein obtaining the key lip-shape points of the target object related to the lip shape includes: Select multiple images of the target object speaking with different lip movements; Based on the multiple images of speech with different mouth shapes, extract the key points of the mouth shape related to the mouth shape.

13. The method according to any one of claims 1 to 12, wherein the input facial image is a video of the target object speaking in a first language; the input audio is audio in a second language; and the first language is different from the second language.

14. An audio-driven face reconstruction device, comprising: The acquisition module is used to acquire input audio. The first driving module is used to obtain the target facial contour of the target object under the driving action of the input audio based on the input audio and the input facial image of the target object, wherein the target facial contour includes a contour related to the mouth shape; The optical flow information determination module is used to determine optical flow information based on the target facial contour of the target object, the reference facial image of the target object, and the reference facial contour of the reference facial image. The optical flow information is used to characterize the offset information between the reconstructed facial image and the reference facial image of the target object. and The reconstruction module is used to reconstruct facial features from the input facial image based on the reference facial image of the target object and the optical flow information, so as to obtain the reconstructed facial image.

15. An electronic device comprising: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-13.

16. A computer-readable storage medium having stored thereon computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-13.

17. A computer program product comprising computer instructions that, when executed by a processor, implement the method of any one of claims 1-13.

Images