Apparatus and method for generating speech video

Abstract

Disclosed are an apparatus and a method for generating a speech video. The apparatus for generating a speech video, according to one embodiment, comprises: an encoder for encoding a person image so as to generate an identity latent vector; a motion latent vector generation model which is a flow matching-based probabilistic generative model for generating a motion latent vector on the basis of a speech audio; and a decoder for generating a speech video of the corresponding person uttering, on the basis of the identity latent vector and the motion latent vector.

Description

Device and method for generating speech video

[0001] It is related to AI-based speech video generation technology.

[0002] With the advancement of artificial intelligence technology, the technology to generate a speech video in which the subject of a given portrait speaks naturally in sync with the audio, using just a single portrait photo and spoken audio, is also making a significant leap forward.

[0003] Key factors determining the naturalness of speech video generation include the generation of lip movements synchronized with the speech audio and the generation of non-speech motion (facial expressions, head movements, etc.) that matches the speech audio. While the former (synchronized lip movements) has recently made significant advancements in terms of accuracy and variety, the latter (generation of non-speech motion that matches the speech audio) remains a challenging problem because the inherently weak correlation between spoken speech and non-speech motion makes it difficult for AI to directly infer this relationship.

[0004] The purpose is to provide a device and method for generating speech videos based on artificial intelligence.

[0005] A speech video generation device according to one aspect may include: an encoder that encodes a person image to generate an identity latent vector; a motion latent vector generation model which is a flow matching-based probabilistic generation model that generates a motion latent vector based on speech audio; and a decoder that generates a speech video in which the person speaks based on the identity latent vector and the motion latent vector.

[0006] The above encoder is an encoder of a motion autoencoder, and the above decoder may be a decoder of the above motion autoencoder.

[0007] The motion autoencoder above can be pre-trained by randomly extracting two frames from a training speech video and minimizing the error loss between the frame generated using the first frame among the two frames as input and the second frame among the two frame images.

[0008] The method further includes a combination unit that generates a combination potential vector by combining the identity potential vector and the motion potential vector, and the decoder can generate the speech video by decoding the combination potential vector.

[0009] The motion latent vector generation model may include: a generation vector field estimation model that estimates a generation vector field for generating the motion latent vector from a noise latent vector based on the speech audio; and a motion latent vector generation unit that generates a motion latent vector corresponding to the speech audio based on the generation vector field.

[0010] The motion potential vector generation unit above can generate the motion potential vector by obtaining a numerical solution of an ordinary differential equation including the generated vector field.

[0011] The above ordinary differential equation can be expressed by the following mathematical formula.

[0012] [Mathematical Formula]

[0013]

[0014] (Here, x0 is a noise latent vector extracted from an arbitrary distribution, and Φ t is a function of time t that transforms x0 into a motion latent vector, and A s is the spoken audio, and θ * is the optimized parameter of the generating vector field estimation model, and v t is the generated vector field estimated by the generated vector field estimation model)

[0015] The above motion latent vector generation model can be trained to estimate a generation vector field that generates a motion latent vector similar to the training motion latent vector from a noise latent vector based on training speech audio.

[0016] The above motion potential vector may include mouth shape and non-speech motion information synchronized with the speech audio.

[0017] A method for generating a speech video performed by a computing device according to a different aspect may include: a step of encoding a person image to generate an identity latent vector; a step of generating a motion latent vector based on speech audio using a motion latent vector generation model, which is a flow matching-based probabilistic generation model; and a step of generating a speech video in which the person speaks based on the identity latent vector and the motion latent vector.

[0018] According to an exemplary embodiment, by using a flow matching-based motion latent vector generation model to generate motion latent vectors corresponding to the speech audio from the speech audio and utilizing them for speech video generation, a speech video can be generated in which the person speaks the content of the speech audio with natural motion.

[0019] FIG. 1 is a drawing illustrating a speech video generating device according to an exemplary embodiment.

[0020] FIG. 2 is a diagram illustrating a motion latent vector generation model according to an exemplary embodiment.

[0021] FIG. 3 is a diagram illustrating a learning method of a motion autoencoder according to an exemplary embodiment.

[0022] FIG. 4 is a flowchart illustrating a method for generating a speech video according to an exemplary embodiment.

[0023] FIG. 5 is a block diagram illustrating a computing environment including a computing device according to an exemplary embodiment.

[0024] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the attached drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description will be omitted.

[0025] The terms described below are defined in consideration of their functions within this disclosure and may vary depending on the intent or practices of the user or operator. Therefore, their definitions should be based on the content throughout this disclosure.

[0026] Terms such as "first," "second," etc., may be used to describe various components, but components should not be limited by these terms. Terms are used solely for the purpose of distinguishing one component from another. A singular expression includes a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0027] Furthermore, the classification of components in this disclosure is merely based on the primary function performed by each component. That is, two or more components may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Additionally, each component may perform some or all of the functions performed by other components in addition to its primary function, and some of the primary functions performed by each component may be exclusively performed by other components. Each component may be implemented in hardware or software, or as a combination of hardware and software.

[0028] FIG. 1 is a drawing illustrating a speech video generating device according to an exemplary embodiment, and FIG. 2 is a drawing illustrating a motion latent vector generating model according to an exemplary embodiment.

[0029] A speech video generation device (100) according to an exemplary embodiment may be an audio-driven speech video generation device based on a flow matching probabilistic generation model. The speech video generation device (100) may convert a person image of a predetermined person into an identity latent vector through a pre-trained motion autoencoder based on an artificial neural network (which may be referred to as a motion latent vector autoencoder or motion latent autoencoder), and may probabilistically generate motion latent vectors containing mouth shapes and non-speechless motions synchronized with a given speech audio (or speech voice) using a flow matching-based motion latent vector generation model. The speech video generation device (100) may generate a speech video by converting the generated motion latent vectors and identity latent vectors into frame images through a motion autoencoder. Here, the speech video may be referred to as a talking portrait video.

[0030] Referring to FIG. 1, a speech image generating device (100) according to an exemplary embodiment may include an encoder (110), a motion latent vector generating model (120), a combination unit (130), and a decoder (140). Here, the encoder (110) and the decoder (140) may constitute a motion autoencoder. For example, the encoder (110) may be an encoder that constitutes a motion autoencoder, and the decoder (140) may be a decoder that constitutes a motion autoencoder.

[0031] According to one embodiment, the motion autoencoder, that is, the encoder (110) and the decoder (140), can be pre-trained. The training method of the encoder (110) and the decoder (140) will be described later with reference to FIG. 3.

[0032] The encoder (110) is a person image (or portrait image) (I s Encode the person image (I s The latent vector corresponding to ) (w s ) can be generated. According to one embodiment, a latent vector (w s ) is the identity latent vector (w i ) and motion latent vector(w m It may include ). Here, the identity latent vector (w i ) includes features related to the identity of a specified person, and the motion latent vector (w m ) may include features related to the motion of a specific person. The encoder (110) encodes a person image of a specific person and an identity latent vector (w) corresponding to the person image. i ) and motion latent vector(w m Can generate ).

[0033] Motion potential vector (w) generated from the encoder (110) m Identity latent vector (w excluding ) i ) can be used when creating a video of the person's speech.

[0034] The motion latent vector generation model (120) is a speech audio (or speech voice) (A s Based on ), motion latent vector(w g_m ) can be generated probabilistically. The motion latent vector (w generated at this time) g_m ) is speech audio (A s Information about mouth shapes and non-speech motions synchronized with ) may be included.

[0035] According to an exemplary embodiment, the motion latent vector generation model (120) may be a probabilistic generation model, such as a flow matching-based probabilistic generation model learned through flow matching.

[0036] Referring to FIG. 2, the motion latent vector generation model (120) may include a generation vector field estimation model (210) and a motion latent vector generation unit (220).

[0037] The generation vector field estimation model (220) is speech audio (A s Based on ), from a noisy latent vector extracted from an arbitrary distribution, e.g., a multidimensional standard normal distribution, a motion latent vector (w g_m It is possible to estimate the generating vector field that generates ).

[0038] To this end, the generating vector field estimation model (210) can be trained to estimate a generating vector field that generates a motion latent vector similar to a learning motion latent vector from a noise latent vector based on the learning speech audio. At this time, the learning motion latent vector can be derived from frames of a learning speech video corresponding to the learning speech audio. For example, a learning motion latent vector can be generated by extracting a plurality of consecutive frames corresponding to the learning speech audio from the learning speech video and encoding each of the extracted plurality of frames through a pre-trained motion autoencoder.

[0039] Specifically, if the noise latent vector is x0 and the learning motion latent vector is x1, the flow can be defined as a function Φ that transforms the learning noise latent vector x0 into the learning motion latent vector x1.

[0040] Flow-matching-based generative models treat these flows as functions dependent on time t ∈ [0, 1], in which case the flow (Φ) for time t t ) can be described as an ordinary differential equation (ODE) of mathematical equation 1.

[0041]

[0042] Here, u constituting mathematical equation 1 t can be defined as a generating vector field. That is, the generating vector field can be defined as the derivative of the flow with respect to time.

[0043] Target generation vector field (u) to be learned via flow matching t If ) is defined as a constant velocity vector field as in Equation 2, then flow(Φ t ) can be expressed by Equation 3, which is a simple linear transformation with respect to time t.

[0044]

[0045]

[0046] The generation vector field estimation model (210) can be trained to estimate a generation vector field based on a training speech audio (A) and a training motion latent vector (x1), and then to reduce the error loss between the estimated generation vector field and the target generation vector field. For example, the optimized parameter (θ) of the generation vector field estimation model (210) * ) can be expressed by mathematical formula 4.

[0047]

[0048] Here, x1 is the training motion latent vector, A is the training utterance audio, θ is the training parameter, and v t is the generated vector field estimated by the generated vector field estimation model, and u t is the target generated vector field expressed by mathematical equation 2, and Φ t can be a flow expressed by mathematical equation 3. argmin can represent a function that finds θ which minimizes the error loss between the estimated generation vector field and the target generation vector field.

[0049] The motion latent vector generation unit (220) generates speech audio (A) from the generation vector field estimation model (210). s Based on the generated vector field estimated using ) and the noise latent vector, speech audio (A s Motion latent vector (w) corresponding to ) g_m Can generate ).

[0050] The generated vector field (v) estimated in the generated vector field estimation model (210) t ) target vector field(u t If substituted into the ordinary differential equation of Equation 1 instead, Equation 1 can be expressed as Equation 5.

[0051]

[0052] The motion potential vector generation unit (220) obtains a numerical solution to the ordinary differential equation of Equation 5, thereby producing speech audio (A s Using ) and noise potential vector (x0), motion potential vector (Φ1(x0))(w g_m ) can be generated. For example, the motion potential vector generation unit (220) can generate speech audio (A s Multiple motion potential vectors (w) corresponding to the length of ) g_m Can generate ).

[0053] According to one embodiment, the motion potential vector generation unit (220) may use the Euler method or the Runge-Kutta method as an algorithm for obtaining a numerical solution to an ordinary differential equation, but is not limited thereto.

[0054] The combination unit (130) is an identity latent vector (w) generated by the encoder (110). i ) and the motion potential vector (w) generated in the motion potential vector generation unit (220). g_m Combining ) to create a combination latent vector (w g_s ) can be generated. For example, the combination part (130) can generate an identity latent vector (w i ) and each motion potential vector(w g_m By combining ) speech audio (A s Multiple combination latent vectors (w) corresponding to the length of ) g_s Can generate ).

[0055] The decoder (140) is a combination potential vector (w g_s Decoding ) to produce a speech video (V) in which a specific person speaks g ) can be generated. For example, the decoder (140) can generate each combination potential vector (w g_s Decoding ) to utterance image (I) corresponding to each combinational latent vector g Generate ) and each generated utterance image (I g Connect the utterance video (V g Can generate ).

[0056] According to an exemplary embodiment, the speech video generating device (100) can generate an identity latent vector containing information about the identity of a person from a person image, and generate a motion latent vector corresponding to the speech audio from the speech audio using a flow matching-based motion latent vector generating model. The speech video generating device (100) can generate a speech video in which the person speaks the content of the speech audio in natural motion using the identity latent vector generated from the person image and the motion latent vector generated from the speech audio using a flow matching-based motion latent vector generating model.

[0057] FIG. 3 is a diagram illustrating a learning method of a motion autoencoder according to an exemplary embodiment.

[0058] As described above, the motion autoencoder may include an encoder (110) and a decoder (140). The motion autoencoder may convert an image into a latent vector corresponding to the image. As described above, the latent vector may be expressed as the sum of an identity vector and a motion vector.

[0059] The reason a motion autoencoder can encode in the form of a latent vector, which is the sum of an identity vector and a motion vector, may be due to the learning method of the motion autoencoder. According to one embodiment, the motion autoencoder can be trained as follows.

[0060] First, two frames (videos) are randomly extracted from a training speech video of a specific person. For example, two frames can be randomly extracted from a training speech video of a specific person as a source frame and a driving frame.

[0061] Next, one of the two extracted frames (x s A frame (x) generated by inputting )(hereinafter, source frame) into a motion autoencoder g)(hereinafter, generated frame) and the other of the two frames (x d A motion autoencoder is trained so that the error loss of )(hereinafter, driving frame) is minimized.

[0062] For example, the encoder (110) receives an input source frame (x s Encoding the source frame (x s Generates a latent representation (w) of ), and the decoder (140) generates a source frame (x s Decoding the latent representation (w) of ) to generate a frame (x g ) can be generated. The encoder (110) and decoder (140) generate a frame (x g ) and driving frame(x d It can be trained so that the error loss of ) is minimized. This can be expressed mathematically as Equation 6.

[0063]

[0064] Here may be an error loss function. According to one embodiment, the error loss function may include, but is not limited to, an L1 loss function, a perceptual loss function, etc.

[0065] According to one embodiment, a source frame (x s ) and driving frame(x d Since ) is extracted from the same speech video and used for training, the motion autoencoder can be trained to include all motion (e.g., mouth shape, facial expression, head movement, etc.) in the latent vector to generate a driving frame from the source frame.

[0066] FIG. 4 is a flowchart illustrating a method for generating a speech video according to an exemplary embodiment. The speech video generation method of FIG. 4 can be performed by the speech video generation device (100) of FIG. 1. Although the method is described in the illustrated flowchart as being divided into a plurality of steps, at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed with one or more steps not illustrated added.

[0067] Referring to FIG. 4, in step 410, the speech video generating device can generate an identity latent vector corresponding to the person image by encoding a person image of a predetermined person. Here, the identity latent vector may include features related to the identity of the predetermined person. For example, the speech video generating device can generate an identity latent vector corresponding to the person image by using an encoder of a pre-trained motion autoencoder.

[0068] In step 420, the speech video generating device can probabilistically generate motion latent vectors based on speech audio. The generated motion latent vectors may include information about mouth shapes and non-speech motions synchronized with the speech audio.

[0069] According to an exemplary embodiment, a speech video generating device can generate motion latent vectors using a probabilistic generating model, such as a flow matching-based probabilistic generating model trained with flow matching.

[0070] For example, a speech video generation device uses a pre-trained generation vector field estimation model, which is a flow matching-based generation model, to produce speech audio (A sBased on ), a generating vector field that generates a motion potential vector from a noise potential vector can be estimated. In addition, the speech video generating device can generate a motion potential vector by obtaining a numerical solution to the ordinary differential equation of Equation 5, which consists of the estimated generating vector field.

[0071] According to one embodiment, the speech video generating device may use the Euler method or the Runge-Kutta method as an algorithm for obtaining a numerical solution to an ordinary differential equation, but is not limited thereto.

[0072] In step 430, the speech video generating device can generate a combination potential vector by combining the identity potential vector generated in step 410 and the motion potential vector generated in step 420.

[0073] In step 440, the combination latent vector generated in step 430 can be decoded to generate a speech video in which a specific person speaks. For example, a speech video generating device can generate a speech video using a decoder of a pre-trained motion autoencoder.

[0074] FIG. 5 is a block diagram illustrating a computing environment including a computing device according to an exemplary embodiment. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0075] The illustrated computing environment (10) includes a computing device (12). The computing device (12) may be one or more components included in an analysis device (100) according to one embodiment.

[0076] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0077] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0078] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0079] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0080] The disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0081] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will understand that the present disclosure may be implemented in modified forms without departing from the essential characteristics of the present disclosure. Accordingly, the scope of the present disclosure should not be limited to the aforementioned embodiments but should be interpreted to include various embodiments within the scope equivalent to those described in the claims.

Claims

1. An encoder that encodes a person image to generate an identity latent vector; A motion latent vector generation model, which is a flow matching-based probabilistic generation model that generates motion latent vectors based on speech audio; and A speech video generating device comprising a decoder that generates a speech video in which a person speaks based on the above-mentioned identity latent vector and the above-mentioned motion latent vector.

2. In Claim 1, A speech video generating device in which the encoder is an encoder of a motion autoencoder and the decoder is a decoder of the motion autoencoder.

3. In Claim 2, The motion autoencoder above randomly extracts two frames from a training speech video and is a speech video generation device pre-trained such that the error loss between a frame generated using a first frame among the two frames as input and a second frame among the two frame images is minimized.

4. In Claim 1, It further includes a combination unit that generates a combination potential vector by combining the above-mentioned identity potential vector and the above-mentioned motion potential vector, and A speech video generating device in which the above decoder decodes the above combination potential vector to generate the speech video.

5. In Claim 1, The above motion latent vector generation model is, A generation vector field estimation model that estimates a generation vector field for generating the motion potential vector from a noise potential vector based on the above-mentioned speech audio; and A speech video generating device comprising a motion latent vector generating unit that generates a motion latent vector corresponding to the speech audio based on the generated vector field.

6. In Claim 5, A speech video generating device in which the motion potential vector generating unit generates the motion potential vector by obtaining a numerical solution of an ordinary differential equation including the generated vector field.

7. In Claim 6, A speech video generating device, wherein the above ordinary differential equation is expressed by the following mathematical formula. [Mathematical Formula] (Here, x0 is a noise latent vector extracted from an arbitrary distribution, and Φ t is a function of time t that transforms x0 into a motion latent vector, and A s is the spoken audio, and θ * is the optimized parameter of the generating vector field estimation model, and v t is the generated vector field estimated by the generated vector field estimation model) 8. In Claim 1, A speech video generating device, wherein the above motion latent vector generating model is trained to estimate a generating vector field that generates a motion latent vector similar to the above training motion latent vector from a noise latent vector based on training speech audio.

9. In Claim 1, A speech video generating device comprising the above motion latent vector, which includes mouth shape and non-speech motion information synchronized with the speech audio.

10. A method for generating a speech video performed by a computing device, A step of encoding a person image to generate an identity latent vector; A step of generating motion latent vectors based on speech audio using a motion latent vector generation model, which is a flow matching-based probabilistic generation model; and A method for generating a speech video, comprising the step of generating a speech video in which a person speaks based on the above-mentioned identity latent vector and the above-mentioned motion latent vector.

Images