Artificial intelligence-based automatic generation of lip-synced dubbing

Abstract

A display device for generating real-time translated lip-synced dubbing comprises: a speaker diarization model for separating input audio into a background audio feed and an individual speaker audio feed; a face detection model for cropping a frame including a face; an active speaker detection model for pairing the individual speaker audio feed with a cropped frame corresponding to an active speaker; a translation model for obtaining translated speech audio; and a lip-sync model for generating lip-synced video frames including a facial movement of the active speaker synchronized with the translated speech audio by using a prediction model, and outputting the translated speech audio while displaying the lip-synced video frames in real time on the basis of the generated video frames, wherein the facial movement of the active speaker is synchronized with the output translated speech audio.

Description

AI-based automatic dubbing lip-sync generation

[0001] The present disclosure relates to an artificial intelligence-based automatic dubbing lip-sync generation method.

[0002] Many programs available via streaming have very limited language choices, and TV broadcasts generally do not offer language options. According to the "Selected Social Characteristics in the United States" study (American Community Survey, ACS 5-Year Estimated Data Profile) published by the U.S. Census Bureau in 2019, 68 million people (about 21.6% of the total population) do not use English at home, and 26 million people (about 8.2% of the total population) answered that they do not speak English "very well."

[0003] This suggests a pressing need for multilingual options, but in many cases, audio translation is not readily available. Furthermore, when audio dubbing is provided, it often fails to synchronize properly with the video, causing inconvenience to viewers.

[0004] Dubbing is a widely known process in video media production that involves replacing an original audio track with a secondary audio track in another language. Historically, dubbing required a manual process involving a large workforce, including scribes to transcribe the original audio, translators, and voice actors to narrate the translated content. This traditional method is not only time-consuming but also very difficult to execute.

[0005] Recently, artificial intelligence (AI) and machine learning systems are being increasingly utilized in modern audio-video dubbing media production. Traditional AI methods synthesize audio-media works by generating synthetic audio content customized according to user-defined characteristics and features. These systems typically involve training neural networks using collected data (text and audio from various speakers). This process can generate synthetic audio-media works by applying recorded instances of a speaker's voice characteristics or by using pre-recorded synthetic audio generated from trained instances of the learning engine. Currently, methods exist for synthetic audio, transcription, translation, and media synthesis where a speaker's facial movements are synchronized with the new audio. These techniques often involve collecting lip movement tracking data from input video.

[0006] Despite advancements in AI-based synthesis and translation technologies, several drawbacks still remain. For instance, multilingual options are necessary in environments where audio translation is difficult, as the language selection for many programs offered via streaming services is very limited. Furthermore, even when automatic dubbing services are provided, the output often fails to synchronize properly with the video, causing significant inconvenience to viewers. Moreover, existing audio-video dubbing solutions primarily focus on generating high-quality video lip-syncing. However, these solutions fail to prioritize or efficiently handle the real-time processing of live video feeds, such as streams or broadcasts. Processing video from non-live media files is significantly different from processing continuously streamed live feeds.

[0007] To address these problems and other problems known to those skilled in the art, embodiments of the present disclosure enable more people to access programs by eliminating media type restrictions and minimizing dependency on network performance, while easily providing real-time voice detection, dubbing, translation, and lip synchronization without negatively impacting the viewing environment.

[0008] The purpose of this disclosure is to solve the above problem by providing a method and system for generating AI-based automatic dubbing lip-sync for various content.

[0009] An embodiment of the present disclosure includes a method for generating translated dubbed lip synchronization in real time on an edge device based on streaming video content, the method comprising: inputting audio of video content into a speaker segmentation model configured to separate input audio into a background audio feed and an individual speaker audio feed containing speech in a first language; inputting video of video content into a face detection model configured to output one or more cropped frames containing a face within one or more cropped frames; inputting the individual speaker audio feed and the one or more cropped frames into an active speaker detection model to pair the individual speaker audio feed with the one or more cropped frames corresponding to the active speaker; and obtaining translated speech audio corresponding to the content translated into a second language of the individual speaker audio feed. The method comprises the steps of: inputting the translated voice audio, the paired individual speaker audio feed, and one or more truncated frames into a lip-sync model; the lip-sync model is configured to encode the translated audio to obtain a latent spatial representation and utilize a prediction model to generate lip-synced video frames including the facial movements of the active speaker synchronized with the translated voice audio; and displaying the lip-synced video frames in real time based on the generated video frames, wherein when the translated voice audio is output, the facial movements of the active speaker are synchronized with the output translated voice audio.

[0010] According to the implementation, the prediction model predicts 3D mesh vertices to generate a mesh of the active speaker's face, and the step of generating the lip-synced video frame may include detecting mesh key points corresponding to the latent spatial representation of the encoded translated audio and fitting the generated mesh to one or more cropped frames.

[0011] According to the implementation, the step of generating the lip-synced video frames may include the step of extracting texture information of the active speaker's face based on one or more cropped frames and the step of rendering the lip-synced video frames based on the aligned generated mesh and the extracted texture information.

[0012] According to the implementation, the prediction model receives as input a reference image of the active speaker, one or more truncated frames containing the active speaker's mouth blurred due to image noise, and a latent embedding corresponding to the translated speech audio, and can perform inverse diffusion based on the one or more blurred truncated frames to generate prediction frames containing at least visual image information of the active speaker's mouth while speaking the translated speech audio.

[0013] According to the implementation, the active speaker detection model performs a convolution on the individual speaker audio feed and the one or more truncated frames to generate a one-dimensional temporal representation of the individual speaker audio feed and the one or more truncated frames, and inputs the one-dimensional temporal representation into multiple layers of gated cyclic units to match a portion of the individual speaker audio feed with a corresponding portion of the one or more truncated frames.

[0014] According to the implementation, the lip-synced video frames may be generated at a frame rate that matches or exceeds the frame rate of the streaming video content.

[0015] According to the implementation, based on the fact that the lip-synced video frames are generated at a frame rate lower than a threshold value than the frame rate of the streaming video content, the lip-sync model may be instructed to skip generating video frames for one or more frames within one or more truncated frames, and to generate the lip-synced video frames by calculating an average value based on adjacent generated video frames for the one or more skipped frames.

[0016] Another embodiment of the present disclosure includes a display device for generating real-time translated dubbing lip synchronization based on streaming video content, said display device; one or more processors; and memory storing instructions that cause the display device to perform the following when executed by one or more processors: inputting the audio of the video content into a speaker segmentation model configured to separate the input audio into a background audio feed and an individual speaker audio feed containing speech in a first language, inputting the video of the video content into a face detection model configured to output one or more cropped frames containing a face within one or more cropped frames, inputting the individual speaker audio feed and the one or more cropped frames into an active speaker detection model to pair the individual speaker audio feed with the one or more cropped frames corresponding to the active speaker, obtaining translated speech audio corresponding to the content translated into a second language of the individual speaker audio feed, inputting the translated speech audio, the paired individual speaker audio feed, and the one or more cropped frames into a lip-sync model, wherein the lip-sync model is configured to encode the translated audio to obtain a latent spatial representation and utilize a prediction model to generate lip-synced video frames containing the facial movements of the active speaker synchronized with the translated speech audio; And based on the generated video frames, the lip-synced video frames are displayed in real time, and when the translated voice audio is output, the facial movements of the active speaker can be synchronized with the output translated voice audio.

[0017] According to the implementation of the display device, the prediction model predicts 3D mesh vertices to generate a mesh of the active speaker's face, and generating the lip-synced video frames includes detecting mesh key points corresponding to the latent spatial representation of the encoded translated audio and fitting the generated mesh to one or more cropped frames.

[0018] According to the implementation of the display device, generating the lip-synced video frames includes extracting texture information of the active speaker's face based on one or more cropped frames, and rendering the lip-synced video frames based on the aligned generated mesh and the extracted texture information.

[0019] According to the implementation of the display device, the prediction model receives as input a reference image of the active speaker, a latent embedding corresponding to the active speaker's mouth blurred due to image noise in one or more truncated frames, and the translated speech audio, and performs inverse diffusion based on the one or more blurred truncated frames to generate prediction frames containing at least visual image information of the active speaker's mouth while speaking the translated speech audio.

[0020] According to the implementation of the display device, the active speaker detection model performs a convolution on the individual speaker audio feed and the one or more truncated frames to generate a one-dimensional temporal representation of the individual speaker audio feed and the one or more truncated frames, and inputs the one-dimensional temporal representation into a multi-layered gated cyclic unit to match a part of the individual speaker audio feed with a corresponding part of the one or more truncated frames.

[0021] According to the implementation of the display device, the lip-synced video frames are generated at a frame rate that matches or exceeds the frame rate of the streaming video content.

[0022] According to the implementation of the display device, based on the fact that the lip-synced video frames are generated at a frame rate lower than a threshold value than the frame rate of the streaming video content, the lip-sync model is instructed to skip the generation of video frames for one or more frames within one or more truncated frames, and to generate the lip-synced video frames by calculating an average value based on adjacent generated video frames for the one or more skipped frames.

[0023] According to an implementation of the display device, the display device further includes a transceiver, and the translated voice audio is obtained by transmitting the individual speaker audio feed through the transceiver to a voice-to-voice translation module in which a second language is selected, and receiving the voice audio translated into the second language through the transceiver.

[0024] According to some implementations, a computing or electronic device comprises one or more processors, non-transient memory, and one or more programs. One or more programs are configured to be stored in non-transient memory and executed by one or more processors, and one or more programs include instructions that perform or cause to perform the methods described herein.

[0025] According to some embodiments, a non-transient computer-readable storage medium stores instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform or cause to perform one of the methods described herein. According to some embodiments, the electronic device includes one or more processors, a non-transient memory, and means for performing or causing to perform one of the methods described herein.

[0026] The present invention is not limited to the description above, and other aspects and advantages of the present invention not mentioned above will be understood through the following description of embodiments of the present invention. Furthermore, it will be understood that aspects and advantages of the present invention can be achieved by the configurations described in the claims and combinations thereof.

[0027] To ensure that the content of this disclosure is understood by those skilled in the art, a more detailed description may be provided by referring to some exemplary aspects of the implementation illustrated in the attached drawings.

[0028] FIG. 1 is an example block diagram of an AI system according to an embodiment of the present disclosure.

[0029] FIG. 2 is a block diagram of an example of an edge device according to an embodiment of the present disclosure.

[0030] FIG. 4 is a block diagram of a neural network according to an embodiment of the present disclosure.

[0031] FIG. 5 shows a diagram illustrating an example of a system that generates converted lip synchronization according to an embodiment of the present disclosure.

[0032] FIG. 6 shows an example of an active speaker detection (ASD) model according to an embodiment of the present disclosure.

[0033] FIG. 7 shows an example of a speech-to-speech translation (ASR / AST) model according to an embodiment of the present disclosure.

[0034] FIG. 8 shows an example of a lip-sync model according to an embodiment of the present disclosure.

[0035] FIG. 9 shows an example of a diffusion-based lip synchronization prediction model according to an embodiment of the present disclosure.

[0036] FIG. 10 is an example of a method for compensating for resource constraints according to an embodiment of the present disclosure.

[0037] FIG. 11 is an example of a method for generating converted lip synchronization according to an embodiment of the present disclosure.

[0038] In accordance with general practice, various features depicted in the drawings may not be drawn to their actual scale. Therefore, the sizes of various features may be arbitrarily enlarged or reduced for clarity. Additionally, some drawings may not depict all components of a given system, method, or device. Finally, the same reference number may be used to denote the same feature throughout the specification and drawings.

[0039] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing numbers are given the same reference numerals, and redundant descriptions are omitted. In the following description, the terms "module" and "part" referring to components are used interchangeably for convenience of explanation and do not necessarily have different meanings or functions. Where possible, the same reference numerals are used throughout the drawings to refer to identical or similar components. In the following description, known functions or configurations that could obscure the essence of the invention are not described. It should be understood that the attached drawings are intended only to facilitate the explanation of various technical features, and the embodiments presented in this specification are not limited by the attached drawings. Accordingly, the present invention should be interpreted as including all modifications, equivalents, and substitutions other than those specifically described in the attached drawings.

[0040] The terms used in this specification are for the purpose of describing specific exemplary implementations only and are not intended to be restrictive. The singular forms "a," "an," and "the" used in this specification may be interpreted to include the plural forms unless the context clearly indicates otherwise. Terms such as "comprises," "comprising," "includes," "including," "containing," "has," "having," or variations thereof are inclusive and thus specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Additionally, terms such as "first," "second," and other numeric terms are used solely to distinguish one element from another. These terms are generally used solely to distinguish one element from another.

[0041] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Throughout the specification, the same reference numerals denote the same components, and redundant descriptions of components are omitted.

[0042] FIG. 1 is a diagram illustrating an example of an AI system comprising an AI device, an AI server, and a network connecting said components. Although relevant features are illustrated, those skilled in the art will understand that various other features from the present disclosure have not been illustrated for brevity and to avoid obscuring more important aspects of the exemplary embodiments disclosed herein.

[0043] Referring to FIG. 1, the AI ​​device (110) may include an artificial intelligence-based device of the present disclosure, and may include at least one of, for example, a robot, an autonomous vehicle, a communication terminal (e.g., a mobile phone, a smartphone, a tablet PC), an edge device, and a home appliance (e.g., a television, a washing machine, a robot vacuum cleaner).

[0044] Here, artificial intelligence refers to the field of researching artificial intelligence or methodologies for creating artificial intelligence, while machine learning refers to the field of researching methodologies for defining and solving various problems addressed within the field of artificial intelligence. Furthermore, machine learning can be defined as an algorithm that improves the performance of a specific task through iterative experience.

[0045] An Artificial Neural Network (ANN) is a model used in machine learning, generally referring to a model capable of problem-solving. It can be constructed by artificial neurons (nodes) forming a network through synaptic connections. An ANN can be defined by connection patterns between neurons of different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0046] An ANN may include an input layer and an output layer, and optionally include one or more hidden layers. Each layer includes one or more neurons, and the ANN may include synapses connecting the neurons to each other. In an ANN, each neuron can output a function value of an activation function, a weight, and a bias for an input signal input through a synapse.

[0047] Model parameters refer to parameters determined through learning and may include synaptic connection weights and neuron biases. Additionally, hyperparameters refer to parameters set prior to training in machine learning algorithms and may include the learning rate, number of iterations, mini-batch size, and initialization function.

[0048] The objective of training an Artificial Neural Network (ANN) may be to determine model parameters that significantly reduce the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process.

[0049] Machine learning can train artificial neural networks through supervised learning.

[0050] Supervised learning can refer to a method of training an artificial neural network using labeled training data. Additionally, a label can represent the target correct answer (or result) that the artificial neural network must predict when training data is input into it.

[0051] Accordingly, the artificial intelligence-based object identification device trains an artificial neural network using a machine learning algorithm or requests a trained artificial neural network from an AI server (120) and receives the trained artificial neural network from the AI ​​server (120). Additionally, when an image is received, the object identification device can estimate the type of object in the received image using the trained artificial neural network.

[0052] When the AI ​​server (120) receives a request for a learned artificial neural network from the AI ​​device (110), the AI ​​server (120) can train the artificial neural network using a machine learning algorithm and provide the learned artificial neural network to the AI ​​device (110). The AI ​​server (120) can be composed of multiple servers to perform distributed processing. In this case, the AI ​​server (120) can be included as part of the AI ​​device (110) to perform at least a part of the AI ​​processing together.

[0053] A network (130) can connect an AI device (110) and an AI server (120). The network (130) may include wired networks such as a local area network (LAN), a wide area network (WAN), a wide area network (MAN), and an integrated service digital network (ISDN), and wireless networks such as wireless LAN, CDMA, Bluetooth®, and satellite communication, but the present disclosure is not limited to these examples. The network (130) may also transmit and receive information using short-range communication and / or long-range communication. Short-range communication may include Bluetooth®, radio frequency identification (RFID), infrared data communication (IrDA), ultra-wideband (UWB), Zigbee, and Wi-Fi (radio fidelity) technologies, and long-range communication may include Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Orthogonal Frequency Division Multiple Access (OFDMA), and Single Carrier Frequency Division Multiple Access (SC-FDMA).

[0054] The network (130) may include connections of network elements such as hubs, bridges, routers, switches, and gateways. The network (130) may include a multi-network environment including one or more connected networks, such as a public network like the Internet and a private network like a secure enterprise-only network. Access to the network (130) may be provided through one or more wired or wireless access networks. Additionally, the network (130) may support an Internet of Things (IoT) network for information exchange and processing between distributed elements such as the Internet of Things (IoT), 3G, 4G, LTE (Long Term Evolution), and 5G communication.

[0055] Now, referring to FIG. 2, a drawing of an exemplary device (200) that may be used to implement, implement, execute, or perform embodiments of the present disclosure is provided. Referring to FIG. 2, the term “device” may be used, but those skilled in the art will understand that the device (200) may be implemented as or as part of various other components and / or devices, including but not limited to robots, autonomous vehicles, communication or computing terminals (e.g., mobile phones, smartphones, laptops, or tablet PCs), edge devices, or home appliances or devices (e.g., televisions, washing machines, refrigerators, or robot vacuum cleaners, etc.).

[0056] In selected embodiments, the device (200) may include a bus (203) (or multiple buses) or other communication mechanism, a processor (201), processor internal memory (201a), main memory (204), read-only memory (ROM, 205), one or more additional storage devices (206) and / or a communication interface (202), or analogs or sub-combinations thereof. The embodiments described herein may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic devices designed to perform the functions described herein, or an optional combination thereof. In all embodiments, the various components described herein may be implemented as a single component or as various individual components.

[0057] A bus (203) or other communication mechanisms including multiple buses or mechanisms may support information communication within the device (200). A processor (201) may be connected to the bus (203) to process information. In selected embodiments, the processor (201) may be a specialized or dedicated microprocessor configured to perform a specific task according to the features and aspects disclosed herein by executing machine-readable software code that defines a specific task. In some embodiments, multiple processors (201) may be provided, each having a processing unit dedicated to a specific specialized task, such as graphics processing or artificial intelligence-related processing.

[0058] Main memory (204, e.g., Random Access Memory (RAM) or other dynamic storage device) may be connected to a bus (203) to store information and instructions executed by the processor (201). The processor (201) may include internal memory (201a), such as a CPU cache implemented as SRAM, to store data used for instruction execution. Utilizing internal memory (201a) can optimize data and memory management by reducing memory bandwidth usage of the main memory (204). Although the internal memory (201a) is depicted as a component of the processor (201) in FIG. 2, it may be understood that embodiments in which the internal memory (201a) is a separate component from the processor (201) are also included. The main memory (204) may also store temporary variables or other intermediate information during such instruction execution.

[0059] A ROM (205) or other static storage device may be connected to a bus (203) to store static information and instructions for a processor (201). An additional storage device (206, e.g., a magnetic disk, an optical disk, a memory card, etc.) may be connected to the bus (203). The main memory (204), ROM (205), and additional storage device (206) may include a non-transient computer-readable medium that stores information, instructions, or a combination thereof, for example, instructions that cause the device (200) to perform one or more operations of the method described herein when executed by the processor (201). A communication interface (202) may also be connected to the bus (203). The communication interface (202) may provide or support bidirectional data communication between the device (200) and one or more external devices (e.g., other devices included within the computing environment).

[0060] In a selected embodiment, the device (200) may be connected to a display (207) (e.g., via a bus). The display (207) may use a suitable mechanism to convey information to the user of the device (200). For example, the display (207) may include or utilize a liquid crystal display (LCD), a light-emitting diode (LED) display, a projector, or other display device to provide information to the user of the computer (100) as a visual display. One or more input devices (208, e.g., an alphanumeric keyboard, a remote control, a mouse, a microphone, a stylus pen) may be connected to the bus (203) to convey information and commands to the device (200). In a selected embodiment, one input device (208) may provide or support cursor position control to allow selection and execution of various objects, files, programs, etc. provided by the device (200) and displayed on the display (207).

[0061] The device (200) may be used to transmit, receive, decode, and display one or more image or video files. In certain embodiments, such transmission, reception, decoding, and display may be performed in response to the processor (201) executing one or more instruction sequences stored in main memory (204). These instructions may be read into main memory (204) from another non-transient computer-readable medium (e.g., storage device).

[0062] When executing an instruction sequence contained in the main memory (204), the processor (201) may perform one or more procedures or steps described herein. In selected embodiments, one or more processors within a multiprocessing unit may be used to execute the instruction sequence contained in the main memory (204). Alternatively, in addition to this, the procedures or steps may be implemented according to the features and aspects disclosed herein by using firmware instead of or together with software instructions. Accordingly, embodiments according to the features and aspects disclosed herein may not be limited to a specific combination of hardware circuits and software.

[0063] A non-transient computer-readable medium means any medium that stores instructions for execution by a processor (201) or data to be processed by a computer, and includes all computer-readable media excluding transient radio signals. Such non-transient computer-readable media may include, but are not limited to, non-volatile media, volatile media, and temporary storage media (e.g., cache memory). Non-volatile media may include optical disks or magnetic disks, such as additional storage devices. Volatile media may include dynamic memory, such as main memory. Common non-transient computer-readable media may include, for example, hard disks, floppy disks, magnetic tapes or other magnetic media, CD-ROMs, DVDs, Blu-rays or other optical media, RAM, PROMs, EPROMs, FLASH-EPROMs, other memory cards, chips or cartridges, or other computer-readable memory media.

[0064] In a selected embodiment, the communication interface (202) may provide or support external bidirectional data communication through a network link. For example, the communication interface (202) may be a wireless network interface controller or a cellular radio that provides a data communication network connection. Alternatively, the communication interface (202) may include a LAN card that provides a data communication connection to a compatible Local Area Network (LAN). In such an embodiment, the communication interface (202) may transmit and receive electrical, electromagnetic, or optical signals that convey information.

[0065] A network link can provide data communication to other data devices (e.g., other devices such as 200 or various types of terminals) through one or more networks. For example, a network link can provide a connection through a local network of a host computer or to data equipment operated by an Internet Service Provider (ISP). An ISP can provide data communication services over the Internet. Thus, the device (200) can transmit and receive commands, data, or a combination thereof, including program code, through one or more networks, network links, and communication interfaces (202). Thus, the device (200) can interface with or communicate with a remote server or a combination thereof.

[0066] The various devices, modules, terminals, etc. discussed in this specification may be implemented on a computer by executing software containing machine instructions read from a computer-readable medium, as previously discussed. In certain embodiments, various hardware aspects may be implemented using a single computer, while in other embodiments, the system may be implemented using multiple computers, input / output systems, and hardware.

[0067] In the case of software implementation, the specific embodiment described herein may be implemented as separate software modules, such as procedures and functions, that perform one or more of the functions and operations described herein. Software code may be implemented as a software application written in a suitable programming language and may be stored in memory and executed by a controller or processor.

[0068] FIG. 3 is a block diagram illustrating an example of a device (301), also referred to as an edge device, a deployed device, a target computing platform, etc., according to some embodiments. While certain features are illustrated, those skilled in the art may understand that various other features are not illustrated for brevity and so as not to obscure more important aspects of the embodiments disclosed herein.

[0069] To this end, as a non-limiting example, in some implementations, an edge device (implemented as the device (200) shown in FIG. 2 in some cases) or device (301) may include one or more processing units (302, e.g., microprocessor, ASIC, FPGA, GPU, CPU, processing core and / or similar), one or more I / O devices and sensors (306), one or more communication interfaces (308, e.g., USB, FIREWIRE, THUNDERBOLT, IEEE(802).3x, IEEE(802).11x, IEEE(802).16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE and / or similar type interfaces), one or more programming (e.g., I / O) interfaces (310), one or more displays (312), one or more external image sensors (314), memory (320), and one or more communication buses (304) for interconnecting these components and various other components.

[0070] In some implementations, one or more communication buses (304) may include circuits that interconnect and control communication between system components.

[0071] In some implementations, one or more displays (312) may display content. In some implementations, one or more displays (312) may also be configured to display flat video content to the user (e.g., 2D or “flat” audio-video interleave (AVI), Flash video (FLV), Windows Media Video (WMV) or similar files related to TV episodes or movies, or live video passthrough of the operating environment).

[0072] In some embodiments, one or more displays (312) correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), silicon liquid crystal (LCoS), organic light-emitting field-effect transition (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more displays (312) correspond to waveguide displays such as diffraction, reflection, polarization, or holography. For example, the device (301) includes a single display. In other examples, the device (301) includes a display for each eye of the user.

[0073] In some implementations, one or more external image sensors (314) may be configured to acquire image data frames. For example, one or more optional external and / or internal image sensors (314) correspond to one or more RGB cameras (e.g., including complementary metal-oxide-semiconductor (CMOS) image sensors or charge-coupled device (CCD) image sensors), infrared (IR) image sensors, event-based cameras, etc.

[0074] Memory (320) includes high-speed random access memory such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some implementations, memory (320) includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory (320) optionally includes one or more storage devices located remotely from one or more processing units (302). Memory (320) includes a non-transient computer-readable storage medium. In some implementations, memory (320) or the non-transient computer-readable storage medium of memory (320) stores the following programs, modules, and data structures, or a subset thereof including an optional operating system (330). The optional operating system (330) includes procedures for handling various basic system services and performing hardware-dependent tasks.

[0075] FIG. 3 may be intended as a functional description of various features that may exist in a particular embodiment, rather than as a structural schematic diagram of the embodiments described herein. As will be known to those skilled in the art, items depicted individually may be combined, and some items may be separated. For example, some functional modules depicted individually in FIG. 3 may be implemented as a single module, and various functions of a single functional block may be implemented as one or more functional blocks in various embodiments. The actual number of modules, the division of specific functions, and the manner in which functions are assigned among modules vary from embodiment to embodiment and may vary in part depending on the specific combination of hardware, software, and / or firmware selected for a particular embodiment.

[0076] FIG. 4 is a block diagram of an exemplary neural network (400) according to some embodiments. While certain features are illustrated, those skilled in the art may understand that various other features are not illustrated for brevity throughout this disclosure and to avoid obscuring more important aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the neural network (400) includes an input layer (420), a first hidden layer (422), a second hidden layer (424), and an output layer (426). Although the neural network (400) is an example of including two hidden layers, those skilled in the art may understand throughout this disclosure that one or more additional hidden layers exist in various embodiments. Adding hidden layers increases computational complexity and memory requirements, but can improve the performance of some applications.

[0077] In various implementations, the input layer (420) is connected (e.g., configured) to receive various inputs (402, e.g., image data). For example, the input layer (420) receives pixel data from one or more image sensors (e.g., sensor 314 shown in FIG. 3). In various implementations, the input layer (420) includes a number of Long Short-Term Memory (LSTM) logic units (420a), which are also referred to to those skilled in the art as neuron model(s). In some of these implementations, the input matrix of the LSTM logic unit (420a) at feature points includes an orthogonal matrix. For example, the size of this matrix is ​​a function of the number of feature points included in the feature stream.

[0078] In some implementations, the first hidden layer (422) includes a number of LSTM logic units (422a). In some implementations, the number of LSTM logic units (422a) ranges from about 10 to 500. Those skilled in the art will understand that in these implementations, the number of LSTM logic units per layer is much smaller than in previously known approaches (approximately O(101) to O(102)). Therefore, such implementations can be embedded in devices with very limited resources. As can be seen in the example of FIG. 4, the first hidden layer (422) receives input from the input layer (420). For example, the first hidden layer (422) performs one or more of the following: convolution operations, non-linear operations, normalization operations, pooling operations, etc.

[0079] In some implementations, the second hidden layer (424) includes a number of LSTM logic units (424a). In some implementations, the number of LSTM logic units (424a) is equal to or similar to the number of LSTM logic units (420a) of the input layer (420) or the number of LSTM logic units (422a) of the first hidden layer (422). As can be seen in the example of FIG. 4, the second hidden layer (424) receives input from the first hidden layer (422). Additionally and / or alternatively, in some implementations, the second hidden layer (424) receives input from the input layer (420). For example, the second hidden layer (424) performs one or more of the following: convolution operations, non-linear operations, normalization operations, pooling operations, etc.

[0080] In some implementations, the output layer (426) includes a plurality of LSTM logic units (426a). In some implementations, the number of LSTM logic units (426a) is equal to or similar to the number of LSTM logic units (420a) of the input layer (420), the number of LSTM logic units (422a) of the first hidden layer (422), or the number of LSTM logic units (424a) of the second hidden layer (424). In some implementations, the output layer (426) is a task-dependent layer that performs computer vision-related tasks such as feature extraction, object recognition, object detection, pose estimation, etc. In some implementations, the output layer (426) includes an implementation of a multinomial logistic function (e.g., a softmax function) that generates a plurality of outputs (430).

[0081] Neural networks such as CNNs are frequently used to solve computer vision problems, including feature extraction, object recognition, object detection, and pose estimation. Modern CNNs can generally consist of an input layer, multiple hidden layers, and an output layer. In at least some scenarios, the input to the input layer of a CNN is an image frame, and the output layer is a task-dependent layer. Hidden layers often include one of several operations, such as convolution, nonlinearity, regularization, or pooling. For example, each convolutional layer may include a set of filters whose weights are learned directly from the data. Continuing this example, the output of these filters is one or more feature maps obtained by applying the filters to the input data of the convolutional layer.

[0082] Embodiments of the present disclosure include an apparatus, system, and method for generating AI-based automatic dubbing lip-sync for language-translated video content. The embodiments include a multimodal TV system that performs transcription, translation, and video lip-syncing by utilizing multiple AI models operating on edge devices (e.g., televisions, laptop computers, tablet computers, etc.) that primarily display video content. Some embodiments may be configured to process live video feeds, such as internet-based streams or television broadcasts, in real time without excessively degrading the quality of the video dubbing.

[0083] In an example involving streaming video, the embodiments of the present disclosure may perform stream processing by receiving a streaming video input, separating audio into voice and background elements, dubbing the voice audio, pairing the translated audio with the corresponding speaker in the scene, and synchronizing the speaker's facial movements accordingly. The embodiments may provide flexibility by performing frame-by-frame processing on a live feed, eliminating media type restrictions, and utilizing diarization for multi-speaker dubbing. The embodiments may be configured to perform computations on a device (i.e., a device such as a TV or tablet), which minimizes dependence on network performance and maximizes internet resources allocated to downloading the video content itself.

[0084] FIG. 5 illustrates an example of a system for generating AI-based automatic dubbing lip synchronization according to an embodiment of the present disclosure.

[0085] The system (500) is designed to coordinate multiple AI models sequentially to enable efficient audio and video processing in various computational scenarios. This system requires a stable internet connection and a powerful graphics processor (GPU) implemented in the device (510) to process all models simultaneously. In this example and in the present disclosure, the device (510) may be referred to as being implemented as a television, but those skilled in the art will understand that various types of edge device implementations may be considered as discussed above.

[0086] The discussion may refer to various modules or models, such as, for example, a segmentation model (502), a face detection model (503), a speech-to-speech conversion model (504), an active speaker detection (ASD) model (505), etc. The present disclosure considers embodiments also referred to as modules or models, which are implemented not only as software components but also as one or more hardware elements (e.g., processor, controller, GPU, etc.) or one or more non-transient computer-readable memories (instructions stored to be executed by hardware elements such as processor, controller, GPU, etc.). These instructions enable a device and its components, such as the device (200), to perform the various operations described herein.

[0087] When receiving a video-audio stream (501) from a source such as cable or internet streaming, the system (500) may introduce a short delay of a few seconds to account for processing time.

[0088] According to the present embodiment, the pipeline begins by splitting the input audio using a splitting model (502) to separate background noise from individual audio feeds corresponding to each detected speaker. Simultaneously, a face detection model (503) processes the video feed to extract and crop faces, focusing on frames where the lower part of the face is visible. Then, the audio and video feeds are synchronized using an active speaker detection (ASD) model (505) that pairs the audio with the corresponding video. The audio feed of each speaker is sent to a cloud-based all-in-one speech-to-speech conversion model (504). This model generates speech transcription (ASR), translated transcript (MT), and translated speech audio (AST).

[0089] In the final step, lip-syncing (506) is performed for each translated audio feed matched with the speaker. The system uses a mesh-based model (506) to predict face mesh attributes and mixes them with the translated audio to create new video frames that match the new audio. The TV then outputs a final result (507) containing the translated subtitles, the translated audio, and the lip-synced video.

[0090] The basic system consists of five main components, each of which implements a real-time system through inference faster than real-time. This is explained in detail below.

[0091] Speaker segmentation (502) separates the audio feed into background and speaker feeds. The segmentation process begins by converting the audio feed into a Mel Spectrogram, which is then separated into background and speaker components.

[0092] A lightweight speaker encoder model utilizing a convolutional augmentation transformer layer (also called a conformer layer) can convert audio into a d-vector corresponding to the number of detected speakers.

[0093] Various clustering methods can be used depending on the number of identified speakers. For example, Aggregative Hierarchical Clustering (AHC) can be used when the number of speakers is below a predefined threshold. As another example, spectral clustering can be applied to a larger number of speakers. Additionally, small speech segments can be filtered out as noise based on a set threshold to optimize the use of computational resources.

[0094] Accordingly, speaker splitting (502) can output separated background and speaker feeds based on the input audio feed (501).

[0095] The face detection model (503) can crop a portion of the frame containing the detected face. Since all subsequent processes may use the frame with the face cropped for computational convenience, the system includes a face detection layer (503) based on a series of lightweight convolution operations to output a bounding box for the face where most of the face is visible in each output frame.

[0096] The active speaker detection (ASD) model (505) may be configured to match the face cropped frames output from the face detection model (503) with the dilated audio. In some embodiments, as illustrated in FIG. 6, for resource conservation, the ASD (600) may use a short chain of 3D convolutions (601) for a video frame sequence and 2D convolutions (605) for an audio sequence to match the given audio with the cropped frames in the shortest possible time.

[0097] For example, for resource conservation, a 3D convolution can be converted into a sequence of 2D spatial (602, 606) convolutions and 1D temporal (603, 607) convolutions, and the 2D convolution can be converted into a sequence of 1D spatial (604) convolutions and 1D temporal (608) convolutions to match the video side with the audio side's time domain.

[0098] Based on one-dimensional representations of video (604) and audio (608), in the example of FIG. 9, multiple layers of gated recirculation units (GRU) (610), such as two layers, can be used together with a fully connected layer (906) to determine whether the one-dimensional representations of the video frames and audio are correctly aligned. This allows verification that the video frames correctly depict the active speaker speaking the voice audio. Thus, the ASD can output alignment information between the video frames and the voice audio to provide a correlation dataset for the active speaker and their voice audio.

[0099] Referring again to FIG. 5, a cloud-based speech-to-speech conversion (ASR / AST) model (504) receives speech audio from a speaker segmentation model (502) and generates translated text and audio. In embodiments of the present disclosure, it is taken into account that a high level of prior knowledge is required for the translation task to be processed on an edge device, such as a television. However, it is also taken into account that, due to advancements in processing, data transmission and reception protocols, and data memory, certain embodiments may include a speech-to-speech conversion (ASR / AST) model (504) implemented locally on an edge device (510). In the example of FIG. 6, the speaker audio segmented from the speaker segmentation model (502) is transmitted to a cloud-based function, such as a server, that outputs translated text and audio.

[0100] FIG. 7 shows an example of a cloud-based speech-to-speech translation model, which is known to those skilled in the art. In the example of FIG. 7, a streaming audio encoder can use a conformer architecture, repetitive convolution, and a self-attention model to translate and transcribe original speech audio (704) to generate transcribed copies (701, 702) and translated speech audio (703) in source and target languages.

[0101] Encoded audio can be used for speech-to-text conversion and translation. Both translation and transcription can be performed in a Connectionist Time Classification (CTC) decoder, and translation is processed via autoregression. However, if accuracy is low in this lightweight decoder, more powerful solutions such as Google Translate can be used.

[0102] To generate audio from a given sample audio and translated text, a cloud-based speech-to-speech conversion (ASR / AST) model (504) applies a Text-to-Unit (T2U) encoder with a CTC decoder, excluding non-auto-regression. It is processed through a HiFi-GAN vocoder to output the final audio.

[0103] Referring again to FIG. 5, the lip-sync model (506) can predict mesh vertices for a face-cropped frame that matches the converted audio received from the speech-to-speech conversion (ASR / AST) model (504). Given a face-cropped frame that matches the converted audio, audio-consistent lip synchronization is achieved to generate a new video frame.

[0104] Referring to FIG. 8, an example of a lip-sync model (800) is illustrated. In the example of FIG. 8, the lip-sync model (800) can receive a translated voice audio (808) from a translation model, an original voice audio feed, information from an active speaker detection model that is temporally aligned with the original voice audio feed, and one or more frames of the active speaker that receive the translated voice audio (808).

[0105] The translated speech audio (808) (see 511 in FIG. 5) can be encoded through a Variational Autoencoder (801) to obtain a latent spatial representation. This data can be passed to a fully connected layer (802) configured to predict vertex coordinates that generate a mesh representation (804) of the speaker's face while the translated speech audio feed is being spoken. Additionally, based on one or more truncated frames of the active speaker (809), a Variational Autodecoder (803) can predict a texture (807) corresponding to the active speaker's face.

[0106] The generated mesh (804) can be aligned in 3D across one or more frames (809) (805). This is possible by detecting mesh key points in one or more frames and aligning / warping the image accordingly (805). Then, the warped image (806) can be rendered (810) and mixed with the generated texture (807) to output one or more frames as a final output video (811), in which the active speaker's face is synchronized with the audio feed.

[0107] In another embodiment, generating an output video having one or more frames of a lip-synced active speaker and a translated speech audio feed can be achieved using a diffusion-based lip synchronization prediction model for generating a lip-synced video output.

[0108] Referring to FIG. 9, a diffusion-based lip synchronization prediction model (902) may be configured to mask the lower part of the active speaker's face in one or more frames (901) by introducing pixelation, blurring, noise, or other visual interference (902), and to pass one or more resulting blurred images to a diffusion model (903). In addition to the blurred images, the diffusion model (903) receives as input a reference image (904) of the active speaker's entire face obtained from original video content, unrelated external content, or one or more image frames of the active speaker himself. Additionally, the diffusion model (903) may receive as input a latent embedding obtained by encoding the active speaker's translated speech audio (905) using an audio encoder such as CLIP. Here, the latent embedding may be aligned with the image embeddings corresponding to the one or more blurred images and the reference images in a shared semantic space.

[0109] Based on a latent embedding of the translated speech audio, one or more blurred images, and a reference image of the active speaker, the diffusion model (903) may be configured to generate one or more frames (907) having predicted pixel information. After noise is removed from the generated one or more frames (907), the model may output one or more final frames (910) having generated visual image information representing the model's prediction of the active speaker's face and mouth while speaking the translated speech audio.

[0110] In the embodiment of FIG. 9, the lip synchronization prediction model (920) may be supported by a learning component including a SyncNet model configured to determine audio synchronization between the final frame (910) and the translated speech audio (905) based on an audio synchronization loss function L_sync (913). Additionally, the final frame (910) may be compared using a model configured to determine the relative similarity of the visual appearance of the content with the original frame based on a perceptual similarity loss function L_lpips (909). Finally, a sequential discriminator model (912) configured to detect temporal consistency between the video aspects of one or more original frames (901) and the final output frame (910) based on a Generative Adversarial Network (GAN) loss function L_GAN (916) may be implemented.

[0111] Accordingly, in some embodiments, particularly when the hardware specifications of the edge device support the computing requirements of a diffusion-based generative model for outputting a final output frame (910) in real time according to input streaming or broadcast content, a rip synchronization prediction model (902) may be implemented instead of a mesh-based rip synchronization model as described in relation to FIG. 8.

[0112] Based on the example above, a multimodal TV system can perform voice transcription, translation, and video lip-syncing by utilizing multiple AI models running on edge devices such as TVs. Upon receiving video input, the system separates the audio into voice audio and background audio, dubs the voice audio to match the corresponding speaker within the scene, and synchronizes the speaker's facial movements accordingly. By maintaining audio-video synchronization of facial movements, this allows for the easy provision of real-time transcripts and translations to the user without negatively impacting the viewing experience.

[0113] In some embodiments, the system may be adjusted according to various operational factors and conditions detected by the system and / or edge device. For example, the following four scenarios may occur.

[0114] A: General Scenario: The system is ahead of the output stream by a specified threshold time. The threshold can be determined by other criteria, such as the lip-sync generation time measured based on the input time of the video content, or the number of frames processed per second based on the number of frames played per second. In this scenario, it can be determined that sufficient computational resources remain to prepare for contingencies.

[0115] B: Resource Shortage Scenario: The system is less than the critical time limit ahead of the output stream. In this scenario, it can be determined that all computational resources are being used.

[0116] C: Severe Constraint Scenario: The system has significantly exceeded the critical time limit mentioned in Scenario B. In this case, the critical time limit may be set to a different time limit than that mentioned in Scenario B. It may be determined that there are insufficient computational resources to process all models in the pipeline.

[0117] D: Limited Internet Scenario: Internet bandwidth is insufficient to support full voice-to-voice model I / O. This scenario can be applied simultaneously with Scenario B or Scenario C.

[0118] Based on one or more of the above scenarios, embodiments of the present disclosure may implement various and corresponding adjustments to the process to ensure appropriate performance while maintaining accuracy and usefulness.

[0119] Given Scenario A, in some embodiments, all systems operate using anecdotes, ASD, voice-to-voice and lip synchronization as described above with reference to FIG. 5.

[0120] In the case of Scenario B, since rip synchronization is the model that requires the most resources in the framework in some implementations, the system may lower the frame rate for mesh prediction, which may include skipping a few frames and averaging the mesh vertex positions for alignment instead.

[0121] In the case of Scenario C, some implementations may not offer the option to lower lip-sync quality to maintain usability and avoid viewer discomfort. Therefore, the system implementation may completely skip a specific model to complement the scenario until the system is ahead of the output by a minimum threshold time limit or more, or by other specified times.

[0122] In some embodiments, the priority for determining model skipping in Scenario C may be as follows.

[0123] 1. Lip synchronization and active speaker detection

[0124] 2. Script translation, voice translation, and diary translation

[0125] 3. Audio Transcription

[0126] In the case of Scenario D, in some implementations, compensation may vary depending on the severity of the connection failure. For example, according to one implementation, the action process and priority may be based directly on bandwidth, and examples of ordered action processes in descending bandwidth are as follows.

[0127] 1. Skip two transcription contents and receive only the voice translation audio feed.

[0128] 2. Transcribe the text from the device and send only the text feed for text-to-text translation.

[0129] 3. If resources allow, generate a translated audio feed on the device.

[0130] 4. Wait until the sentence is complete before sending the text transcribed in a buffered manner.

[0131] If an internet failure in Scenario D occurs simultaneously with the above Scenario B or C, the embodiments of this disclosure can always fall back to Scenario C and control operations to avoid complications.

[0132] FIG. 10 is a flowchart illustrating an exemplary method according to an embodiment of the present invention. In step 1001, the system determines whether it is ahead of the output stream by a critical time limit Tthres,1.

[0133] If the system is ahead of the output stream by the critical time limit Tthres,1 (i.e., 'No' in step 1001), determine in step 1003 whether the system is ahead of the output stream by 1 second by the critical time limit Tthres,0. In this case, it can be determined that all computing resources are being utilized.

[0134] If the system is not ahead of the output stream by the first critical time limit Tthres,1 (i.e., yes in step 1001), the system may completely skip a specific model to compensate until it is ahead of the output again by more than the critical time limit or by another specified time (steps 1002, 1007).

[0135] If the system is ahead of the output stream by the first threshold time limit Tthres,1 (i.e., no at step 1001), or if the system is ahead of the second threshold time limit Tthres,0 (i.e., yes at step 1003), the system may lower the frame rate for mesh prediction. This may include skipping a few frames and instead averaging the mesh vertex positions for alignment step 1004.

[0136] If the system is ahead of the output stream by the first threshold time limit Tthres,1 (i.e., no in step 1001) but the system is not ahead of the second threshold time limit Tthres,0 (i.e., no in step 1003), the system may compensate based on the severity of the connection failure, including the following (step 1006): 1. Receive only the speech translation audio feed (skip transcription); 2. Transcribe the text from the edge device and send the text feed to the translation model for text-to-text translation only; or 3. Text-to-text buffering (e.g., wait until the sentence is complete before sending the transcribed text to the translation model).

[0137] FIG. 11 is an example of a method 1100 for generating a translated dubbed lip-sync in real time on an edge device based on streaming video content. This method may include the step 1101 of receiving streaming video content through a streaming service, but various other implementations are considered, including receiving content through wireless broadcasting, local network casting, short-range wireless communication content sharing, content broadcasting between devices, etc.

[0138] This method may include step 1102 of inputting the audio of an input stream into a speaker splitting model, and the model may be configured to separate the input audio into a background audio feed and individual speaker audio feeds containing speech in the first language.

[0139] This method may also include step 1103 of inputting the video of the input stream into a face detection model configured to output one or more frames containing a face.

[0140] This method may further include step 1104 of inputting an individual speaker audio feed and one or more truncated frames into an active speaker detection model, and the model may be configured to pair the individual speaker audio feed with one or more truncated frames corresponding to the active speaker.

[0141] This method may also include step 1105 of transmitting individual speaker audio feeds to a voice-to-voice translation module in which a second language is selected, and receiving voice audio translated into the second language.

[0142] This method may further include the step of inputting an individual speaker audio feed paired with the translated speech audio and one or more truncated frames into a lip-sync model 1106, and the lip-sync model may be configured to encode the translated audio to obtain a latent spatial representation, predict 3D mesh vertices to generate a mesh for the active speaker's face, detect mesh key points and align the generated mesh with one or more truncated frames to generate a video frame synchronized with the translated speech audio.

[0143] This method may also include step 1107, which displays a synchronized video frame in real time based on the generated video frame, wherein the facial movements of the active speaker are synchronized with the output translated speech audio.

[0144] The embodiments according to the present disclosure described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such computer program may be written on a computer-readable medium. Examples of computer-readable media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROM disks and DVD-ROM disks; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program code, such as ROM, RAM, and flash memory devices.

[0145] Meanwhile, the computer program may be specifically designed and produced for the purposes of the present invention, or it may be one that is widely known and available to those skilled in the art of computer software. Examples of program code include both machine code generated by a compiler and high-level code that can be executed by a computer using an interpreter.

[0146] The singular forms "a," "an," and "the" used in this disclosure (particularly in the appended claims) include both singular and plural expressions unless the context clearly indicates otherwise. Additionally, all numerical ranges described herein should be understood to include all sub-ranges contained therein (unless explicitly stated otherwise), and thus, the disclosed numerical ranges include all individual values ​​between the minimum and maximum values ​​of the numerical range.

[0147] The operations constituting the method of the present invention may be performed in an appropriate order unless explicitly described otherwise. The present invention is not necessarily limited to the order of operations presented in the description. All embodiments described herein or terms indicating such embodiments (e.g., "e.g.") are merely for the purpose of further explaining the present invention. Accordingly, it should be understood that the scope of the present invention is not limited by the exemplary embodiments described above or the use of such terms, unless limited by the appended claims. Furthermore, it will be apparent to those skilled in the art that various changes, substitutions, and modifications may be made within the scope of the appended claims or their equivalents.

[0148] Accordingly, the technical concept of the present disclosure is not limited to the embodiments mentioned above and should be considered to fall within the scope of the present disclosure.

Claims

1. A method for generating translated dubbed lip synchronization in real-time on an edge device based on streaming video content, A step of inputting the audio of video content into a speaker segmentation model configured to separate the input audio into a background audio feed and individual speaker audio feeds containing speech in the first language; A step of inputting a video of video content into a face detection model configured to output one or more cropped frames containing a face within one or more cropped frames; A step of inputting the individual speaker audio feed and the one or more truncated frames into an active speaker detection model to pair the individual speaker audio feed with the one or more truncated frames corresponding to the active speaker; A step of obtaining translated voice audio corresponding to the content translated into a second language of the individual speaker audio feed; The step of inputting the translated speech audio, the paired individual speaker audio feed, and the one or more truncated frames into a lip-sync model; the lip-sync model is configured to encode the translated audio to obtain a latent spatial representation and utilize a prediction model to generate lip-synced video frames including the facial movements of the active speaker synchronized with the translated speech audio; and The method includes the step of displaying the lip-synced video frames in real time based on the generated video frames, and when the translated voice audio is output, the facial movements of the active speaker are synchronized with the output translated voice audio. method.

2. In paragraph 1, the prediction model predicts 3D mesh vertices to generate a mesh of the active speaker's face, and The step of generating the lip-synced video frame above The step of detecting mesh key points corresponding to the latent spatial representation of the encoded translated audio and fitting the generated mesh to the one or more truncated frames method.

3. In paragraph 2, the step of generating the lip-synced video frames A step of extracting texture information of the active speaker's face based on one or more of the above cropped frames and A step of rendering the lip-synced video frames based on the aligned generated mesh and the extracted texture information. method.

4. In Paragraph 1, The above prediction model The system receives as input a reference image of the active speaker, the mouth of the active speaker blurred due to image noise in one or more cropped frames, and a latent embedding corresponding to the translated speech audio, and Performing inverse diffusion based on one or more blurred truncated frames to generate prediction frames containing visual image information of at least the active speaker's mouth while speaking the above translated voice audio. method.

5. In Paragraph 1, The active speaker detection model performs a convolution on the individual speaker audio feed and the one or more truncated frames to generate a one-dimensional temporal representation of the individual speaker audio feed and the one or more truncated frames, and Inputting the above one-dimensional temporal representation into multiple layers of gated cyclic units to match a portion of the individual speaker audio feed with the corresponding portion of one or more truncated frames. method.

6. In Paragraph 1, The lip-synced video frames are generated at a frame rate that matches or exceeds the frame rate of the streaming video content. method.

7. In Paragraph 1, Based on the fact that the lip-synced video frames are generated at a frame rate lower than a threshold value than the frame rate of the streaming video content, the lip-sync model is instructed to skip the generation of video frames for one or more frames within one or more truncated frames, and to generate the lip-synced video frames by calculating an average value based on adjacent generated video frames for the one or more skipped frames. method.

8. In Paragraph 1, The above-mentioned translated voice audio is obtained by transmitting the above-mentioned individual speaker audio feed to a voice-to-voice translation module in which a second language is selected, and receiving the voice audio translated into the second language. method.

9. A display device for generating real-time translated dubbing lip synchronization based on streaming video content, display; One or more processors; and Memory storing instructions that cause the display device to perform the following when executed by one or more processors: The audio of video content is input into a speaker segmentation model configured to separate the input audio into a background audio feed and individual speaker audio feeds containing speech in the first language, and Input a video of video content into a face detection model configured to output one or more cropped frames containing a face within one or more cropped frames, and The individual speaker audio feed and the one or more truncated frames are input into an active speaker detection model to pair the individual speaker audio feed with the one or more truncated frames corresponding to the active speaker, and Obtain translated voice audio corresponding to the content translated into the second language of the individual speaker audio feed above, and The translated speech audio, the paired individual speaker audio feed, and the one or more truncated frames are input into a lip-sync model, and the lip-sync model is configured to encode the translated audio to obtain a latent spatial representation and utilize a prediction model to generate lip-synced video frames including the facial movements of the active speaker synchronized with the translated speech audio; and Based on the generated video frames, the lip-synced video frames are displayed in real time, and when the translated voice audio is output, the facial movements of the active speaker are synchronized with the output translated voice audio. Display device.

10. In Paragraph 9, The above prediction model predicts 3D mesh vertices to generate a mesh of the active speaker's face, and Generating the above lip-synced video frames is Detecting mesh key points corresponding to the latent spatial representation of the encoded translated audio and fitting the generated mesh to the one or more cropped frames Display device.

11. In Paragraph 10, Generating the above lip-synced video frames is Extracting texture information of the active speaker's face based on one or more cropped frames, and rendering the lip-synced video frames based on the aligned generated mesh and the extracted texture information. Display device.

12. In Paragraph 9, The above prediction model The system receives as input a reference image of the active speaker, the mouth of the active speaker blurred due to image noise in one or more cropped frames, and a latent embedding corresponding to the translated speech audio, and Performing inverse diffusion based on one or more blurred truncated frames to generate prediction frames containing visual image information of at least the active speaker's mouth while speaking the above translated voice audio. Display device.

13. In Paragraph 9, The active speaker detection model performs a convolution on the individual speaker audio feed and the one or more truncated frames to generate a one-dimensional temporal representation of the individual speaker audio feed and the one or more truncated frames, and Inputting the above one-dimensional temporal representation into multiple layers of gated cyclic units to match a portion of the individual speaker audio feed with the corresponding portion of one or more truncated frames. Display device.

14. In Paragraph 9, The lip-synced video frames are generated at a frame rate that matches or exceeds the frame rate of the streaming video content. Display device.

15. In Paragraph 9, Based on the fact that the lip-synced video frames are generated at a frame rate lower than a threshold value than the frame rate of the streaming video content, the lip-sync model is instructed to skip the generation of video frames for one or more frames within one or more truncated frames, and to generate the lip-synced video frames by calculating an average value based on adjacent generated video frames for the one or more skipped frames. Display device.

16. In Paragraph 9, It further includes a transmitter and receiver, The above-mentioned translated voice audio is obtained by transmitting the individual speaker audio feed through the above-mentioned transceiver to a voice-to-voice translation module in which a second language is selected, and receiving the voice audio translated into the second language through the above-mentioned transceiver. Display device.

17. A non-transient computer-readable medium, wherein instructions stored in the medium cause the electronic device to perform the following when executed by one or more processors of the electronic device: The audio of video content is input into a speaker segmentation model configured to separate the input audio into a background audio feed and individual speaker audio feeds containing speech in the first language, and Input a video of video content into a face detection model configured to output one or more cropped frames containing a face within one or more cropped frames, and The individual speaker audio feed and the one or more truncated frames are input into an active speaker detection model to pair the individual speaker audio feed with the one or more truncated frames corresponding to the active speaker, and Obtain translated voice audio corresponding to the content translated into the second language of the individual speaker audio feed above, and The translated speech audio, the paired individual speaker audio feed, and the one or more truncated frames are input into a lip-sync model, and the lip-sync model is configured to encode the translated audio to obtain a latent spatial representation and utilize a prediction model to generate lip-synced video frames including the facial movements of the active speaker synchronized with the translated speech audio; and Based on the generated video frames, the lip-synced video frames are displayed in real time, and when the translated voice audio is output, the facial movements of the active speaker are synchronized with the output translated voice audio. Non-transient computer-readable medium.

18. In paragraph 17, the prediction model predicts 3D mesh vertices to generate a mesh of the active speaker's face, and Generating the above lip-synced video frames is Detecting mesh key points corresponding to the latent spatial representation of the encoded translated audio and fitting the generated mesh to the one or more cropped frames Generating the above lip-synced video frames is Extracting texture information of the active speaker's face based on one or more cropped frames, and rendering the lip-synced video frames based on the aligned generated mesh and the extracted texture information. Non-transient computer-readable medium.

19. In Paragraph 17, The above prediction model The system receives as input a reference image of the active speaker, the mouth of the active speaker blurred due to image noise in one or more cropped frames, and a latent embedding corresponding to the translated speech audio, and Performing inverse diffusion based on one or more blurred truncated frames to generate prediction frames containing visual image information of at least the active speaker's mouth while speaking the above translated voice audio. The active speaker detection model performs a convolution on the individual speaker audio feed and the one or more truncated frames to generate a one-dimensional temporal representation of the individual speaker audio feed and the one or more truncated frames, and Inputting the above one-dimensional temporal representation into multiple layers of gated cyclic units to match a portion of the individual speaker audio feed with the corresponding portion of one or more truncated frames. Non-transient computer-readable medium.

20. In paragraph 17, the lip-synced video frames are generated at a frame rate that matches or exceeds the frame rate of the streaming video content. Non-transient computer-readable medium.

Images