Video generation method and apparatus, device, and storage medium

Abstract

Embodiments of the present disclosure relate to a video generation method and apparatus, a device, and a computer-readable storage medium. The method provided herein comprises: acquiring a reference image and at least one control signal; and using a model to generate a target video on the basis of the reference image and the at least one control signal, wherein the model comprises a first attention layer, and the first attention layer is configured to: update an input video feature and an input control feature on the basis of an attention mechanism, to determine an intermediate video feature and an intermediate control feature; acquire a first group of sub-features of the intermediate video feature and / or a second group of sub-features of the intermediate control feature; and determine an output video feature and an output control feature of an attention layer on the basis of the first group of sub-features and / or the second group of sub-features.

Description

Methods, apparatus, devices and storage media for generating video

[0001] This application claims priority to Chinese Patent Application No. 202510096924.4, filed on January 21, 2025, entitled "Method, Apparatus, Device and Storage Medium for Generating Video", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, and computer-readable storage media for generating video. Background Technology

[0003] With the development of computer technology, some machine learning techniques allow users to drive images through control signals, thereby generating dynamic video content. For example, users can input audio signals to drive digital humans to perform actions corresponding to the audio signals. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for generating a video is provided. The method includes: acquiring a reference image and at least one control signal; and generating a target video based on the reference image and at least one control signal using a model, wherein the model includes a first attention layer configured to: update input video features and input control features based on an attention mechanism to determine intermediate video features and intermediate control features; acquire a first set of sub-features of the intermediate video features and / or a second set of sub-features of the intermediate control features; and determine output video features and output control features of the attention layer based on the first set of sub-features and / or the second set of sub-features.

[0005] In a second aspect of this disclosure, an apparatus for generating video is provided. The apparatus includes: an acquisition module configured to acquire a reference image and at least one control signal; and a generation module configured to generate a target video based on the reference image and at least one control signal using a model, wherein the model includes a first attention layer configured to: update input video features and input control features based on an attention mechanism to determine intermediate video features and intermediate control features; acquire a first set of sub-features of the intermediate video features and / or a second set of sub-features of the intermediate control features; and determine output video features and output control features of the attention layer based on the first set of sub-features and / or the second set of sub-features.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method of the first aspect.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure may be implemented;

[0012] Figure 2 illustrates an example model for generating video according to some embodiments of the present disclosure;

[0013] Figure 3 illustrates a flowchart of an example process for generating video according to some embodiments of the present disclosure;

[0014] Figure 4 shows a schematic structural block diagram of an example apparatus for generating video according to some embodiments of the present disclosure; and

[0015] Figure 5 shows a block diagram of an electronic device capable of implementing several embodiments of the present disclosure. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0020] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0021] Embodiments of this disclosure propose a scheme for generating video. The scheme includes: acquiring a reference image and at least one control signal; and generating a target video based on the reference image and at least one control signal using a model, wherein the model includes a first attention layer configured to: update input video features and input control features based on an attention mechanism to determine intermediate video features and intermediate control features; acquire a first set of sub-features of the intermediate video features and / or a second set of sub-features of the intermediate control features; and determine the output video features and output control features of the attention layer based on the first set of sub-features and / or the second set of sub-features.

[0022] In this way, the embodiments of this disclosure can improve the effectiveness of feature representation, thereby improving the quality of video generation.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example Environment

[0025] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in Figure 1, the example environment 100 may include an electronic device 110.

[0026] As shown in Figure 1, the electronic device 110 can acquire a control signal 130 and a reference image 120. The control signal 130 may include, for example, one or more modal control signals, such as text content, audio content, video content, etc. As will be described in detail below, the electronic device 110 can provide the reference image 120 and the control signal 130 to the model 140 to generate a target video 150.

[0027] As an example, the target video 150 may include motion content associated with a preset object in the reference image 120. As an example, the motion content corresponds to the received multi-modal control signals 130.

[0028] For example, control signal 130 may include reference video content and audio content. Accordingly, motion content in target video 150 may include body movements generated based on skeletal information from the reference video content and mouth movements generated based on the audio content.

[0029] In some embodiments, electronic device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 may also support any type of user-facing interface (such as "wearable" circuitry).

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0032] Example Model

[0033] Figure 2 illustrates an example model 200 for generating video according to some embodiments of the present disclosure. The specific process of generating the target video by model 200 will be described below with reference to Figure 2.

[0034] As shown in Figure 2, model 200 may include multiple attention layers, such as attention layer 216. As an example, model 220 may include a suitable generative model, such as a diffusion model.

[0035] As shown in Figure 2, model 200 can acquire input video features 204 and input control features 214.

[0036] For the first attention layer in model 200, the input video features 204 and input control features 214 of this attention layer can be determined based on the input information of model 200. Specifically, the input information of model 200 may include a reference image to be driven and at least one control signal.

[0037] As shown in Figure 2, at least one control signal may include control signals of multiple modes. Model 200 can acquire audio control signal 206, which may be based on reference audio content provided by the user.

[0038] Alternatively or additionally, model 200 may acquire text control signals 208, which may include, for example, prompts entered by the user.

[0039] Additionally or alternatively, model 200 may also acquire object control signals 210, which may include, for example, object features associated with a preset object (e.g., a face) in a reference image to ensure content consistency in video drive.

[0040] Alternatively or additionally, model 200 may also acquire attribute control signal 212, which may include attribute description features determined based on the reference image. For example, such attribute description features may describe the scene corresponding to the reference image, such as an indoor scene or an outdoor scene.

[0041] In some embodiments, model 200 may also acquire one or more of the above control signals and determine their corresponding features as input control features of attention layer 216.

[0042] On the other hand, for the first attention layer 216, the video features 204 may be determined, for example, based on the reference image to be driven.

[0043] Conversely, for subsequent attention layers, the corresponding input video features and input control features can be the updated output video features and output control features of the previous attention layer.

[0044] As shown in Figure 2, the attention layer 216 can update the input video features 204 and the input control features 214 based on the attention mechanism to obtain intermediate video features and intermediate control features.

[0045] In some embodiments, as shown in FIG2, intermediate video features may be provided to feature routing unit 218 to determine a first set of sub-features related to the intermediate video features.

[0046] Specifically, model 200 can use a first set of feature transformation units corresponding to multiple preset image regions to process intermediate video features in order to determine a first set of sub-features.

[0047] As shown in the figure, the first set of feature transformation units may include multiple expert models, such as a face expert model 222, a body expert model 224, a hand expert model 226, and a background expert model 228. As an example, the expert models may be implemented using a multilayer perceptron (MLP) or other suitable models.

[0048] As an example, the face expert model 222 can be used to extract facial sub-features associated with the face region from intermediate video features; the body expert model 224 can be used to extract body sub-features associated with the body region from intermediate video features; the hand expert model 226 can be used to extract hand sub-features associated with the hand region from intermediate video features; and the background expert model 228 can be used to extract background sub-features associated with the background region from intermediate video features.

[0049] Furthermore, model 200 can determine the output video features 230 of attention layer 216 based on the first set of sub-features. In this way, embodiments of this disclosure can ensure that the output video features 230 cover network parameters corresponding to different image regions, thereby ensuring high generation quality for different regions.

[0050] Alternatively or additionally, the intermediate control features updated by the attention layer 216 may also be provided to the feature routing unit 220 to determine a second set of sub-features.

[0051] Specifically, model 200 can use a second set of feature transformation units corresponding to multiple preset categories of control signals to process intermediate control features in order to determine a second set of sub-features.

[0052] As shown in the figure, the second set of feature transformation units may include multiple expert models, such as audio expert model 232, text expert model 234, object expert model 236, and attribute expert model 238. As an example, the expert models may be implemented using a multilayer perceptron (MLP) or other suitable models.

[0053] As an example, audio expert model 232 can be used to extract audio sub-features 240 associated with audio control signals from intermediate control features; text expert model 234 can be used to extract text sub-features 242 associated with text control signals from intermediate control features; object expert model 236 can be used to extract object sub-features 244 associated with object control signals from intermediate control features; and attribute expert model 238 can be used to extract attribute sub-features 246 associated with attribute control signals from intermediate control features.

[0054] Furthermore, model 200 can determine the output control features of attention layer 216 based on the second set of sub-features. In this way, embodiments of this disclosure can ensure that the output control features 230 can maintain the expression of control signals for different modalities during feature transmission, thereby improving the generation quality of video content.

[0055] In some embodiments, for each attention layer 216, it may use only multiple expert models to extract video-related sub-features, or it may use only multiple expert models to extract control signal-related sub-features, or it may use expert models to perform feature decomposition for both video and control signals.

[0056] In some embodiments, different expert models can have mutually independent model parameters, thereby avoiding conflicts between different modal information or conflicts between image information from different regions.

[0057] In some embodiments, during the training phase of model 200, the model parameters corresponding to each expert model can also be trained collaboratively.

[0058] Specifically, for the expert model associated with video features, prior image segmentation information and / or pose information can be obtained. Specifically, a set of training sub-features generated by the first set of feature transformation units (e.g., face expert model 222, body expert model 224, hand expert model 226, and background expert model 228) can be compared with a first set of reference features corresponding to multiple preset image regions (e.g., face region, body region, hand region, and background region), and the training loss of model 200 can be determined based on the feature differences.

[0059] As an example, the first set of reference features can be determined based on prior image segmentation information and / or pose information. For instance, keypoint information and / or mask information corresponding to multiple preset image regions (e.g., face region, body region, hand region, and background region) can be obtained to determine the first set of reference features. In this way, prior information can be used to ensure that each expert model can extract sub-features of the corresponding region.

[0060] Similarly, for expert models associated with control features, a second set of reference features can be determined based on the input training control signal. Further, the difference between the second set of training sub-features output by the second set of feature transformation units (e.g., audio expert model 232, text expert model 234, object expert model 236, and attribute expert model 238) during training and the second set of reference features can be determined, and the training loss of model 200 can be determined based on this difference.

[0061] Furthermore, the parameters of each unit in Model 200 can be adjusted collaboratively based on the training loss of Model 200, such as the independent network parameters corresponding to each expert model.

[0062] Therefore, by training model 200, embodiments of this disclosure enable different expert models to extract related sub-features, thereby allowing each category of sub-features to be better represented during feature transfer.

[0063] Correspondingly, in the application stage of the model, the trained expert models can be used to extract video features from different regions and / or control features from different modalities, thereby improving the quality of feature representation.

[0064] Furthermore, Model 200 can generate the target video based on the video features output by multiple attention layers, that is, the video content obtained based on the reference image driven by the control signal.

[0065] As an example, a user can provide only text content to drive a reference image. Alternatively, a user can provide only audio content to drive a reference image. Or, a user can provide only skeleton information (e.g., reference video content) to drive a reference image. Alternatively, a user can provide a combination of these different modalities to collectively drive the reference image, thereby increasing the flexibility of video-driven processing.

[0066] Example process

[0067] Figure 3 illustrates a flowchart of an example process 300 for generating video according to some embodiments of the present disclosure. Process 300 can be implemented in an electronic device 110.

[0068] As shown in the figure, in box 310, electronic device 110 acquires a reference image and at least one control signal.

[0069] In box 320, electronic device 110 uses a model to generate a target video based on a reference image and at least one control signal, wherein the model includes a first attention layer configured to: update input video features and input control features based on an attention mechanism to determine intermediate video features and intermediate control features; acquire a first set of sub-features of the intermediate video features and / or a second set of sub-features of the intermediate control features; and determine the output video features and output control features of the attention layer based on the first set of sub-features and / or the second set of sub-features.

[0070] In some embodiments, the model further includes a second attention layer connected to the first attention layer, and the output video features and output control features of the first attention layer are provided as input video features and input control features of the second attention layer.

[0071] In some embodiments, the first set of sub-features corresponds to a plurality of preset image regions, and the process 300 further includes: processing intermediate video features using a first set of feature conversion units corresponding to the plurality of preset image regions to determine the first set of sub-features.

[0072] In some embodiments, the training loss of the model includes the difference between a first set of training sub-features generated by a first set of feature transformation units and a first set of reference features corresponding to a plurality of preset image regions.

[0073] In some embodiments, the first set of reference features is determined based on key point information or mask information corresponding to multiple preset image regions.

[0074] In some embodiments, the second set of sub-features corresponds to multiple preset categories of control signals, and the process 300 further includes: processing intermediate control features using a second set of feature conversion units corresponding to the multiple preset categories of control signals to determine the second set of sub-features.

[0075] In some embodiments, the training loss of the model includes the difference between a second set of training sub-features generated by a second set of feature transformation units and a second set of reference features corresponding to control signals of multiple preset categories.

[0076] In some embodiments, the target video includes motion content associated with a preset object in a reference image, the motion content corresponding to at least one control signal.

[0077] In some embodiments, at least one control signal includes at least one of the following: an audio control signal; a text control signal; an object control signal, the object control signal including object features associated with a preset object in the reference image; and an attribute control signal, the attribute control signal including attribute description features determined based on the reference image.

[0078] Example devices and equipment

[0079] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. Figure 4 shows a schematic structural block diagram of an example apparatus 400 for generating video according to certain embodiments of this disclosure. Apparatus 400 may be implemented as or included in electronic device 110. The various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0080] As shown in Figure 4, the device 400 includes an acquisition module 410 configured to acquire a reference image and at least one control signal; and a generation module 420 configured to generate a target video based on the reference image and at least one control signal using a model, wherein the model includes a first attention layer, the first attention layer being configured to: update input video features and input control features based on an attention mechanism to determine intermediate video features and intermediate control features; acquire a first set of sub-features of the intermediate video features and / or a second set of sub-features of the intermediate control features; and determine the output video features and output control features of the attention layer based on the first set of sub-features and / or the second set of sub-features.

[0081] In some embodiments, the model further includes a second attention layer connected to the first attention layer, and the output video features and output control features of the first attention layer are provided as input video features and input control features of the second attention layer.

[0082] In some embodiments, the first set of sub-features corresponds to a plurality of preset image regions, and the apparatus 400 further includes a first determining module configured to: process intermediate video features using a first set of feature conversion units corresponding to the plurality of preset image regions to determine the first set of sub-features.

[0083] In some embodiments, the training loss of the model includes the difference between a first set of training sub-features generated by a first set of feature transformation units and a first set of reference features corresponding to a plurality of preset image regions.

[0084] In some embodiments, the first set of reference features is determined based on key point information or mask information corresponding to multiple preset image regions.

[0085] In some embodiments, the second set of sub-features corresponds to multiple preset categories of control signals, and the device 400 further includes a second determining module configured to: process intermediate control features using a second set of feature conversion units corresponding to the multiple preset categories of control signals to determine the second set of sub-features.

[0086] In some embodiments, the training loss of the model includes the difference between a second set of training sub-features generated by a second set of feature transformation units and a second set of reference features corresponding to control signals of multiple preset categories.

[0087] In some embodiments, the target video includes motion content associated with a preset object in a reference image, the motion content corresponding to at least one control signal.

[0088] In some embodiments, at least one control signal includes at least one of the following: an audio control signal; a text control signal; an object control signal, the object control signal including object features associated with a preset object in the reference image; and an attribute control signal, the attribute control signal including attribute description features determined based on the reference image.

[0089] Figure 5 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 500 shown in Figure 5 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 500 shown in Figure 5 can be used to implement the electronic device 110 of Figure 1.

[0090] As shown in Figure 5, the electronic device 500 is in the form of a general-purpose electronic device. Components of the electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. The processor 510 may be a physical or virtual processor and is capable of performing various processes according to the programs stored in the memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 500.

[0091] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0092] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0093] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0094] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0095] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0096] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0098] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0100] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating video, comprising: Acquire a reference image and at least one control signal; as well as The target video is generated using the model based on the reference image and the at least one control signal. The model includes a first attention layer, which is configured as follows: Based on the attention mechanism, the input video features and input control features are updated to determine the intermediate video features and intermediate control features; Obtain the first set of sub-features of the intermediate video features and / or the second set of sub-features of the intermediate control features; and Based on the first set of sub-features and / or the second set of sub-features, the output video features and output control features of the attention layer are determined.

2. The method of claim 1, wherein the model further comprises a second attention layer connected to the first attention layer, and the output video features and the output control features of the first attention layer are provided as input video features and input control features of the second attention layer.

3. The method according to any one of claims 1 to 2, wherein the first set of sub-features corresponds to a plurality of preset image regions, and obtaining the first set of sub-features of the intermediate video features includes: The intermediate video features are processed using a first set of feature conversion units corresponding to the plurality of preset image regions to determine the first set of sub-features.

4. The method according to claim 3, wherein the training loss of the model includes the difference between the first set of training sub-features generated by the first set of feature transformation units and the first set of reference features corresponding to the plurality of preset image regions.

5. The method according to claim 4, wherein the first set of reference features is determined based on key point information or mask information corresponding to the plurality of preset image regions.

6. The method according to any one of claims 1 to 5, wherein the second set of sub-features corresponds to a plurality of preset categories of control signals, and obtaining the second set of sub-features of the intermediate video features includes: The intermediate control features are processed using a second set of feature conversion units corresponding to the multiple preset categories of control signals to determine the second set of sub-features.

7. The method according to claim 6, wherein the training loss of the model includes the difference between the second set of training sub-features generated by the second set of feature transformation units and the second set of reference features corresponding to the control signals of the plurality of preset categories.

8. The method according to any one of claims 1 to 7, wherein the target video includes motion content associated with a preset object in the reference image, the motion content corresponding to the at least one control signal.

9. The method according to any one of claims 1 to 8, wherein the at least one control signal comprises at least one of the following: Audio control signals; Text control signals; An object control signal, the object control signal including object features associated with a preset object in the reference image; An attribute control signal, the attribute control signal including attribute description features determined based on the reference image.

10. An apparatus for generating video, comprising: The acquisition module is configured to acquire a reference image and at least one control signal; as well as The generation module is configured to generate a target video using a model based on the reference image and the at least one control signal. The model includes a first attention layer, which is configured as follows: Based on the attention mechanism, the input video features and input control features are updated to determine the intermediate video features and intermediate control features; Obtain the first set of sub-features of the intermediate video features and / or the second set of sub-features of the intermediate control features; and Based on the first set of sub-features and / or the second set of sub-features, the output video features and output control features of the attention layer are determined.

11. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images