Video generation method and apparatus, device, medium, and program product
By acquiring scene images and mapping user interaction operations to the dynamic changes of the virtual camera, the camera parameter matrix is determined, which solves the problem of inaccurate viewpoint control in video generation and improves the controllability and flexibility of video generation.
Patent Information
- Application Number
- PCT/CN2026/081267
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-03-04
- Publication Date
- 2026-11-12
AI Technical Summary
In existing technologies, when intelligently generating videos, it is difficult to precisely control the viewing angle of the video. The viewing angle of the generated video has a certain degree of randomness, resulting in low controllability.
By acquiring scene images and detecting user interaction operations, the interaction operations are mapped to dynamic changes in the virtual camera, multiple camera parameter matrices of the virtual camera are determined, and video clips are generated based on these parameters, achieving precise control over the video viewpoint.
It improves the controllability of video generation, allowing the generated video clips to be flexibly controlled from the perspective based on user interaction, increasing the flexibility and smoothness of video generation, and making the generated perspectives richer and more varied.
Smart Images

Figure CN2026081267_12112026_PF_FP_ABST
Abstract
Description
Video generation methods, apparatus, equipment, media and program products
[0001] This application claims priority to Chinese Patent Application No. 202510599845.5, filed on May 9, 2025, entitled "Video Generation Method, Apparatus, Electronic Device and Storage Medium". Technical Field
[0002] This application relates to the field of vision, and in particular to a video generation method, apparatus, device, medium, and program product.
[0003] Background of the Invention
[0004] With the development of computer technology, the ways to process videos are becoming increasingly diverse. Currently, videos can be intelligently generated based on artificial intelligence technology. For example, by specifying a reference image, a video that matches the content of the reference image can be intelligently generated.
[0005] In related technologies, when generating videos intelligently, the corresponding videos can only be generated randomly based on the reference images provided by the user. However, it is difficult to accurately control the perspective of the video. The perspective of the generated video has a certain degree of randomness, so the controllability of video generation is low. Summary of the Invention
[0006] This application provides a video generation method, apparatus, device, medium, and program product, which can improve the controllability of video generation. The technical solution is as follows:
[0007] On the one hand, a video generation method is provided, executed by an electronic device, the method comprising:
[0008] Acquire a first scene image, which is used to indicate the scene contained in the first video segment to be generated;
[0009] When the first interaction operation is detected, the first interaction operation is mapped to the first dynamic change information of the virtual camera, and the first dynamic change information is used to indicate the movement mode of the virtual camera when shooting the scene.
[0010] Based on the first dynamic change information, a plurality of first camera parameter matrices of the virtual camera are determined, each first camera parameter matrix corresponding to a video frame in the first video segment, and the first camera parameter matrix representing the pose of the virtual camera when the video frame is captured; and,
[0011] The first video segment is generated based on the first scene image and the plurality of first camera parameter matrices.
[0012] On the other hand, a video generation apparatus is provided, the apparatus comprising:
[0013] The acquisition module is used to acquire a first scene image, which is used to indicate the scene contained in the first video segment to be generated;
[0014] The first conversion module is used to map the first interactive operation into the first dynamic change information of the virtual camera when the first interactive operation is detected. The first dynamic change information is used to indicate the movement mode of the virtual camera when shooting the scene.
[0015] The second conversion module is used to determine multiple first camera parameter matrices of the virtual camera based on the first dynamic change information. Each first camera parameter matrix corresponds to a video frame in the first video segment, and the first camera parameter matrix represents the pose of the virtual camera when the video frame is captured.
[0016] The generation module is used to generate the first video segment based on the first scene image and the plurality of first camera parameter matrices.
[0017] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed by the video generation method described above.
[0018] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the video generation method as described above.
[0019] On the other hand, a computer program product is provided, including a computer program loaded and executed by a processor to perform the operations performed by the video generation method as described above.
[0020] Brief description of the attached figures
[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 is a schematic diagram of a video generation system provided in an embodiment of this application;
[0023] Figure 2 is a flowchart of a video generation method provided in an embodiment of this application;
[0024] Figure 3 is a flowchart of a video generation method provided in an embodiment of this application;
[0025] Figure 4 is a flowchart of a video generation method provided in an embodiment of this application;
[0026] Figure 5 is a schematic diagram of the structure of a video generation model provided in an embodiment of this application;
[0027] Figure 6 is a flowchart of a video generation method provided in an embodiment of this application;
[0028] Figure 7 is a flowchart of a video generation method provided in an embodiment of this application;
[0029] Figure 8 is a schematic diagram of a video generation method provided in an embodiment of this application;
[0030] Figure 9 is a flowchart of a training method for a video generation model provided in an embodiment of this application;
[0031] Figure 10 is a flowchart of another training method for a video generation model provided in an embodiment of this application;
[0032] Figure 11 is a schematic diagram of a game scene video clip provided in an embodiment of this application;
[0033] Figure 12 is a schematic diagram of a video generation device provided in an embodiment of this application;
[0034] Figure 13 is a schematic diagram of another video generation device provided in an embodiment of this application;
[0035] Figure 14 is a schematic diagram of the structure of a terminal provided in an embodiment of this application;
[0036] Figure 15 is a schematic diagram of the structure of a server provided in an embodiment of this application.
[0037] Implementation
[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.
[0039] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first scene image may be referred to as a second scene image, and similarly, a second scene image may be referred to as a first scene image.
[0040] "At least one" refers to one or more video frames. For example, at least one video frame can be any integer number of video frames greater than or equal to one, such as one video frame, two video frames, three video frames, etc. "Multiple" refers to two or more video frames. For example, multiple video frames can be any integer number of video frames greater than or equal to two, such as two video frames, three video frames, etc. "Each" refers to each of the at least one video frame. For example, each video frame refers to each of the multiple video frames. If the multiple video frames are three video frames, then each video frame refers to each of the three video frames.
[0041] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have all been fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0042] For example, the scene images, interactive operations, video clips, etc. involved in this application are all fully authorized by the user or relevant parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0043] For ease of understanding, the concepts involved in the embodiments of this application are explained below.
[0044] (1) Video generation model: Given a starting frame (e.g., the scene image below), the video generation model will output a video segment that meets expectations based on text and other information (e.g., the camera parameter matrix below).
[0045] (2) Continuous motion space: Unlike encoding the actions of the virtual camera (such as forward, backward, left turn, right turn, etc.) into a discrete codebook, this application defines a continuous motion space for the virtual camera (such as the dynamic change information below), maps different interactive operations into unified dynamic change information, and realizes precise control of the pose of the virtual camera in the continuous motion space.
[0046] (3) Dynamic interactive video generation: This refers to the process of converting user-performed interactive operations into dynamic changes in a virtual camera in real time. Based on the pose of the virtual camera, the video generation model dynamically adjusts the viewpoint in the video clip, achieving low-latency closed-loop interaction between the user and the video generation model, and completing real-time control of complex movements. Furthermore, dynamism means that some content in the generated video clip (such as pedestrians and vehicles) also has dynamic effects that conform to real-world physical logic.
[0047] The video generation method provided in this application can be used in electronic devices. Optionally, the electronic device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.
[0048] Figure 1 is a schematic diagram of a video generation system provided in an embodiment of this application. Referring to Figure 1, the video generation system includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.
[0049] Terminal 101 has a client 111 installed and running. This client 111 can be a social application client, online payment client, online shopping client, game client, medical service client, video client, etc. When terminal 101 runs client 111, the user interface of client 111 is displayed on the screen of terminal 101. Terminal 101 is the terminal used by user 121.
[0050] Optionally, terminal 101 can refer to one of a plurality of terminals. As those skilled in the art will know, the number of terminals can be more or less. For example, there may be only one terminal, or there may be six, eight or more terminals. This application does not limit the number of terminals or the type of device.
[0051] Figure 1 shows only one terminal, but in different embodiments, multiple other terminals 103 can access the server 102. Optionally, at least one terminal 103 is a terminal corresponding to the developer, on which a client development and editing platform is installed. The developer can edit and update the client on the terminal 103 and transmit the updated client installation package to the server 102 via wired or wireless network. The terminal 101 can download the client installation package from the server 102 to update the client.
[0052] Terminal 101 and other terminals 103 are connected to server 102 via wired or wireless networks.
[0053] Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 is used to provide backend services to clients. Optionally, server 102 undertakes the primary computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the primary computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.
[0054] In this embodiment, terminal 101 receives a scene image uploaded by a user. After detecting an interactive operation performed by the user, it maps the interactive operation to dynamic change information of a virtual camera and sends a video generation request to server 102. The video generation request carries the scene image and the dynamic change information of the virtual camera. After receiving the video generation request, server 102 uses the method provided in this embodiment to convert the dynamic change information of the virtual camera into multiple camera parameter matrices of the virtual camera. Based on the scene image and the multiple camera parameter matrices, it generates a video clip, and server 102 returns the video clip to terminal 101.
[0055] It should be noted that the above implementation environment is only an example. The method provided in this application embodiment can also be executed by the terminal 101 or the server 102 alone, or by other electronic devices. This application embodiment does not limit this.
[0056] In related technologies, intelligent video generation can only randomly generate corresponding videos based on user-provided reference images. However, it is difficult to precisely control the viewing angle of the generated video, resulting in a degree of randomness and low controllability. Another related technology provides discrete action command sets (e.g., forward, backward, left turn, and right turn) to control the camera viewpoint. However, due to limitations in the number and types of action command sets, it can only be applied to specific video generation tasks, exhibiting significant limitations.
[0057] Figure 2 is a flowchart of a video generation method provided in an embodiment of this application. As shown in Figure 2, the method includes at least one of the following steps:
[0058] 1. The user uploads a scene image 201 and performs an interactive operation 202. The scene image 201 indicates the scene contained in the video clip to be generated. The interactive operation 202 can be one of several preset interactive operations, used to control the shooting angle of the virtual camera when generating the video clip. Specifically, by performing the interactive operation 202, the user controls the virtual camera's movement to simulate shooting, obtaining a video clip with a specific perspective.
[0059] 2. Map the user-executed interactive operation 202 from the discrete operation space to parameters in the continuous action space. Specifically, map the user-executed interactive operation 202 to the dynamic change information 203 of the virtual camera, convert the dynamic change information 203 into a camera parameter matrix 204 of the virtual camera in multiple video frames, and convert the multiple camera parameter matrices 204 into frame-level directional features 205 of multiple video frames.
[0060] The dynamic change information 203 is used to indicate the movement of the virtual camera during the shooting scene, that is, the camera movement of the virtual camera can be controlled through the dynamic change information 203 during the simulated shooting scene. The camera parameter matrix 204 of the virtual camera in each video frame is used to indicate the pose of the virtual camera in each video frame. The frame-level orientation feature 205 of each video frame is used to reflect the direction of each pixel in the video frame projected into world space under a specified camera pose. World space is a globally unified three-dimensional reference system, which is the absolute position reference of all cameras. Its core function is to transform local observations (such as pixels in video frames) from different viewpoints (different camera poses) to the same coordinate system, achieving spatial consistency across frames and viewpoints. World space uses the world coordinate system to give each scene point a unique (x, y, z) coordinate. Establishing a world coordinate system includes determining the origin and the directions of the x, y, and z axes.
[0061] 3. Input the scene image 201 and the frame-level orientation features 205 of multiple video frames into the video generation model 206 to obtain the video segment 207 output by the video generation model 206. For example, the video generation model 206 can be any type of diffusion model, such as an AI-based video generation model.
[0062] In this embodiment, discrete interactive operations are mapped to a unified continuous action space. Users can simulate the movement of a virtual camera in a scene by performing simple interactive operations, similar to camera movement in the real world. This simulates controlling the virtual camera to shoot in a specified way within the scene, thereby controlling the perspective of the generated video clips. This achieves flexible control of the perspective of intelligently generated video clips based on interactive operations, improving the controllability of video generation. Furthermore, compared to directly using discrete action instruction sets for control, mapping discrete interactive operations to a continuous action space in this embodiment improves the flexibility and smoothness of video generation control, making the perspectives in the video richer and more varied, and improving the quality of video generation.
[0063] Figure 3 is a flowchart of a video generation method provided in an embodiment of this application. This embodiment is executed by an electronic device. Referring to Figure 3, the method includes:
[0064] 301. The electronic device acquires a first scene image, which is used to indicate the scene contained in the first video segment to be generated.
[0065] The electronic device acquires a first scene image, which serves as reference information for generating a video clip and indicates the scene contained in the first video clip to be generated. That is, the scene contained in the first video clip to be generated matches the scene shown in the first scene image.
[0066] The scene shown in the first scene image can include any object. The objects in the scene can be static objects or dynamic objects. Static objects are immovable objects, while dynamic objects are movable objects. For example, the objects in the scene include static objects such as trees, houses, rivers, and grasslands. The objects in the scene can also include dynamic objects such as people, animals, and vehicles.
[0067] The number of the first scene images can be one or more, and the number of the first scene images is not limited in this embodiment.
[0068] 302. When the electronic device detects the first interactive operation, it maps the first interactive operation to the first dynamic change information of the virtual camera, and the first dynamic change information is used to indicate the movement mode of the virtual camera in the shooting scene.
[0069] The electronic device detects whether the user performs a preset interactive operation. If the user performs a preset first interactive operation, the first interactive operation is mapped to the first dynamic change information of the virtual camera. In this embodiment, the process of intelligently generating video clips is simulated as the process of obtaining video clips by shooting a scene with a virtual camera. The virtual camera is the camera used to shoot in the scene provided by the first scene image.
[0070] The first dynamic change information is used to indicate the movement of the virtual camera when shooting the scene. That is, the first dynamic change information can reflect the continuous movement of the virtual camera during the shooting process. Based on the first dynamic change information, the virtual camera can be controlled to move during the shooting process, thereby simulating the process of controlling the virtual camera to perform camera movement.
[0071] The movement methods include translation speed and rotation angle.
[0072] 303. The electronic device determines multiple first camera parameter matrices of the virtual camera based on the first dynamic change information. Each first camera parameter matrix corresponds to a video frame in the first video segment. The first camera parameter matrix represents the pose of the virtual camera when the video frame is captured.
[0073] Since the first dynamic change information is used to indicate the movement of the virtual camera in the shooting scene, the movement of the virtual camera during the shooting process can be controlled based on the first dynamic change information. Therefore, based on the first dynamic change information, the first camera parameter matrix of the virtual camera in each of the multiple video frames can be determined. The first camera parameter matrix of each video frame represents the pose of the virtual camera when shooting that video frame in the scene.
[0074] The pose of the virtual camera includes its position and orientation. The position of the virtual camera is changed by translating it, and the orientation is changed by rotating it. In one embodiment, the position can be represented as three-dimensional spatial coordinates (x, y, z), and the orientation / direction can be represented as a 3×3 rotation matrix. Therefore, unifying the position and orientation into a single camera parameter matrix can be represented by a 4×4 homogeneous transformation matrix.
[0075] In this embodiment, the discrete interactive operations performed by the user are ultimately converted into camera parameter matrices of the virtual camera on multiple consecutive video frames, realizing the control of the virtual camera in a continuous action space, which is beneficial for achieving generalizable and fine-grained interactive video generation.
[0076] 304. The electronic device generates a first video segment based on the first scene image and multiple first camera parameter matrices.
[0077] After determining the first scene image and multiple first camera parameter matrices of the virtual camera, the electronic device generates a first video segment based on the first scene image and multiple first camera parameter matrices. The first video segment includes images captured by the virtual camera in the scene displayed in the first scene image according to the poses indicated by the multiple first camera parameter matrices.
[0078] The method provided in this application, when intelligently generating video clips, allows users to perform interactive operations as needed after providing scene images. These user-performed interactive operations can be mapped to dynamic change information of a virtual camera, controlling the movement of the virtual camera during shooting. Based on this dynamic change information, the camera parameter matrix of the virtual camera in each video frame can be determined, representing the pose of the virtual camera in each frame. Then, based on the scene image and multiple camera parameter matrices of the virtual camera, a video clip captured by the virtual camera moving in that scene according to the specified movement method can be generated. Therefore, by performing interactive operations, users can control the camera movement of the virtual camera during video clip generation, thereby controlling the viewpoint displayed in the generated video clip. This achieves flexible control of the viewpoint of the intelligently generated video clip based on interactive operations, improving the controllability of video generation.
[0079] The above is only a brief description of the video generation method provided in the embodiments of this application. For detailed implementation processes, please refer to the following embodiments. Figure 4 is a flowchart of a video generation method provided in an embodiment of this application. This embodiment is executed by an electronic device. Referring to Figure 4, the method includes:
[0080] 401. The electronic device acquires a first scene image, which is used to indicate the scene contained in the first video segment to be generated.
[0081] The electronic device acquires a first scene image, which serves as reference information for generating a video clip and indicates the scene contained in the first video clip to be generated. That is, the scene contained in the first video clip to be generated matches the scene shown in the first scene image.
[0082] The first scene image can be a real-life image or an intelligently generated image; this application embodiment does not limit the type of image.
[0083] 402. When the electronic device detects the first interactive operation, it maps the first interactive operation to the first dynamic change information of the virtual camera, and the first dynamic change information is used to indicate the movement mode of the virtual camera in the shooting scene.
[0084] In one possible implementation, the electronic device records a variety of preset interactive operations. These operations are used to control the generation of video clips. Each interactive operation corresponds to different dynamic change information. Therefore, by executing different interactive operations, the virtual camera can be controlled to shoot under different movement modes, thereby controlling the generation of video clips from different perspectives. The various interactive operations include a first interactive operation. If the electronic device detects that the user has executed the first interactive operation, it maps this first interactive operation to the first dynamic change information of the virtual camera.
[0085] Optionally, the multiple interactive operations include triggering operations on multiple different controls. These multiple controls can be virtual buttons displayed on the electronic device or physical buttons on the input device to which the electronic device is connected, such as the "W" key, "A" key, "S" key, "D" key, etc. on a keyboard.
[0086] For example, an interactive operation can be a trigger operation on a single control, or it can be a trigger operation on multiple controls. For instance, the first dynamic change information is used to indicate the translation speed and rotation angle of the virtual camera, and the first interactive operation refers to a trigger operation performed simultaneously on a first control and a second control, where the first control is used to control the translation speed of the virtual camera, and the second control is used to control the rotation angle of the virtual camera, etc.
[0087] In one possible implementation, the first interaction operation can be a selection operation on camera control text, which describes how the virtual camera moves.
[0088] For example, the camera control text can be "move slowly upwards", "move quickly downwards to the right", "move slowly to the left and rotate forward", etc. The electronic device maps the camera control text corresponding to the first interactive operation to more precise first dynamic change information, such as the first dynamic change information recording the translation speed and rotation angle of the virtual camera.
[0089] The camera control text can be text entered by the user or text selected by the user from a number of preset camera control texts provided by the electronic device.
[0090] In one possible implementation, the first dynamic change information includes translation information, rotation information, and shooting parameters. The translation information indicates the translation method of the virtual camera, the rotation information indicates the rotation method of the virtual camera, and the shooting parameters indicate the shooting method of the virtual camera. Optionally, the translation information can be a translation velocity or a translation vector, the rotation information can be a rotation angle or a rotation speed, and the shooting parameters can be the focal length of the virtual camera.
[0091] For example, the translation speed XYZ ∈ [0,5] m / s, the rotation angle YPR ∈ [0°,360°], and the focal length FOV ∈ [60°,120°].
[0092] In one possible implementation, when the electronic device detects a first interactive operation, it acquires mapping information, which includes mapping relationships between multiple interactive operations and multiple dynamic change information; and in the mapping information, it searches for the first dynamic change information that has a mapping relationship with the first interactive operation.
[0093] For example, the multiple interactive operations are preset interactive operations, including a first interactive operation, and the multiple dynamic change information are preset dynamic change information. Each interactive operation and each dynamic change information has a mapping relationship. The mapping information records each interactive operation and its corresponding mapped dynamic change information. By querying the mapping information, the electronic device can determine the first dynamic change information that has a mapping relationship with the first interactive operation.
[0094] In this implementation, the mapping information records the mapping relationship between various interactive operations and various dynamic changes of the virtual camera. It can map discrete interactive operations into movement modes in continuous dynamic space. Therefore, users can specify the movement mode of the virtual camera by performing simple interactive operations without having to edit the dynamic change information themselves. This realizes an interactive virtual camera control mode, which is beneficial to improving the efficiency and controllability of video generation.
[0095] 403. The electronic device determines multiple first camera parameter matrices of the virtual camera based on the first dynamic change information. Each first camera parameter matrix corresponds to a video frame in the first video segment. The first camera parameter matrix represents the pose of the virtual camera when the video frame is captured.
[0096] Since the movement of the virtual camera during shooting can be controlled based on the first dynamic change information, the first camera parameter matrix of the virtual camera in each of the multiple video frames can be determined based on this first dynamic change information.
[0097] In one possible implementation, the first dynamic change information includes translation speed, rotation angle, and focal length, and the first camera parameter matrix includes an intrinsic parameter matrix and an extrinsic parameter matrix. The intrinsic parameter matrix describes the projection relationship from the camera coordinate system to the pixel coordinate system. The extrinsic parameter matrix describes the transformation from the world coordinate system to the camera coordinate system. Step 403 includes: determining the extrinsic parameter matrix of the virtual camera in each video frame based on the translation speed and rotation angle in the first dynamic change information; and determining the intrinsic parameter matrix of the virtual camera in each video frame based on the focal length in the first dynamic change information.
[0098] The virtual camera's extrinsic parameters include a translation matrix and a rotation matrix. The translation matrix indicates the virtual camera's translation direction and distance, while the rotation matrix indicates the virtual camera's rotation direction and angle. The virtual camera's intrinsic parameters indicate the virtual camera's focal length.
[0099] Assume the camera's translation distances along the X, Y, and Z axes are t, respectively. x , t y , t z Then the translation matrix t is expressed as:
[0100] If the translation speed XYZ∈[0,5]m / s, the time interval between two adjacent frames is 0.1s, and the translation speeds are 2m / s, 1m / s, and 3m / s respectively, then the translation distances are t respectively. x =0.2m,t y =0.1m,t z =0.3m.
[0101] When calculating the rotation matrix R, rotate around the X, Y, and Z axes by a certain angle to obtain three basic rotation matrices, as shown in formulas (2), (3), and (4) below. Then, by multiplying and combining them, the rotation matrix R in any direction can be obtained.
[0102] If rotated θ around the X-axis
[0103] If rotated around the Y-axis
[0104] If rotating around the Z-axis
[0105] Therefore, if combined in the order ZYX, the rotation matrix R is expressed as:
[0106] Furthermore, the intrinsic parameter matrix K of the pinhole camera is represented as follows:
[0107] Among them, f x and f y The focal length (in pixels or world units, depending on the calibration method) represents the focal length in the x and y directions. x and c y This indicates the position of the principal point on the image plane, which can be taken as the center of the image.
[0108] For example, if the rotation angle in the first dynamic change information is a fixed value, then the rotation matrix of the virtual camera is the same in each video frame, meaning the rotation angle of the virtual camera remains unchanged during the shooting of the video clip. Similarly, if the focal length in the first dynamic change information is a fixed value, then the intrinsic parameter matrix of the virtual camera is the same in each video frame, meaning the focal length of the virtual camera remains unchanged during the shooting of the video clip.
[0109] In this implementation, the dynamic change information includes translation speed, rotation angle, and focal length. Based on the translation speed and rotation angle of the virtual camera, the extrinsic parameter matrix of the virtual camera in each video frame can be determined. Based on the focal length of the virtual camera, the intrinsic parameter matrix of the virtual camera in each video frame can be determined. This enables the control of the pose and focal length of the virtual camera in each video frame based on the dynamic change information, improving the flexibility and continuity of controlling the virtual camera.
[0110] Furthermore, the dynamic change information includes translation speed and rotation angle, which enables the coupling of motion control of the virtual camera. For example, the virtual camera can be controlled to translate and rotate simultaneously. The dynamic change information can also include the focal length of the virtual camera, enabling finer-grained control and effectively generalizing to the needs of natural scenes with complex motion trajectories.
[0111] 404. The electronic device converts each first camera parameter matrix into a frame-level orientation feature, which represents the direction in which each pixel in the video frame is projected into world space under the pose indicated by the first camera parameter matrix.
[0112] After determining the first camera parameter matrix of the virtual camera across multiple video frames, the electronic device converts the first camera parameter matrix of the multiple video frames into frame-level orientation features of the multiple video frames. The frame-level orientation features of the video frames represent the direction in which each pixel in the video frame is projected into world space under the pose indicated by the first camera parameter matrix.
[0113] In one possible implementation, for each pixel in a video frame, a first orientation feature of the pixel is determined based on the first camera parameter matrix of the video frame and the planar coordinate vector of the pixel. The first orientation feature of the pixel represents the direction from the virtual camera to the projection point of the pixel in world space, which is the point in world space where the pixel is projected under the pose indicated by the first camera parameter matrix. Based on the first orientation feature of the pixel and the spatial coordinate vector of the virtual camera in world space, a pixel-level orientation feature (also known as pixel coding) of the pixel is determined. The pixel-level orientation features of each pixel in the video frame are merged to obtain the frame-level orientation feature (also known as frame coding) of the video frame.
[0114] The planar coordinate vector of a pixel indicates its position in a two-dimensional plane, that is, its position within a video frame. The spatial coordinate vector of the virtual camera indicates its position in world space, that is, its position within the scene.
[0115] In this context, the pixel-level orientation features of a pixel are 6-dimensional, and the size of a video frame is h×w. The pixel-level orientation features of each pixel in the video frame are then concatenated according to their positions within the frame to obtain the frame-level orientation features of the video frame, which can be denoted as P. i P i ∈R 6×h×w .
[0116] Optionally, the first camera parameter matrix includes an intrinsic parameter matrix and an extrinsic parameter matrix, wherein the extrinsic parameter matrix includes a rotation matrix and a translation matrix. The determination of the first directional feature of a pixel includes: multiplying the rotation matrix, the inverse of the intrinsic parameter matrix, and the transpose of the planar coordinate vector to obtain an intermediate matrix; and adding the intermediate matrix and the translation matrix to obtain the first directional feature of the pixel.
[0117] For example, an electronic device uses the following formula (7) to determine the first directional feature of a pixel: d = R·K -1 ·[u,v,1] T +t; Formula (7)
[0118] Where d represents the first directional feature of the pixel, R represents the rotation matrix, and K represents the intrinsic parameter matrix. -1 Let represent the inverse of the intrinsic parameter matrix, t represent the translation matrix, [u,v,1] represent the planar coordinate vector of the pixel, T represent the transpose, and K represent the inverse of the intrinsic parameter matrix. -1 ·[u,v,1] T This represents converting pixel coordinates to normalized planar coordinates in the camera coordinate system; R·K -1 ·[u,v,1] T The calculated result is the intermediate matrix, representing the rotation of the direction from the camera coordinate system to the world coordinate system using matrix R; then, a translation is performed using matrix t. The final first direction feature indicates the direction along the world ray corresponding to that pixel, originating from the camera's optical center.
[0119] Optionally, determining the pixel-level directional features of a pixel based on the first directional features of the pixel and the spatial coordinate vector of the virtual camera includes: normalizing the first directional features of the pixel to obtain normalized first directional features; multiplying the normalized first directional features by the spatial coordinate vector to obtain the second directional features of the pixel; and merging the second directional features and the normalized first directional features to obtain the pixel-level directional features of the pixel.
[0120] For example, an electronic device uses the following formula (8) to determine the pixel-level orientation features of a pixel: p = (o × d′, d′); Formula (8)
[0121] Where p represents the pixel-level directional feature of a pixel, o represents the spatial coordinate vector of the virtual camera, and d′ represents the normalized first directional feature.
[0122] In this implementation, the orientation features of each pixel are accurately calculated by combining the first camera parameter matrix, the planar coordinate vector of the pixel, and the spatial coordinates of the virtual camera, thereby generating pixel-level orientation features. After merging the pixel-level orientation features of each pixel, frame-level orientation features are generated, which retains the orientation information of each pixel in the video frame under a specific camera pose. This helps to improve the accuracy of generating frame-level orientation features, thereby improving the accuracy of the generated video.
[0123] 405. The electronic device generates a first video segment based on the first scene image and multiple frame-level directional features.
[0124] After determining the frame-level orientation features of the first scene image and multiple video frames, the electronic device generates a first video segment based on the first scene image and multiple frame-level orientation features. The first video segment includes images captured by a virtual camera in the scene displayed in the first scene image according to the pose indicated by multiple first camera parameter matrices.
[0125] In one possible implementation, the electronic device performs step 405 through a video generation model. Step 405 includes steps 4051-4055 below. For a detailed implementation process, please refer to the embodiment in Figure 6 below.
[0126] In this implementation, the camera parameter matrix of the virtual camera is converted into the frame-level directional features of the video frame to be generated. That is, the camera-dimensional data is converted into frame-dimensional data so that video segments can be generated based on the scene image and the frame-level directional features. This allows the camera parameter matrix to be used as a constraint during the generation of video segments, thereby generating video segments with a specified viewpoint and improving the controllability of video generation.
[0127] The method provided in this application, when intelligently generating video clips, allows users to perform interactive operations as needed after providing scene images. These user-performed interactive operations can be mapped to dynamic change information of a virtual camera, controlling the movement of the virtual camera during shooting. Based on this dynamic change information, the camera parameter matrix of the virtual camera in each video frame can be determined, representing the pose of the virtual camera in each frame. Then, based on the scene image and multiple camera parameter matrices of the virtual camera, a video clip captured by the virtual camera moving in that scene according to the specified movement method can be generated. Therefore, by performing interactive operations, users can control the camera movement of the virtual camera during video clip generation, thereby controlling the viewpoint displayed in the generated video clip. This achieves flexible control of the viewpoint of the intelligently generated video clip based on interactive operations, improving the controllability of video generation.
[0128] It should be noted that in the above embodiments, the user guides the generation of the first video segment by providing a first scene image and performing a first interactive operation. In other embodiments, in addition to providing a first scene image and performing the first interactive operation, other prompts can be provided to guide the generation of the first video segment. These other prompts include content description information, which describes the content contained in the first video segment to be generated. For example, the content description information can describe how the content in the scene changes in the first video segment, such as what actions the characters in the scene perform. Therefore, in step 405 above, the electronic device generates the first video segment based on the first scene image, the content description information, and the frame-level directional features of multiple video frames. The content description information can be in text format or audio format, etc.
[0129] Therefore, the embodiments of this application combine multimodal control factors such as scene images, interactive operations, and content description information to achieve dynamic interactive generation of video clips, thereby improving the quality and controllability of video generation.
[0130] In some embodiments, the process of generating the first video segment based on the first scene image and multiple frame-level directional features in step 405 is performed by a video generation model. The model structure of this video generation model is shown in Figure 5. Figure 5 is a schematic diagram of the structure of a video generation model provided in an embodiment of this application. As shown in Figure 5, the video generation model includes a visual feature extraction network 501, an interactive feature extraction network 502, a feature fusion network 503, multiple first denoising networks 504, and a decoding network 505. The visual feature extraction network 501 and the interactive feature extraction network 502 are respectively connected to the feature fusion network 503. The feature fusion network 503 is connected to the first first denoising network 504. The multiple first denoising networks 504 are connected sequentially, and the last first denoising network 504 is connected to the decoding network 505.
[0131] The system includes a visual feature extraction network 501 for extracting visual features from scene images, and an interaction feature extraction network 502 for extracting frame-level directional interaction features. This extraction network can be implemented using a Vision Transformer (ViT). A feature fusion network 503 fuses the extracted visual and interaction features to obtain video features. Multiple first denoising networks 504 denoise the video features after adding noise, and a decoding network 505 decodes the denoised video features to obtain video segments.
[0132] Optionally, the video generation model can be any type of diffusion model, such as a mixed-image video generation model. Optionally, the size of the visual feature extraction network 501 matches the size of the interaction feature extraction network 502, and the downsampling rate of the visual feature extraction network 501 matches the downsampling rate of the interaction feature extraction network 502.
[0133] It should be noted that the video generation model in Figure 5 is only an exemplary demonstration of the model structure of the video generation model. The video generation model in this embodiment can also be a video generation model with other model structures. This embodiment does not limit the model structure of the video generation model.
[0134] The following embodiment, shown in Figure 6, illustrates the detailed implementation process of generating video segments using the video generation model shown in Figure 4. Figure 6 is a flowchart of a video generation method provided in this application embodiment, which is executed by an electronic device. Referring to Figure 6, the method includes:
[0135] 4051. The electronic device extracts features from the first scene image through the visual feature extraction network in the video generation model to obtain the first visual features.
[0136] As shown in Figure 5, the video generation model includes a visual feature extraction network. The electronic device inputs a first scene image into the visual feature extraction network, performs feature extraction on the first scene image, and obtains the first visual feature output by the visual feature extraction network.
[0137] In one possible implementation, the visual feature extraction network includes a pachify layer, which is used to divide the input data into several small blocks for processing.
[0138] In one possible implementation, the electronic device segments the first scene image into multiple image blocks using a visual feature extraction network, extracts features from each of the multiple image blocks using the visual feature extraction network, and obtains the local visual features corresponding to each image block; then, the visual feature extraction network fuses the local visual features corresponding to the multiple image blocks to obtain the first visual feature.
[0139] Optionally, the first scene image is divided into multiple image blocks of equal size using a visual feature extraction network. Then, feature extraction is performed on each of the multiple image blocks to obtain the local visual features corresponding to each image block.
[0140] Optionally, the local visual features corresponding to each of the multiple image blocks are stitched together according to their respective positions in the first scene image to obtain the first visual feature.
[0141] In this implementation, the scene image is segmented into multiple image blocks and local visual features are extracted from each image block. This allows for more detailed capture of local details in the scene, resulting in visually richer and more delicate video clips and improving the accuracy of video generation.
[0142] 4052. Electronic devices extract the features of each frame-level directional feature through the interaction feature extraction network in the video generation model to obtain the first interaction feature of the first scene image.
[0143] As shown in Figure 5, the video generation model includes an interactive feature extraction network. The electronic device inputs multiple frame-level directional features into the interactive feature extraction network, performs feature extraction operations on the multiple frame-level directional features, and outputs the first interactive features of the first scene image.
[0144] In one possible implementation, the first interactive feature of the first scene image includes the first interactive feature of each of the multiple video frames. For each video frame, the electronic device inputs the frame-level directional feature of the video frame into the interactive feature extraction network to obtain the first interactive feature of the video frame output by the interactive feature extraction network.
[0145] In one possible implementation, the interactive feature extraction network includes a pachify layer, which is used to divide the input data into several small blocks for processing.
[0146] In one possible implementation, the interactive feature extraction network has the same network structure as the aforementioned visual feature extraction network, but different network parameters.
[0147] In one possible implementation, an interaction feature extraction network is used to segment each frame-level directional feature into multiple coding blocks. The network then performs feature extraction on each of these coding blocks to obtain local interaction features corresponding to each block, resulting in multiple local interaction features. Finally, the network fuses these local interaction features from the frame-level directional features to obtain the first interaction feature of the first scene image. Specifically, fusing multiple local interaction features yields the first interaction feature of a video frame; fusing the first interaction features of all video frames yields the first interaction feature of the first scene image.
[0148] Optionally, for each video frame's frame-level orientation features, a visual feature extraction network is used to divide the video frame's frame-level orientation features into multiple coding blocks of equal size.
[0149] Optionally, for each video frame's frame-level directional features, the local interactive features corresponding to each of the multiple coding blocks are concatenated according to their respective positions in the frame-level directional features to obtain the first interactive feature of the video frame.
[0150] In this implementation, the frame-level directional features are segmented into multiple coding blocks and local interaction features are extracted from each coding block. This allows for more detailed capture of the local dynamic changes of the virtual camera, resulting in more detailed video clips under the dynamic camera movement and improving the accuracy of video generation.
[0151] 4053. The electronic device integrates the first visual feature, the preset noise feature, and the first interactive feature to obtain the video feature after adding noise.
[0152] The video generation model can use a feature fusion network to fuse the initial visual features and noise features to obtain noisy visual features. Then, the noisy visual features are fused with the initial interaction features to obtain noisy video features. The noise features represent random noise, which can be obtained by randomly sampling Gaussian noise or uniformly distributed noise. This can simulate uncertainties in real videos, such as sensor noise, compression artifacts, and motion blur.
[0153] As shown in Figure 5, the video generation model includes a feature fusion network. The electronic device uses this network to connect a first visual feature and a noise feature along the sequence dimension to obtain a noisy visual feature. Connecting along the sequence dimension means connecting frame by frame. For example, if the first visual feature has 'a' frames and the noise feature has 'b' frames, connecting them along the sequence dimension yields the noisy visual feature, which has 'a+b' frames. The electronic device then uses the feature fusion network to add the noisy visual feature to the first interactive feature to obtain the noisy video feature. Since the noisy visual feature and the first interactive feature have the same dimension, they can be directly added together.
[0154] In one possible implementation, the electronic device acquires a binary mask feature that indicates the position of a first visual feature and the position of a noise feature in the noisy visual features. For example, the position of a '1' in the binary mask feature indicates the position of the first visual feature, and the position of a '0' in the binary mask feature indicates the position of the noise feature, thereby marking the reference visual features and the features to be denoised. The electronic device concatenates the binary mask feature and the noisy visual features along the channel dimension to obtain noisy visual features carrying the marks. Then, it fuses the noisy visual features carrying the marks with the first interactive feature to obtain noisy video features.
[0155] For example, the noisy visual feature has c channels and the binary mask feature has d channels. By connecting the noisy visual feature and the binary mask feature along the channel dimension, we obtain the noisy visual feature with the label, which has c+d channels.
[0156] 4054. Electronic devices perform denoising operations on the noisy video features through multiple first denoising networks in the video generation model to obtain video features.
[0157] As shown in Figure 5, the video generation model includes multiple first denoising networks connected in sequence. The electronic device inputs the denoised video features into the first first denoising network. From the first first denoising network to the last first denoising network, the denoising operation is performed sequentially to complete the multi-step denoising and obtain the denoised video features.
[0158] In this embodiment of the application, by executing the above steps 4053-4054, video features are generated based on the first visual features and the first interactive features.
[0159] 4055. The electronic device decodes the video features through the decoding network in the video generation model to obtain the first video segment.
[0160] As shown in Figure 5, the video generation model includes a decoding network. The electronic device inputs video features into the decoding network, and the decoding network outputs the first video segment.
[0161] In one possible implementation, the video features include multiple video frame features. The electronic device inputs the multiple video frame features to a decoding network, and the decoding network outputs multiple video frames, which constitute the first video segment.
[0162] The method provided in this application performs feature extraction operations on scene images and frame-level directional features respectively to obtain visual features and interactive features. The visual features capture information related to the scene, while the interactive features capture information related to the dynamic changes of the camera. By combining these two, the generated video clips can accurately present the images captured by the virtual camera as it moves through the scene, thus improving the accuracy of video generation.
[0163] The above embodiments illustrate the process of generating a first video segment. In another embodiment, multiple video segments can be generated continuously by executing interactive operations, and the content of the multiple video segments is coherent. For a detailed implementation process, please refer to the embodiment shown in Figure 7 below. Figure 7 is a flowchart of a video generation method provided by an embodiment of this application. This embodiment is executed by an electronic device. Referring to Figure 7, the method includes:
[0164] 701. The electronic device acquires a first scene image, which is used to indicate the scene contained in the first video segment to be generated.
[0165] 702. When the electronic device detects the first interactive operation, it maps the first interactive operation to the first dynamic change information of the virtual camera, and the first dynamic change information is used to indicate the movement mode of the virtual camera in the shooting scene.
[0166] 703. The electronic device determines multiple first camera parameter matrices of the virtual camera based on the first dynamic change information. Each first camera parameter matrix corresponds to a video frame in the first video segment. The first camera parameter matrix represents the pose of the virtual camera when the video frame is captured.
[0167] 704. The electronic device generates a first video segment based on the first scene image and multiple first camera parameter matrices.
[0168] The process of steps 701-704 is the same as that of steps 401-405 above, and will not be repeated here.
[0169] 705. When the electronic device detects the second interactive operation, it maps the second interactive operation to second dynamic change information.
[0170] After generating the first video clip, the user can continue to perform interactive operations, such as a second interactive operation. If the electronic device detects that the user has performed the second interactive operation, it maps this second interactive operation to the second dynamic change information of the virtual camera. This second dynamic change information indicates the movement of the virtual camera during shooting.
[0171] The second interactive operation can be the same as the first interactive operation, or it can be a different kind of interactive operation.
[0172] The process by which the electronic device maps the second interactive operation to the second dynamically changing information is the same as the process of mapping the first interactive operation to the first dynamically changing information in step 402 above, and will not be repeated here.
[0173] 706. The electronic device determines multiple second camera parameter matrices based on the second dynamic change information.
[0174] Each second camera parameter matrix corresponds to a video frame in the second video segment to be generated, and each second camera parameter matrix represents the pose of the virtual camera when the video frame is captured.
[0175] The process by which the electronic device determines multiple second camera parameter matrices based on the second dynamic change information is the same as the process by which it determines multiple first camera parameter matrices based on the first dynamic change information in step 403 above, and will not be repeated here.
[0176] 707. The electronic device acquires a second scene image from a first video segment, wherein the second scene image is any video frame in the first video segment.
[0177] The first video clip is the first intelligently generated video clip, and the scene image used in the generation process of the first video clip can be a first scene image provided by the user. When generating the second video clip, the second video clip is not the first video clip. In order to ensure the content coherence between the generated video clips, the scene image used in the generation process of the second video clip is a second scene image obtained from the first video clip.
[0178] In one possible implementation, the second scene image may be the first video frame or the last video frame of the second video segment, etc., and this application embodiment does not limit this.
[0179] The number of the second scene images can be one or more, and the number of the second scene images is not limited in this embodiment.
[0180] 708. The electronic device generates a second video segment based on the second scene image and multiple second camera parameter matrices.
[0181] The process of generating the second video segment in step 708 is the same as the process of generating the first video segment in the above embodiments, and will not be described again here.
[0182] Figure 8 is a schematic diagram of a video generation method provided in an embodiment of this application. As shown in Figure 8, the user inputs a scene image 1 and performs an interaction operation 1. The interaction operation 1 is mapped to the continuous parameter space of the virtual camera, ultimately obtaining frame-level directional features 1. The scene image 1 and frame-level directional features 1 are input into the video generation model to obtain video segment 1 output by the video generation model. The user continues to perform an interaction operation 2, which is mapped to the continuous parameter space of the virtual camera, ultimately obtaining frame-level directional features 2. The scene image 2 obtained in video segment 1 and frame-level directional features 2 are input into the video generation model to obtain video segment 2 output by the video generation model. The user continues to perform an interaction operation 3, which is mapped to the continuous parameter space of the virtual camera, ultimately obtaining frame-level directional features 3. The scene image 3 obtained in video segment 2 and frame-level directional features 3 are input into the video generation model to obtain video segment 3 output by the video generation model. In this way, by continuously performing interaction operations, the user can generate multiple coherent video segments, achieving logical coherence, smooth motion without abrupt changes, and no quality collapse in the content of the video segments over a long period of time (minutes / hours).
[0183] The method provided in this application embodiment, after providing a first scene image and performing a first interactive operation to generate a corresponding first video segment, continues to perform a second interactive operation, which can generate the next second video segment along with the second scene image in the first video segment and the second interactive operation. Therefore, by continuously performing interactive operations, continuous video segments can be generated in sequence, realizing the scalability and continuity of video generation, which is conducive to generating long videos with coherent content and improves the convenience and efficiency of video generation.
[0184] The training process of the video generation model used in the above embodiments can be seen in the embodiment shown in Figure 9 below. Figure 9 is a flowchart of a video generation model training method provided by an embodiment of this application. This embodiment is executed by an electronic device. Referring to Figure 9, the method includes:
[0185] 901. An electronic device acquires sample scene images, multiple sample camera parameter matrices, and labeled video clips. The sample scene images are used to indicate the sample scene contained in the labeled video clips, and each sample camera parameter matrix represents the pose of the virtual camera when capturing a video frame in the labeled video clip.
[0186] The tagged video clip can be a real video clip containing a sample scene, and the sample scene image can be any video frame obtained from the tagged video clip.
[0187] The multiple sample camera parameter matrices are sample camera parameter matrices for multiple video frames in the tagged video clip. The sample camera parameter matrix for each video frame represents the pose of the virtual camera when that video frame in the tagged video clip was captured.
[0188] The sample scene image is similar to the first scene image mentioned above, and the sample camera parameter matrix is similar to the first camera parameter matrix mentioned above, so it will not be repeated here.
[0189] 902. Electronic devices generate sample video clips based on sample scene images and multiple sample camera parameter matrices using a video generation model.
[0190] In one possible implementation, multiple sample camera parameter matrices are converted into multiple sample frame-level orientation features; sample video clips are then generated using a video generation model based on the sample scene images and the multiple sample frame-level orientation features.
[0191] The process by which the electronic device generates sample video segments through the video generation model is the same as the process of generating the first video segment through the video generation model described above, and will not be repeated here.
[0192] 903. Electronic devices train a video generation model based on sample video clips and labeled video clips.
[0193] Labeled video clips are real video clips, while sample video clips are intelligently generated by a video generation model. The smaller the difference between the sample video clips and the label video clips generated by the video generation model, the more accurate the model is. Therefore, electronic devices adjust the model parameters of the video generation model based on sample and label video clips to obtain a more accurate model. The training objective is to reduce the difference between sample and label video clips, thereby improving the video generation capability of the model.
[0194] The method provided in this application training method uses sample scene images, multiple sample camera parameter matrices, and labeled video clips to train a video generation model. This ensures that the video generation model can learn how to generate video clips based on scene images and camera parameter matrices, enabling the model to better understand the influence of the virtual camera's pose on the images captured in the scene. This enhances the video generation model's ability to generate videos and improves the quality of generated videos.
[0195] It should be noted that the above embodiments illustrate the process of training a video generation model by generating a single video clip. In another embodiment, the video generation method provided in this application can generate multiple consecutive video clips. Therefore, during the training process of the video generation model, it is also possible to train the video generation model by generating multiple consecutive video clips.
[0196] An electronic device acquires tagged videos, including a first tagged video segment and a second tagged video segment. The electronic device acquires a first sample scene image from the first tagged video segment, multiple first sample camera parameter matrices corresponding to the first tagged video segment, and multiple second sample camera parameter matrices corresponding to the second tagged video segment. The electronic device generates a first sample video segment using a video generation model based on the first sample scene image and the multiple first sample camera parameter matrices. The electronic device trains the video generation model based on the first sample video segment and the first tagged video segment. Then, the electronic device acquires a second sample scene image from the first sample video segment, generates a second sample video segment using the video generation model based on the second sample scene image and the multiple second sample camera parameter matrices, and continues to train the video generation model based on the second sample video segment and the second tagged video segment.
[0197] In one possible implementation, the electronic device uses the last n video frames from a sample video clip as the second sample scene image, where n is an integer.
[0198] In this implementation, by jointly training the task of continuating video segments with the task of generating individual video segments, the video generation model supports segment autoregressive video generation. This not only improves the high-quality generation capability of the video generation model for individual video segments, but also provides the ability to generate coherent video segments, which is beneficial for maintaining the coherence and consistency of multiple generated video segments.
[0199] Figure 10 is a flowchart of another training method for a video generation model provided in this application embodiment. This application embodiment is executed by an electronic device. Referring to Figure 10, the method includes:
[0200] 1001. The electronic device acquires sample scene images, multiple sample camera parameter matrices, and labeled video clips. The sample scene images are used to indicate the sample scene contained in the labeled video clips, and each sample camera parameter matrix represents the pose of the virtual camera when capturing video frames in the labeled video clips.
[0201] 1002. Electronic devices convert multiple sample camera parameter matrices into multiple sample frame-level orientation features.
[0202] The process of step 1002 is the same as that of step 404 above, and will not be repeated here.
[0203] 1003. Electronic devices perform feature extraction operations on sample scene images through a video generation model to obtain the first sample visual features. Through the video generation model, they extract the features of each sample frame-level directional features to obtain the sample interaction features of the sample scene images.
[0204] The process of step 1003 is the same as that of steps 4051-4052 above, and will not be repeated here.
[0205] 1004. Electronic devices generate sample video features based on the visual features and interaction features of the first sample through a video generation model.
[0206] In one possible implementation, the electronic device performs feature extraction on multiple video frames in a sample video segment using a video generation model to obtain second sample visual features; then, using the video generation model, it fuses the second sample visual features, sample noise features, and first sample visual features to obtain noisy sample visual features, where the sample noise features represent random noise; finally, using the video generation model, it fuses the noisy sample visual features and sample interaction features to obtain noisy sample video features; and finally, using the video generation model, it performs denoising on the noisy sample video features to obtain sample video features.
[0207] Optionally, sample noise features are added to the second sample visual features to obtain noisy second sample visual features. The noisy second sample visual features and the first sample visual features are then concatenated along the sequence dimension to obtain the noisy sample visual features, where concatenation along the sequence dimension means concatenating frame by frame. For example, if the first sample visual features have 'a' frames and the noisy second sample visual features have 'b' frames, concatenating the first sample visual features and the noisy second sample visual features along the sequence dimension yields the noisy sample visual features, which have 'a+b' frames.
[0208] Optionally, the visual features after adding noise to the sample and the interaction features of the sample are added together to obtain the video features after adding noise to the sample. The visual features after adding noise to the sample and the interaction features of the sample have the same dimension, so the visual features after adding noise to the sample and the interaction features of the sample can be added directly.
[0209] Optionally, the electronic device acquires sample binary mask features, which are used to indicate the positions of the noisy second sample visual features and the first sample visual features in the noisy visual features. For example, the position of a '1' in the sample binary mask features indicates the position of the first sample visual feature, and the position of a '0' in the sample binary mask features indicates the position of the noisy second sample visual feature, thereby marking the reference first sample visual features and the noisy second sample visual features. The electronic device concatenates the sample binary mask features and the noisy sample visual features along the channel dimension to obtain the marked noisy sample visual features, and then fuses the marked noisy sample visual features with the sample interaction features to obtain the noisy sample video features.
[0210] For example, the visual features of a sample after adding noise have c channels, and the binary mask features of the sample have d channels. By connecting the visual features of the sample after adding noise and the binary mask features of the sample along the channel dimension, we obtain the visual features of the sample after adding noise with the label. The visual features of the sample after adding noise with the label have c+d channels.
[0211] 1005. Electronic devices decode the features of sample videos using a video generation model to obtain sample video segments.
[0212] The process of step 1005 is the same as that of step 4055 above, and will not be repeated here.
[0213] In this implementation, feature extraction operations are performed on the sample scene image and sample frame-level directional features respectively to obtain sample visual features and sample interaction features. The sample visual features capture information related to the scene, while the sample interaction features capture information related to the dynamic changes of the camera. By combining these two, the generated sample video clips can accurately represent the images captured by the virtual camera as it moves through the scene, improving the accuracy of the video generation model.
[0214] 1006. Electronic devices train a video generation model based on sample video clips and labeled video clips.
[0215] In one possible implementation, the electronic device determines a first loss parameter based on sample video clips and labeled video clips, the first loss parameter representing the degree of difference between the sample video clips and the labeled video clips. The electronic device trains the video generation model with the goal of reducing this first loss parameter.
[0216] 1007. Electronic devices determine the discrimination results of sample video features through a discrimination model, and train a video generation model based on the discrimination results.
[0217] The discrimination result represents the probability that the sample video features belong to real video features. Real video features refer to the features obtained by performing feature extraction operations on real video segments.
[0218] The higher the probability that a sample video feature belongs to a real video feature, the more accurate the video generation model; conversely, the lower the probability, the less accurate the model. Therefore, electronic devices train their video generation models based on the sample video feature discrimination results, making the generated sample video features increasingly closer to real video features, thereby improving the model's performance.
[0219] In one possible implementation, the electronic device acquires the label video features of the labeled video segments, determines the discrimination result of the label video features through a discrimination model, and trains the discrimination model based on the discrimination result of the sample video features and the discrimination result of the label video features.
[0220] The training objective of the discriminative model is to reduce the probability represented by the discrimination result of the sample video features and increase the probability represented by the discrimination result of the labeled video features, thereby enhancing the discriminative model's ability to distinguish between real video features and intelligently generated video features.
[0221] In one possible implementation, the electronic device trains a video generation model based on the discrimination results of sample video features and the difference information between the discrimination results of sample video features and the discrimination results of labeled video features.
[0222] Optionally, the electronic device determines a second loss parameter based on the discrimination result of the sample video features. This second loss parameter is negatively correlated with the probability represented by the discrimination result of the sample video features. Based on the difference between the discrimination results of the sample video features and the label video features, a third loss parameter is determined. This third loss parameter represents the degree of difference between the discrimination results of the sample video features and the label video features. The electronic device trains a video generation model with the goal of reducing the second and third loss parameters.
[0223] 1008. When the video generation model is used as the student model, the electronic device trains the video generation model based on the trained teacher model corresponding to the video generation model.
[0224] In this embodiment, the video generation model is a student model, which corresponds to a trained teacher model. Both the student and teacher models function identically, generating video clips based on scene images and multiple camera parameter matrices. The difference lies in the size of the student model, which is smaller than that of the teacher model. During the training phase of the student model, the lightweight student model is guided and trained using the teacher model, which has a complex structure. This allows the student model to inherit the knowledge learned by the teacher model; that is, the student model is trained through model distillation, thus retaining performance close to that of the teacher model. Furthermore, because the student model is simpler in size than the teacher model, it has lower computational resource requirements and faster inference speed. Therefore, when using the student model to perform video generation tasks, the efficiency of video generation is improved while maintaining accuracy, which is beneficial for achieving real-time generation of multiple coherent video clips.
[0225] The student model includes M first denoising networks, which means M denoising operations need to be performed. The teacher model includes N second denoising networks, which means N denoising operations need to be performed. M and N are both integers greater than 1, and M is less than N.
[0226] During training, the M first denoising networks of the student model are divided into K first denoising trajectories, and the N second denoising networks of the teacher model are divided into K second denoising trajectories. Each denoising trajectory includes multiple denoising networks connected in series. The i-th first denoising trajectory corresponds to the i-th second denoising trajectory, and the number of first denoising networks in the i-th first denoising trajectory is less than the number of second denoising networks in the i-th second denoising trajectory. K is a positive integer less than M, and i is a positive integer not greater than K. In the training and inference process of the denoising model, the "trajectory" originates from the noise level change path of the diffusion process. Data is gradually denoised from real samples to pure noise, and then gradually denoised back to samples. This process can be viewed as moving forward or backward along a path in the "noise space." A "trajectory" refers to a set of denoising networks executed sequentially, forming a continuous path along the noise level sequence.
[0227] The electronic device inputs sample scene images and multiple sample frame-level directional features into the teacher model for processing. After processing, the electronic device obtains the input and output features of the i-th second denoising trajectory of the teacher model. The input features refer to the input features of the first second denoising network in the i-th second denoising trajectory, which can be the image after a certain number of noise addition steps. All second denoising networks of the i-th second denoising trajectory are executed sequentially, and the final output feature (i.e., the denoising result of the last step of this trajectory) is the output feature of the i-th second denoising trajectory.
[0228] The electronic device inputs the input features of the i-th second denoised trajectory of the teacher model to the i-th first denoised trajectory of the student model. Through the i-th first denoised trajectory, denoising is performed on the input features of the i-th second denoised trajectory, resulting in the output features of the i-th first denoised trajectory. Based on the output features of the i-th first denoised trajectory and the i-th second denoised trajectory, the electronic device trains the first denoising network in the i-th first denoised trajectory. All first denoising networks in the i-th first denoised trajectory are executed sequentially, and the final output feature (i.e., the denoising result of the last step of this trajectory) is the output feature of the i-th first denoised trajectory.
[0229] That is, each first denoising trajectory of the student model corresponds to a second denoising trajectory in the teacher model. For any first denoising trajectory, the electronic device trains the first denoising network in the first denoising trajectory based on the output features of the first denoising trajectory and the output features of the corresponding second denoising trajectory, so that the denoising ability of the first denoising trajectory is aligned with the denoising ability of the second denoising trajectory. Since the number of second denoising networks in the second denoising trajectory is greater than the number of first denoising networks in the first denoising trajectory, it is equivalent to inheriting the more complex and superior denoising ability of the second denoising trajectory on the lightweight first denoising trajectory, thereby realizing the distillation of complex network structures.
[0230] Furthermore, since this application aligns each denoised trajectory of the student model with the corresponding denoised trajectory in the teacher model on a denoised trajectory basis, it provides a phased consistent distillation method. By learning on each denoised trajectory, the problem of error accumulation can be avoided, which is beneficial to improving the accuracy of model distillation and improving the performance of the student model.
[0231] In one possible implementation, the electronic device determines a fourth loss parameter based on the output features of the i-th first denoised trajectory and the output features of the i-th second denoised trajectory. The fourth loss parameter represents the degree of difference between the output features of the i-th first denoised trajectory and the output features of the i-th second denoised trajectory. The first denoising network in the i-th first denoised trajectory is trained with the goal of reducing the fourth loss parameter.
[0232] In one possible implementation, the number of denoising networks in the i-th second denoising trajectory is P times the number of denoising networks in the i-th first denoising trajectory, and each first denoising network in the i-th first denoising trajectory corresponds to P second denoising networks in the i-th second denoising trajectory, where P is a positive integer.
[0233] The electronic device trains the first denoising network in the i-th first denoising trajectory based on the output features of each first denoising network in the i-th first denoising trajectory and the label output features corresponding to each first denoising network; where the label output features refer to the output features of the last second denoising network among the P second denoising networks corresponding to the first denoising network.
[0234] Optionally, each first denoising network in the student model corresponds to P consecutive second denoising networks in the teacher model. The electronic device randomly divides the M first denoising networks of the student model into K first denoising trajectories, each first denoising trajectory including multiple consecutive first denoising networks. After determining the K first denoising trajectories, K second denoising trajectories are adaptively divided.
[0235] For example, the student model includes 8 first denoising networks, and the teacher model includes 64 second denoising networks. The first first denoising network corresponds to the 1st to 8th second denoising networks, the second first denoising network corresponds to the 9th to 16th second denoising networks, and so on. The 8th first denoising network corresponds to the 57th to 64th second denoising networks.
[0236] In one iteration of training, the student model is randomly divided into three first denoising trajectories. For example, the first first denoising trajectory includes the first denoising networks 1-3, and the corresponding second denoising trajectory includes the second denoising networks 1-24; the second first denoising trajectory includes the first denoising networks 4-6, and the corresponding second denoising trajectory includes the second denoising networks 25-48; the third first denoising trajectory includes the first denoising networks 7-8, and the corresponding second denoising trajectory includes the second denoising networks 49-64.
[0237] Taking the first denoising trajectory as an example, the electronic device inputs the input features of the first second denoising trajectory into the first denoising trajectory to obtain the output features of each first denoising network in the first denoising trajectory. The first denoising trajectory includes the first to third first denoising networks. Then, the electronic device can train the first to third first denoising networks based only on the output features of the third first denoising network and the output features of the 24th second denoising network, or it can train the first to third first denoising networks based on the output features of the first first denoising network and the output features of the eighth second denoising network, the second first denoising network and the sixteenth second denoising network, the third first denoising network and the 24th second denoising network.
[0238] The method provided in this application training method uses sample scene images, multiple sample camera parameter matrices, and labeled video clips to train a video generation model. This ensures that the video generation model can learn how to generate video clips based on scene images and camera parameter matrices, enabling the model to better understand the influence of the virtual camera's pose on the images captured in the scene. This enhances the video generation model's ability to generate videos and improves the quality of generated videos.
[0239] Furthermore, a discriminative model is introduced to discriminate the features of sample videos, obtaining a discriminative result. The discriminative result represents the probability that the features of the sample video belong to the features of the real video. Then, the video generation model is trained based on the discriminative result to perform adversarial training on the video generation model, prompting the video generation model to optimize the generated video features, making them gradually approach the features of the real video. This helps to enhance the processing power of the video generation model and improve its accuracy.
[0240] Furthermore, during the training phase of the video generation model, a teacher model with a complex structure is used to guide the training of a lightweight student model. The lightweight student model is then used to perform the video generation task. The student model inherits the knowledge learned by the teacher model, retaining performance close to that of the teacher model during inference. Moreover, the student model's structure is simpler than that of the teacher model, resulting in lower computational resource requirements and faster inference speed. Therefore, while ensuring the accuracy of video generation, the efficiency of video generation is improved, which is beneficial for achieving real-time video generation.
[0241] Furthermore, during the model distillation process, the entire denoising process is divided into multiple denoising trajectories. Taking the denoising trajectory as a unit, each denoising trajectory of the student model is aligned with the corresponding denoising trajectory in the teacher model, providing a method for phased consistent distillation. Since learning is performed on each denoising trajectory, the problem of error accumulation can be avoided, which is conducive to improving the accuracy of model distillation and improving the performance of the student model.
[0242] The video generation method provided in this application can be applied to any scenario where video generation is required.
[0243] For example, it can be applied to the field of game development. In game development, developers can provide a game scene image, and through interactive operations, generate a series of coherent game scene video clips based on that image, forming a game scene video. Applying this game scene video to video games can improve the generation effect of game scenes.
[0244] Figure 11 is a schematic diagram of a game scene video clip provided in an embodiment of this application. As shown in Figure 11, for a first game scene, the developer provides a scene image of the first game scene. By performing interactive operations, game scene video clips 11, 12, and 13 containing the first game scene can be generated sequentially. For a second game scene, the developer provides a scene image of the second game scene. By performing interactive operations, game scene video clips 21, 22, and 23 containing the second game scene can be generated sequentially.
[0245] Figure 12 is a schematic diagram of a video generation apparatus provided in an embodiment of this application. Referring to Figure 12, the apparatus includes:
[0246] The acquisition module 1201 is used to acquire a first scene image, which is used to indicate the scene contained in the first video segment to be generated;
[0247] The first conversion module 1202 is used to map the first interactive operation into the first dynamic change information of the virtual camera when the first interactive operation is detected. The first dynamic change information is used to indicate the movement mode of the virtual camera in the shooting scene.
[0248] The second conversion module 1203 is used to determine multiple first camera parameter matrices of the virtual camera based on the first dynamic change information. Each first camera parameter matrix corresponds to a video frame in the first video segment, and the first camera parameter matrix represents the pose of the virtual camera when the video frame is captured.
[0249] The generation module 1204 is used to generate a first video segment based on a first scene image and multiple first camera parameter matrices.
[0250] The video generation apparatus provided in this application allows users to perform interactive operations as needed after providing scene images when intelligently generating video clips. These user-performed interactive operations can be mapped to dynamic change information of a virtual camera to control the movement of the virtual camera during shooting. Based on this dynamic change information, the camera parameter matrix of the virtual camera in each video frame can be determined, representing the pose of the virtual camera in each frame. Therefore, based on the scene image and multiple camera parameter matrices of the virtual camera, a video clip captured by the virtual camera moving in the scene according to this movement method can be generated. Thus, by performing interactive operations, users can control the camera movement of the virtual camera during video clip generation, thereby controlling the viewpoint displayed in the generated video clip. This achieves flexible control of the viewpoint of the intelligently generated video clip based on interactive operations, improving the controllability of video generation.
[0251] Optionally, referring to Figure 13, the first conversion module 1202 is used for:
[0252] When the first interactive operation is detected, the mapping information is obtained. The mapping information includes the mapping relationship between various interactive operations and various dynamically changing information.
[0253] In the mapping information, find the first dynamic change information that has a mapping relationship with the first interactive operation.
[0254] Optionally, referring to Figure 13, the first dynamic change information includes translation speed, rotation angle, and focal length; the first camera parameter matrix includes an intrinsic parameter matrix and an extrinsic parameter matrix; the second conversion module 1203 is also used for:
[0255] Based on the translation speed and rotation angle in the first dynamic change information, the extrinsic parameter matrix of the virtual camera is determined for each video frame;
[0256] Based on the focal length in the first dynamic change information, the intrinsic parameter matrix of the virtual camera is determined for each video frame.
[0257] Optionally, referring to Figure 13, the generation module 1204 is used for:
[0258] Each first camera parameter matrix is converted into frame-level orientation features.
[0259] This indicates the direction in which each pixel in the video frame is projected into world space under the pose indicated by the first camera parameter matrix;
[0260] A first video segment is generated based on the first scene image and multiple frame-level directional features.
[0261] Optionally, referring to Figure 13, the generation module 1204 is used for:
[0262] For each pixel in a video frame, a first directional feature of the pixel is determined based on the first camera parameter matrix and the planar coordinate vector of the pixel. The first directional feature represents the direction from the virtual camera to the projection point of the pixel in the world space. Based on the first directional feature and the spatial coordinate vector of the virtual camera in the world space, a pixel-level directional feature of the pixel is determined.
[0263] By merging the pixel-level directional features of each pixel in a video frame, the frame-level directional features of the video frame are obtained.
[0264] Optionally, referring to Figure 13, the first camera parameter matrix includes an intrinsic parameter matrix and an extrinsic parameter matrix, the extrinsic parameter matrix including a rotation matrix and a translation matrix; the generation module 1204 is used for:
[0265] Multiply the rotation matrix, the inverse of the intrinsic parameter matrix, and the transpose of the planar coordinate vector to obtain the intermediate matrix;
[0266] The first directional feature of a pixel is obtained by adding the intermediate matrix and the translation matrix.
[0267] Optionally, referring to Figure 13, the generation module 1204 is used for:
[0268] The first directional feature of the pixel is normalized to obtain the normalized first directional feature;
[0269] Multiply the normalized first directional feature by the spatial coordinate vector to obtain the second directional feature of the pixel;
[0270] By merging the second directional feature and the normalized first directional feature, pixel-level directional features of the pixel are obtained.
[0271] Optionally, referring to Figure 13, the generation module 1204 is used for:
[0272] Extract the features from the first scene image to obtain the first visual features;
[0273] Extract the features of each frame-level directional feature to obtain the first interaction feature of the first scene image;
[0274] Based on the first visual features and the first interaction features, video features are generated;
[0275] The video features are decoded to obtain the first video segment.
[0276] Optionally, referring to Figure 13, the device executes via a video generation model, which includes a visual feature extraction network; the generation module 1204 is used for:
[0277] Through the aforementioned visual feature extraction network
[0278] The first scene image is divided into multiple image blocks;
[0279] Extract the features of each image patch from the plurality of image patches to obtain local visual features;
[0280] The first visual feature is obtained by fusing multiple local visual features.
[0281] Optionally, referring to Figure 13, the device executes through a video generation model, which includes an interactive feature extraction network; the generation module 1204 is used for:
[0282] Through the interaction feature extraction network
[0283] For each frame-level directional feature, the frame-level directional feature is segmented into multiple coding blocks; features of each coding block are extracted to obtain local interaction features;
[0284] The first interaction feature is obtained by fusing multiple local interaction features of multiple frame-level directional features.
[0285] Optionally, referring to Figure 13, the generation module 1204 is used for:
[0286] By fusing the first visual feature, the preset noise feature, and the first interaction feature, the video features with added noise are obtained;
[0287] A denoising operation is performed on the noisy video features to obtain the video features.
[0288] Optionally, referring to Figure 13, the first conversion module 1202 is also used to map the second interactive operation into second dynamic change information when the second interactive operation is detected;
[0289] The second conversion module 1203 is also used to determine multiple second camera parameter matrices based on the second dynamic change information;
[0290] The acquisition module 1201 is also used to acquire a second scene image from the first video segment, wherein the second scene image is any video frame in the first video segment;
[0291] The generation module 1204 is also used to generate a second video segment based on the second scene image and multiple second camera parameter matrices.
[0292] Optionally, referring to Figure 13, the device performs the operation via a video generation model. The device also includes a model training module 1205 for:
[0293] Acquire sample scene images, multiple sample camera parameter matrices, and labeled video clips. The sample scene images are used to indicate the sample scenes contained in the labeled video clips, and each sample camera parameter matrix represents the pose of the virtual camera when capturing video frames in the labeled video clip.
[0294] Using a video generation model, sample video clips are generated based on sample scene images and parameter matrices of multiple sample cameras;
[0295] A video generation model is trained based on sample video clips and labeled video clips.
[0296] Optionally, referring to Figure 13, the model training module 1205 is used for:
[0297] Convert multiple sample camera parameter matrices into multiple sample frame-level orientation features;
[0298] The video generation model generates sample video clips based on sample scene images and multiple sample frame-level directional features.
[0299] Optionally, referring to Figure 13, the model training module 1205 is used for:
[0300] Through the video generation model
[0301] Extract the features from the sample scene image to obtain the first sample visual features;
[0302] Extract the features of each sample frame-level directional feature to obtain the sample interaction features of the sample scene image;
[0303] Based on the visual features of the first sample and the interaction features of the sample, generate sample video features;
[0304] The sample video features are decoded to obtain the sample video segment.
[0305] Optionally, referring to Figure 13, the device further includes:
[0306] The discrimination module 1206 is used to determine the discrimination result of the sample video features through the discrimination model. The discrimination result represents the probability that the sample video features belong to real video features. Real video features refer to the features obtained by performing feature extraction operations on real video segments.
[0307] The model training module 1205 is also used to train the video generation model based on the discrimination results.
[0308] Optionally, referring to Figure 13, the video generation model is a student model, which corresponds to a trained teacher model. The teacher model is used to generate video clips based on scene images and multiple camera parameter matrices. The student model includes M first denoising networks, and the teacher model includes N second denoising networks, where M and N are both integers greater than 1, and M is less than N. The model training module 1205 is also used for:
[0309] The M first denoising networks are divided into K first denoising trajectories, and the N second denoising networks are divided into K second denoising trajectories. Each denoising trajectory includes multiple denoising networks connected in series. The i-th first denoising trajectory corresponds to the i-th second denoising trajectory, and the number of first denoising networks in the i-th first denoising trajectory is less than the number of second denoising networks in the i-th second denoising trajectory. K is a positive integer less than M, and i is a positive integer not greater than K.
[0310] Obtain the input and output features of the i-th second denoised trajectory;
[0311] By using the i-th first denoised trajectory, a denoising operation is performed on the input features of the i-th second denoised trajectory to obtain the output features of the i-th first denoised trajectory;
[0312] Based on the output features of the i-th first denoising trajectory and the output features of the i-th second denoising trajectory, train the first denoising network in the i-th first denoising trajectory.
[0313] Optionally, referring to Figure 13, the number of denoising networks in the i-th second denoising trajectory is P times the number of denoising networks in the i-th first denoising trajectory, and each first denoising network in the i-th first denoising trajectory corresponds to P second denoising networks in the i-th second denoising trajectory, where P is a positive integer; the model training module 1205 is used for:
[0314] Based on the output features of each first denoising network in the i-th first denoising trajectory and the label output features corresponding to each first denoising network, train the first denoising network in the i-th first denoising trajectory; where the label output features refer to the output features of the last second denoising network among the P second denoising networks corresponding to the first denoising network.
[0315] It should be noted that the video generation apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above. In addition, the video generation apparatus and the video generation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
[0316] This application also provides an electronic device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the video generation method of the above embodiments.
[0317] Optionally, the electronic device is provided as a terminal. Figure 14 shows a schematic diagram of the structure of a terminal 1400 provided in an exemplary embodiment of this application.
[0318] Terminal 1400 includes a processor 1401 and a memory 1402.
[0319] Processor 1401 may include at least one processing core, such as a quad-core processor, an octa-core processor, etc. Processor 1401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.
[0320] The memory 1402 may include at least one computer-readable storage medium, which may be non-transitory. The memory 1402 may also include high-speed random access memory and non-volatile memory, such as at least one disk storage device or flash memory device. In some embodiments, the non-transitory computer-readable storage medium in the memory 1402 is used to store at least one computer program, which is used by the processor 1401 to implement the video generation method provided in the method embodiments of this application.
[0321] In some embodiments, the terminal 1400 may also optionally include: a peripheral device interface 1403 and at least one peripheral device. The processor 1401, memory 1402, and peripheral device interface 1403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of: a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, an audio circuit 1407, and a power supply 1408.
[0322] Peripheral device interface 1403 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1401 and memory 1402. In some embodiments, processor 1401, memory 1402 and peripheral device interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1401, memory 1402 and peripheral device interface 1403 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
[0323] The radio frequency (RF) circuit 1404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1404 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1404 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1404 includes: an antenna system, an RF transceiver, at least one amplifier, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1404 can communicate with other devices via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1404 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.
[0324] Display screen 1405 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1405 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1401 for processing. In this case, display screen 1405 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1405, disposed on the front panel of terminal 1400; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1400 or in a folded design; in still other embodiments, display screen 1405 may be a flexible display screen, disposed on a curved or folded surface of terminal 1400. Furthermore, display screen 1405 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1405 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0325] The camera assembly 1406 is used to acquire images or videos. Optionally, the camera assembly 1406 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal 1400, and the rear-facing camera is disposed on the back of the terminal 1400. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blur function by fusion of the main camera and the depth-sensing camera, panoramic shooting function by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting function, or other fusion shooting function. In some embodiments, the camera assembly 1406 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation under different color temperatures.
[0326] The audio circuit 1407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1401 for processing, or input to the radio frequency circuit 1404 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1400. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1401 or the radio frequency circuit 1404 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1407 may also include a headphone jack.
[0327] Power supply 1408 is used to power the various components in terminal 1400. Power supply 1408 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 1408 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.
[0328] Those skilled in the art will understand that the structure shown in FIG14 does not constitute a limitation on the terminal 1400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.
[0329] Optionally, the electronic device is provided as a server. Figure 15 is a schematic diagram of the structure of a server provided in an embodiment of this application. The server 1500 can vary considerably due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 1501 and one or more memories 1502. The memory 1502 stores at least one computer program, which is loaded and executed by the processor 1501 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.
[0330] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the video generation method of the above embodiments.
[0331] This application also provides a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the video generation method of the above embodiments.
[0332] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0333] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.
Claims
1. A video generation method, executed by an electronic device, the method comprising: Acquire a first scene image, which is used to indicate the scene contained in the first video segment to be generated; When the first interaction operation is detected, the first interaction operation is mapped to the first dynamic change information of the virtual camera, and the first dynamic change information is used to indicate the movement mode of the virtual camera when shooting the scene. Based on the first dynamic change information, a plurality of first camera parameter matrices of the virtual camera are determined, each first camera parameter matrix corresponding to a video frame in the first video segment, and the first camera parameter matrix representing the pose of the virtual camera when the video frame is captured. and, The first video segment is generated based on the first scene image and the plurality of first camera parameter matrices.
2. The method according to claim 1, wherein, The step of mapping the first interaction operation to the first dynamic change information of the virtual camera when the first interaction operation is detected includes: When the first interactive operation is detected, mapping information is obtained, and the mapping information includes the mapping relationship between multiple interactive operations and multiple dynamically changing information. In the mapping information, find the first dynamic change information that has a mapping relationship with the first interactive operation.
3. The method according to claim 1 or 2, wherein, The first dynamic change information includes translation speed, rotation angle, and focal length; the first camera parameter matrix includes an intrinsic parameter matrix and an extrinsic parameter matrix; determining multiple first camera parameter matrices of the virtual camera based on the first dynamic change information includes: Based on the translation speed and the rotation angle, the extrinsic parameter matrix of the virtual camera is determined for each video frame; Based on the focal length, the intrinsic parameter matrix of the virtual camera is determined for each video frame.
4. The method according to any one of claims 1-3, wherein, The step of generating the first video segment based on the first scene image and the plurality of first camera parameter matrices includes: Each of the first camera parameter matrices is converted into a frame-level orientation feature, which represents the direction in which each pixel in the video frame is projected into world space under the pose indicated by the first camera parameter matrix. The first video segment is generated based on the first scene image and multiple frame-level directional features.
5. The method according to claim 4, wherein, The step of converting each of the first camera parameter matrices into frame-level directional features includes: For each pixel in the video frame Based on the first camera parameter matrix and the planar coordinate vector of the pixel, a first directional feature of the pixel is determined, wherein the first directional feature represents the direction from the virtual camera to the projection point of the pixel in the world space; Based on the first directional feature and the spatial coordinate vector of the virtual camera in the world space, the pixel-level directional feature of the pixel is determined; The pixel-level directional features of each pixel in the video frame are merged to obtain the frame-level directional features of the video frame.
6. The method according to claim 5, wherein, The first camera parameter matrix includes an intrinsic parameter matrix and an extrinsic parameter matrix, wherein the extrinsic parameter matrix includes a rotation matrix and a translation matrix; Determining the first directional feature of the pixel based on the first camera parameter matrix and the planar coordinate vector of the pixel includes: Multiply the rotation matrix, the inverse of the intrinsic parameter matrix, and the transpose of the planar coordinate vector to obtain the intermediate matrix; The first directional feature of the pixel is obtained by adding the intermediate matrix and the translation matrix.
7. The method according to claim 5 or 6, wherein, Determining the pixel-level orientation feature of the pixel based on the first orientation feature and the spatial coordinate vector of the virtual camera in the world space includes: The first directional feature is normalized to obtain the normalized first directional feature; Multiply the normalized first directional feature by the spatial coordinate vector to obtain the second directional feature; The second directional feature and the normalized first directional feature are combined to obtain the pixel-level directional feature.
8. The method according to any one of claims 4-7, wherein, The step of generating the first video segment based on the first scene image and multiple frame-level directional features includes: Extract the features from the first scene image to obtain the first visual features; Extract the features of each frame-level directional feature to obtain the first interaction feature of the first scene image; Based on the first visual features and the first interaction features, video features are generated; The video features are decoded to obtain the first video segment.
9. The method according to claim 8, wherein, The method is executed through a video generation model, which includes a visual feature extraction network; the step of extracting features from the first scene image to obtain first visual features includes: Through the aforementioned visual feature extraction network The first scene image is divided into multiple image blocks; Extract the features of each image patch from the plurality of image patches to obtain local visual features; The first visual feature is obtained by fusing multiple local visual features.
10. The method according to claim 8, wherein, The method is executed through a video generation model, which includes an interaction feature extraction network; the step of extracting features from each frame-level directional feature to obtain the first interaction feature of the first scene image includes: Through the interaction feature extraction network For each frame-level directional feature, the frame-level directional feature is segmented into multiple coding blocks; features of each coding block are extracted to obtain local interaction features; The first interaction feature is obtained by fusing multiple local interaction features of multiple frame-level directional features.
11. The method according to any one of claims 8-10, wherein, The step of generating video features based on the first visual features and the first interaction features includes: By fusing the first visual feature, the preset noise feature, and the first interaction feature, the video features with added noise are obtained. A denoising operation is performed on the noisy video features to obtain the video features.
12. The method according to any one of claims 1 to 11, further comprising: When a second interactive operation is detected, the second interactive operation is mapped to second dynamically changing information; Based on the second dynamic change information, multiple second camera parameter matrices are determined; A second scene image is obtained from the first video segment, wherein the second scene image is any video frame in the first video segment; A second video segment is generated based on the second scene image and the plurality of second camera parameter matrices.
13. The method according to any one of claims 1 to 12, wherein, The method is executed through a video generation model, the training process of which includes: Acquire sample scene images, multiple sample camera parameter matrices, and labeled video clips. The sample scene images are used to indicate the sample scenes contained in the labeled video clips, and each sample camera parameter matrix represents the pose of the virtual camera when a video frame in the labeled video clip is captured. Based on the sample scene images and the multiple sample camera parameter matrices, the video generation model generates sample video clips. The video generation model is trained based on the sample video clips and the labeled video clips.
14. The method according to claim 13, wherein, The step of generating sample video clips using the video generation model, based on the sample scene images and the multiple sample camera parameter matrices, includes: The multiple sample camera parameter matrices are converted into multiple sample frame-level orientation features; The video generation model generates the sample video segments based on the sample scene images and the multiple sample frame-level directional features.
15. The method according to claim 14, wherein, The step of generating the sample video segment using the video generation model, based on the sample scene image and the multiple sample frame-level directional features, includes: Through the video generation model Extract the features from the sample scene image to obtain the first sample visual features; Extract the features of each sample frame-level directional feature to obtain the sample interaction features of the sample scene image; Based on the visual features of the first sample and the interaction features of the sample, generate sample video features; The sample video features are decoded to obtain the sample video segment.
16. The method according to any one of claims 13-15, wherein, The video generation model is a student model, which corresponds to a trained teacher model. The student model includes M first denoising networks, and the teacher model includes N second denoising networks. M and N are both integers greater than 1, and M is less than N. The method further includes: The M first denoising networks are divided into K first denoising trajectories, and the N second denoising networks are divided into K second denoising trajectories. Each denoising trajectory includes multiple denoising networks connected in series. The i-th first denoising trajectory corresponds to the i-th second denoising trajectory, and the number of first denoising networks in the i-th first denoising trajectory is less than the number of second denoising networks in the i-th second denoising trajectory. K is a positive integer less than M, and i is a positive integer not greater than K. Obtain the input and output features of the i-th second denoised trajectory; By using the i-th first denoised trajectory, a denoising operation is performed on the input features of the i-th second denoised trajectory to obtain the output features of the i-th first denoised trajectory; Based on the output features of the i-th first denoising trajectory and the output features of the i-th second denoising trajectory, train the first denoising network in the i-th first denoising trajectory.
17. A video generation apparatus, comprising: The acquisition module is used to acquire a first scene image, which is used to indicate the scene contained in the first video segment to be generated; The first conversion module is used to map the first interactive operation into the first dynamic change information of the virtual camera when the first interactive operation is detected. The first dynamic change information is used to indicate the movement mode of the virtual camera when shooting the scene. The second conversion module is used to determine multiple first camera parameter matrices of the virtual camera based on the first dynamic change information. Each first camera parameter matrix corresponds to a video frame in the first video segment, and the first camera parameter matrix represents the pose of the virtual camera when the video frame is captured. The generation module is used to generate the first video segment based on the first scene image and the plurality of first camera parameter matrices.
18. An electronic device comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the video generation method as described in any one of claims 1 to 16.
19. A computer-readable storage medium storing at least one computer program, said at least one computer program being loaded and executed by a processor to perform the operations performed by the video generation method as described in any one of claims 1 to 16.
20. A computer program product comprising a computer program loaded and executed by a processor to perform the operations performed by the video generation method as described in any one of claims 1 to 16.
