Video processing method and apparatus, and electronic device, computer-readable storage medium and computer program product

Abstract

Provided in the present application are a video processing method and apparatus, and an electronic device, a computer-readable storage medium and a computer program product. The method comprises: acquiring a video to be processed, which comprises at least one face; for each face in the video to be processed, on the basis of an image frame of the face comprised the video to be processed, generating a facial video corresponding to the face; adjusting the face in the facial video, so as to obtain an adjusted facial video; and fusing the adjusted facial video and the video to be processed, so as to obtain a target video.

Description

Video processing methods, apparatuses, electronic devices, computer-readable storage media, and computer program products

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202411558835.9, filed on November 1, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a video processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0004] In related technologies, to improve the user's viewing experience, videos can be adjusted to enhance their visual effects. For example, image restoration can be performed on each frame of the video individually. However, this frame-by-frame image restoration method suffers from the problem of inconsistent consistency between adjacent frames, resulting in poor display quality of the processed video. Summary of the Invention

[0005] This application provides a video processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of adjusting facial videos, thereby improving the display effect of the target video.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a video processing method, the method comprising:

[0008] Obtain the video to be processed, which includes at least one face;

[0009] For each face in the video to be processed, a face video corresponding to the face is generated based on the image frames in the video to be processed that include the face;

[0010] The faces in the facial video are adjusted to obtain the adjusted facial video;

[0011] The target video is obtained by fusing the adjusted face video and the video to be processed.

[0012] This application embodiment also provides a video processing apparatus, the apparatus comprising:

[0013] The acquisition module is configured to acquire a video to be processed that includes at least one face;

[0014] The generation module is configured to generate a face video corresponding to each face in the video to be processed, based on image frames in the video to be processed that include the face.

[0015] The adjustment module is configured to adjust the faces in the face video to obtain the adjusted face video;

[0016] The fusion module is configured to fuse the adjusted face video and the video to be processed to obtain the target video.

[0017] This application provides an electronic device, the electronic device comprising:

[0018] Memory, configured to store computer-executable instructions or computer programs;

[0019] When a processor is configured to execute computer-executable instructions or computer programs stored in the memory, it implements the video processing method provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the video processing method provided in this application when executed by a processor.

[0021] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the video processing method provided in this application.

[0022] The embodiments of this application have the following beneficial effects:

[0023] The video processing method provided in this application embodiment can acquire a video to be processed including at least one face. For each face, a face video corresponding to the face is generated based on the image frames including the face in the video to be processed. The face in the face video is adjusted to obtain an adjusted face video. The adjusted face video and the video to be processed are fused to obtain a target video.

[0024] This application can generate a face video corresponding to a face based on the video to be processed, and then adjust the face video as a whole, rather than adjusting each image frame separately. This results in the target video. This method avoids the problem of inconsistency between adjacent image frames, ensuring consistency between adjacent image frames in the target video. It fundamentally solves the video flickering and jump problems caused by frame-by-frame repair in related technologies, improving the accuracy of face video adjustment and thus enhancing the display effect of the target video. Furthermore, it does not adjust the entire "video to be processed," but only the key area of ​​the cropped "face video." Background and other non-face areas in the video are not involved in this time-consuming adjustment process. This greatly reduces the amount of data to be processed and significantly lowers the consumption of computing resources, thereby greatly improving the overall speed of video processing while ensuring the face display effect. Attached Figure Description

[0025] Figure 1 is a schematic diagram of the video processing system provided in an embodiment of this application;

[0026] Figure 2 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application;

[0027] Figure 3 is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0028] Figure 4A is a schematic flowchart of the video processing method provided in an embodiment of this application;

[0029] Figure 4B is a schematic diagram of the video generation model provided in an embodiment of this application;

[0030] Figure 5 is a flowchart illustrating the video processing method provided in this embodiment of the application.

[0031] Figure 6 is a schematic diagram of the identification box provided in an embodiment of this application;

[0032] Figure 7 is a schematic diagram of the identification box provided in an embodiment of this application;

[0033] Figure 8 is a schematic diagram of the identification box provided in an embodiment of this application;

[0034] Figure 9 is a schematic flowchart of the video processing method provided in the embodiments of this application;

[0035] Figure 10 is a structural schematic diagram of the video adjustment model provided in an embodiment of this application;

[0036] Figure 11 is a flowchart illustrating the video processing method provided in an embodiment of this application.

[0037] Figure 12 is a schematic flowchart of the video processing method provided in the embodiment of this application;

[0038] Figure 13 is a schematic flowchart of the video processing method provided in the embodiments of this application;

[0039] Figure 14 is a schematic diagram of the structure of the image description model provided in an embodiment of this application;

[0040] Figure 15 is a flowchart illustrating the video processing method provided in an embodiment of this application.

[0041] Figure 16 is a flowchart illustrating the video processing method provided in an embodiment of this application.

[0042] Figure 17 is a flowchart illustrating the video processing method provided in an embodiment of this application;

[0043] Figure 18 is a schematic diagram of the image frames of an adjusted video in the prior art;

[0044] Figure 19 is a schematic diagram of the image frames of the adjusted video in the prior art (II).

[0045] Figure 20 is a schematic diagram of image frames of the target video provided in an embodiment of this application;

[0046] Figure 21 is a schematic diagram of image frames of the target video provided in an embodiment of this application;

[0047] Figure 22 is a flowchart of the video processing method provided in an embodiment of this application;

[0048] Figure 23 is a schematic flowchart of the video processing method provided in the embodiments of this application, number twelve;

[0049] Figure 24 is a flowchart of the video processing method provided in the embodiments of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0053] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0054] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0055] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0056] Before providing a detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0057] 1) A text prompt, also known as a cue text, is a piece of instruction used in the field of artificial intelligence to guide the model to generate specific outputs, such as guiding the model to generate video. It typically includes task requirements, background information, formatting guidelines, and examples to fully utilize the model's ability to generate appropriate responses for a specific task. The purpose of a prompt is to guide the model to produce the desired response by providing clear instructions or questions, thus allowing for better control over the generated output.

[0058] For example, in a scenario where a video is generated from text, the prompt text may include facial description information (task requirement), action description information of the person corresponding to the face (task requirement), emotion description information of the person corresponding to the face (task requirement), appearance information of the person corresponding to the face (task requirement), scene information of the person (background information), and may also include the format information of the video to be generated (i.e., format specifications).

[0059] 2) Variational Autoencoder (VAE): A VAE is a deep generative model. The representation of the latent space by a VAE is probabilistic; that is, the VAE learns the probability distribution of the input data. The VAE not only learns to extract features from the input data but also learns to acquire the probability distribution of the input data, which can be a normal distribution. A VAE consists of a VAE encoder and a VAE decoder. The VAE encoder is configured to map the input data to the probability distribution in the latent space, and the VAE decoder is configured to sample and reconstruct the input data from the probability distribution.

[0060] 3) Forward Diffusion Process: In diffusion models, the forward diffusion process refers to the gradual addition of noise to the original data until the data is completely transformed into noise. In other words, the forward diffusion process can be viewed as a process where original data gradually becomes noisy data. In diffusion models, the forward diffusion process describes how the original data becomes more random and ambiguous over time, where the noise addition at each step is known. That is, the forward diffusion process starts with the original data, gradually adds noise, and eventually obtains a pure noise distribution (i.e., noisy data).

[0061] 4) Reverse Diffusion Process: The reverse diffusion process is the inverse of the forward diffusion process. It starts with noisy data, gradually removes noise, and finally recovers the original data. The reverse diffusion process can be viewed as the process of recovering original data from noisy data. In diffusion models, the reverse diffusion process describes how to generate original data with specific properties from noise using optimization algorithms. For example, for images, the reverse diffusion process uses optimization algorithms to gradually generate a clear image from noisy data.

[0062] During the research process, the inventors discovered the following technical problems in the relevant technology:

[0063] Technical Issue 1: In related technologies, videos contain multiple image frames. During the video repair process, the image frames in the video can be repaired sequentially. However, the method of repairing image frames separately has the problem that adjacent image frames cannot maintain consistency, resulting in poor display effect of the processed video.

[0064] Technical Issue 2: In related technologies, a face restoration model (codeformer) can be used to repair the image frame to be repaired. For example, the face restoration model can increase the details of the image frame to be repaired, thereby improving the display effect of the repaired image frame. However, there are distortions in the image frame to be repaired. Increasing the details of the image frame to be repaired by the face restoration model will aggravate the distortions in the image frame to be repaired.

[0065] Technical Issue 3: In related technologies, during the process of repairing image frames using a model, the model loses information about smaller parts of the image frame during image processing, resulting in distortion of these smaller parts.

[0066] Technical Issue 4: Following Technical Issue 3, in order to avoid Technical Issue 3, images with a resolution greater than a preset resolution threshold can be used for repair. However, images with a resolution greater than the preset resolution threshold require a lot of computing resources, resulting in a slow speed for image adjustment, which cannot meet the user's needs.

[0067] This application provides a video processing method, video processing apparatus, electronic device, computer-readable storage medium, and computer program product, which can ensure the consistency between adjacent image frames in the target video and improve the accuracy of face video adjustment, thereby improving the display effect of the target video.

[0068] Referring to Figure 1, which is a schematic diagram of the structure of the video processing system provided in the embodiment of this application, the video processing system 100 shown in Figure 1 can support a video processing application. The terminal 400-1 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0069] When a user needs to process a video, in response to a video processing command triggered by the display interface 410-1 of terminal 400-1, the video to be processed can be uploaded to service 200, so that server 200 can receive the video to be processed. The video to be processed includes at least one human face.

[0070] After the server 200 obtains a video to be processed that includes at least one face, it can generate a face video corresponding to each face in the video to be processed based on the image frames containing the face in the video to be processed. The face in the face video is then adjusted to obtain an adjusted face video. The adjusted face video and the video to be processed are then merged to obtain the target video.

[0071] Then, server 200 can send the target video to terminal 400-1, and the user can then view the target video displayed on display interface 410-1 of terminal 400-1. The video processing method provided in this application embodiment can ensure the consistency between adjacent image frames in the target video and improve the accuracy of face video adjustment, thereby improving the display effect of the target video.

[0072] The following describes an electronic device that performs the video processing method provided in the embodiments of this application. The electronic device that performs the video processing method in the embodiments of this application may be a terminal, a server, or a combination of the two.

[0073] The terminal can be any type of device, such as a laptop, tablet, desktop computer, set-top box, smartphone, smart speaker, smartwatch, smart TV, or in-vehicle terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and server can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0074] Referring to Figure 2, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, the electronic device shown in Figure 2 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is configured to enable communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440 in Figure 2.

[0075] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0076] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0077] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0078] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0079] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0080] Operating system 451 includes system programs configured to handle various basic system services and perform hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0081] The network communication module 452 is configured to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0082] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.);

[0083] The input processing module 454 is configured to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0084] In some embodiments, the video processing apparatus provided in this application can be implemented in software. FIG2 shows a video processing apparatus 455 stored in memory 450, which can be software in the form of programs and plug-ins, including the following software modules: acquisition module 4551, generation module 4552, adjustment module 4553, and fusion module 4554. These modules are logically related, and therefore can be arbitrarily combined or split according to the functions they implement. The functions of each module will be described below.

[0085] In other embodiments, the video processing apparatus provided in this application can be implemented in hardware. As an example, the video processing apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0086] In some embodiments, the terminal or server can implement the video processing method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as video APPs or social APPs; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0087] The following describes the video processing method provided in the embodiments of this application. As mentioned above, the electronic device implementing the video processing method of the embodiments of this application can be a terminal, a server, or a combination of both. Referring to Figure 3, which is a flowchart of the video processing method provided in the embodiments of this application, the following describes the video processing method provided in the embodiments of this application with reference to the steps shown in Figure 3, taking the server as the executing entity as an example.

[0088] In step 101, a video to be processed, including at least one human face, is acquired.

[0089] In practical applications, the terminal can be equipped with social applications capable of publishing social information including videos, and video applications capable of adjusting videos. Regarding video applications, while using either social or video applications, users can upload videos containing one or more faces, or users can input text, which the terminal or server then generates a corresponding video containing one or more faces. In other words, the video to be processed can be a user-uploaded video containing one or more faces, or it can be a video generated based on user-input text containing one or more faces.

[0090] Referring to Figure 4A, which is a second schematic flowchart of the video processing method provided in the embodiment of this application, step 101 shown in Figure 3 will be described below with reference to the steps shown in Figure 4A.

[0091] In step 1011, the prompt text used for video generation is obtained.

[0092] In some embodiments, step 1011 can be implemented in the following way: when a user wants to generate a video from text, in response to text input on the terminal's display interface, the terminal can obtain the text input by the user, that is, the terminal can obtain the prompt text used for video generation. In some embodiments, the terminal can send the prompt text to the server, and the server can then obtain the prompt text used for video generation.

[0093] The prompt text includes facial description information, action description information of the person corresponding to the face, emotion description information of the person corresponding to the face, appearance information of the person corresponding to the face, scene information of the person, and format information of the video to be generated, etc., which can be set according to actual usage needs.

[0094] For example, the prompt text could be multiple people (the number of people) playing in a park (the scene where the people are). Some people are flying kites (action description information), some are reading books (action description information), and some are playing soccer (action description information). Among them, the person reading the book occasionally makes a thinking expression (action description information).

[0095] In step 1012, the prompt text is input into the video generation model to obtain a video generated by the video generation model that includes at least one face. The video generated by the video generation model is used as the video to be processed.

[0096] In some embodiments, step 1012 can be implemented in the following way: after obtaining the prompt text, the prompt text can be input into the video generation model. The video generation model can perform text analysis on the prompt text, thereby constructing one or more scenes based on the prompt text. The scenes include people, backgrounds, etc., thereby generating images corresponding to people (including faces) based on the prompt text. Then, the corresponding images can be used to synthesize a video, that is, a video generated by the video generation model including at least one face can be obtained. Then, the video generated by the video generation model including at least one face (one or more faces) can be used as the video to be processed.

[0097] Here, the prompt text may include at least one of the following: facial description information, character action description information, emotion description information, appearance information, scene information of the character, and video format information.

[0098] The training method of the video generation model is described below. The video generation model includes a text analysis layer and a video generation layer. For example, refer to Figure 4B, which is a schematic diagram of the structure of the video generation model provided in the embodiment of this application. The video generation model 401 may include a text analysis layer 402 and a video generation layer 403.

[0099] The text analysis layer can perform natural language processing and entity recognition. When sample prompt text is input into the text analysis layer, it can parse the meaning and context of the sample prompt text, and also identify entities in the sample prompt text. The text analysis layer can output text features.

[0100] The video generation layer can include a feature matching sublayer, a feature concatenation layer, and a feature decoding layer. The feature matching layer can be a Transformer architecture, which preprocesses the sample video to map the sample video into video features. The feature matching layer can learn the correspondence between video features and text features, and thus output the video features corresponding to the text features.

[0101] The feature concatenation layer can concatenate the video features output by the feature matching layer, thereby enabling the concatenation of people and scenes in a single image frame, as well as the concatenation between multiple image frames, to obtain concatenated features. The feature decoding layer can decode the concatenated features output by the feature matching layer, thereby outputting the generated reference video. Based on the differences between the reference video and the sample video, the parameters of the video generation model can be adjusted to obtain the trained video generation model.

[0102] Accordingly, after inputting the prompt text into the video generation model, the sample prompt text in the training process described above can be replaced with the prompt text, thereby obtaining the video to be processed. This will not be elaborated here. In this way, a video to be processed containing at least one face can be obtained.

[0103] Referring again to Figure 3, in step 102, for each face in the video to be processed, a face video corresponding to the face is generated based on the image frames containing the face in the video to be processed.

[0104] The video to be processed may include one or more faces, with one face corresponding to one person. For example, if the video to be processed includes person A, the face is the face corresponding to person A, meaning the video to be processed includes one person's face. Alternatively, if the video to be processed includes people B and C, the faces are the faces corresponding to person B and the faces corresponding to person C, meaning the video to be processed includes multiple faces.

[0105] In some embodiments, step 102 can be implemented by performing face recognition on the video to be processed to obtain a face recognition result, wherein the face recognition result includes the number of faces and the identification information of the faces.

[0106] When a video to be processed contains a face, each image frame of the video can be identified to obtain image frames containing the face. Then, based on the image frames containing the face in the video to be processed, a video of the face corresponding to the face can be generated.

[0107] When a video contains multiple faces, each image frame of the video can be identified to obtain image frames containing faces. These image frames are then divided according to the faces to obtain image frames corresponding to each face. For each image frame corresponding to a face, a face video corresponding to that face can be generated. In other words, there is a one-to-one correspondence between a face and a face video. If there are multiple faces, there will also be multiple face videos.

[0108] In some embodiments, for each face, there are multiple image frames including the face, that is, there are multiple image frames including the face corresponding to one face. Referring to Figure 5, Figure 5 is a schematic flowchart of the video processing method provided in the embodiment of this application. The following describes the method of "generating a face video corresponding to the face based on the image frames including the face in the video to be processed" in step 102 shown in Figure 3 for one face, in conjunction with the steps shown in Figure 5.

[0109] In step 1021, for each image frame in the video to be processed that includes a face, a recognition box including a face is determined in the image frame.

[0110] Here, the recognition box refers to the rectangular bounding box generated by locating a single face region in a single frame of the video to be processed using a face recognition algorithm (such as YOLOv8 or RetinaFace). Its core function is to mark the specific location and extent of the face in a single frame image, typically defined in two ways: one is the pixel coordinates of the upper left and lower right corners of the face region; the other is the pixel coordinates of the center point, width, and height of the face region. In this embodiment, the recognition box is used to initially locate the face region in a single frame image, providing a basis for the subsequent generation of the "target recognition box" based on the single-frame face location.

[0111] After acquiring multiple image frames containing a face, a face recognition algorithm can be used to determine bounding boxes (Bboxes) containing the face within each image frame. The face recognition algorithm can be a first face recognition algorithm (Yolov8), a second face recognition algorithm (RetinaFace), etc., and can be specifically configured according to actual usage requirements.

[0112] A bounding box including a face can be defined by the coordinates of the top-left and bottom-right corners of the face region in an image frame. Alternatively, it can be defined by the coordinates of the center point, width, and height of the face region. The bounding box essentially provides a boundary for the face, used to determine the position and extent of the face within an image frame.

[0113] For example, referring to Figure 6, which is a schematic diagram of the recognition box provided in the embodiment of this application, the video to be processed includes image frame 601. Face recognition is performed on image frame 601, and face 1 and face 2 can be recognized. That is, a recognition box 602 including face 1 and a recognition box 603 including face 2 are determined in image frame 601.

[0114] For example, referring to Figure 7, which is a schematic diagram of the recognition box provided in an embodiment of this application, the video to be processed includes image frame 701. Face recognition is performed on image frame 701, and face 1 can be identified, that is, a recognition box 702 including face 1 is determined in image frame 701. In this way, the area of ​​the face in the image frame can be accurately determined, and the area of ​​the face can be marked by the recognition box.

[0115] Referring to Figure 5, in step 1022, the union of multiple recognition boxes of the face is obtained to obtain the target recognition box.

[0116] In some embodiments, step 1022 can be implemented in the following way: for a face, there are multiple image frames including the face, and correspondingly, there are multiple recognition boxes corresponding to the face, the number of recognition boxes being consistent with the number of image frames including the face.

[0117] The target recognition box is obtained by obtaining the union of multiple recognition boxes of the face. That is, the size of any one of the multiple recognition boxes is less than or equal to the size of the target recognition box. For example, referring to Figures 6 and 7, and referring to Figure 8, which is a schematic diagram of the recognition box provided in the embodiment of this application, the target recognition box 801 is obtained by obtaining the union of two recognition boxes of face 1, that is, the union of recognition box 602 shown in Figure 6 and recognition box 702 shown in Figure 7. In this way, an accurate target recognition box can be obtained.

[0118] Referring to Figure 5, in step 1023, for each image frame of the face, the region corresponding to the target recognition box is extracted from the image frame to obtain a sub-image frame.

[0119] In some embodiments, step 1023 is implemented in the following way: one face corresponds to one target recognition box. After obtaining the target recognition box of the face, for each image frame of the face, the region corresponding to the target recognition box can be extracted from the image frame to obtain a sub-image frame.

[0120] In some embodiments, an image frame has a corresponding recognition box. The center point of the recognition box corresponding to the image frame can be aligned with the center point of the target recognition box, thereby extracting the area corresponding to the target recognition box from the image frame to obtain a sub-image frame. The number of sub-image frames is the same as the number of image frames corresponding to the face.

[0121] Since the target recognition box is the union of multiple recognition boxes, and the size of the target recognition box is larger than the size of the multiple recognition boxes, it is equivalent to being able to extract sub-image frames by using the larger target recognition box. The sub-image frame can include more information about the face. For example, the recognition box may not include the outer corner of the eyes, but the target recognition box, being larger than the recognition box, can include the outer corner of the eyes. In this way, more accurate sub-image frames can be obtained.

[0122] During the alignment and cropping process described above, the integrity of such extended regions can be ensured through "detail preservation verification": After cropping, the contour edges of the face in the sub-image frame are extracted using an edge detection algorithm (such as the Canny algorithm). If the edges of extended regions such as the outer corner of the eye, the tip of the ear, and the chin contour are found to be broken (e.g., the continuous pixel length of the outer corner of the eye is less than 5 pixels), the target recognition box is expanded outward by 5-10 pixels in the corresponding direction (it must be ensured that it is still within the image frame boundary after expansion) and cropped again until the edges of the extended regions are continuous and complete. In this way, sub-image frames that are not only size-fitting but also have higher detail integrity can be obtained, providing richer original feature support for subsequent enlargement and adjustment of face videos.

[0123] In some embodiments, in order to acquire sub-image frames that include more information about the face, that is, to ensure that the sub-image frames include the complete face, the size of the target recognition box can be enlarged after the target recognition box is acquired.

[0124] In some embodiments, the size of the target recognition box can be increased based on the size of the image frame to obtain an enlarged target recognition box. Specifically, the enlargement ratio corresponding to the face can be obtained, and the enlargement ratio corresponding to the face can be preset. For example, the enlargement ratio can be any ratio less than or equal to 2 and greater than or equal to 0.

[0125] The magnification ratio for faces can also be determined based on the size of the target recognition box and the size of the image frame. For example, if the size of the image frame is 5 times the size of the target recognition box, then the magnification ratio can be any ratio between less than 5 and greater than 0. The magnification ratio for faces can also be determined based on the distance between an edge of the target recognition box and the edge of the image frame closest to that edge. The specific settings can be customized according to actual usage requirements.

[0126] After obtaining the scaling factor, based on the scaling factor and the image frame size, the target recognition box can be expanded outwards within the image frame, with the center point of the target recognition box as the origin, to obtain the expanded target recognition box. The size of the expanded target recognition box is less than or equal to the image frame size; that is, the longest side of the expanded target recognition box is less than or equal to the shortest side of the image frame.

[0127] In some embodiments, when the target recognition box is rectangular, the target recognition box can be first expanded into a square along its longest side. Then, based on the square and the expansion ratio, with the center point of the square as the origin, the square is expanded in the image frame to obtain the expanded target recognition box.

[0128] For example, if the target recognition box is 1 cm (width) x 2 cm (length), it can be expanded by 2 cm to form a square of 2 cm x 2 cm. The expansion ratio is 2. The center point of the 2 cm x 2 cm square can be used as the origin to expand the square in the image frame to obtain the expanded target recognition box, which is 4 cm x 4 cm in size.

[0129] In some embodiments, an initial magnification ratio corresponding to the face can be obtained. The initial magnification ratio is a preset magnification ratio. The target recognition box is magnified according to the initial magnification ratio with the center point of the target recognition box as the origin. If the magnified target recognition box does not exceed the boundary of the image frame, the magnified target recognition box can be obtained.

[0130] If the enlarged target recognition box exceeds the boundary of the image frame, the initial enlargement ratio is reduced to obtain the target enlargement ratio. Then, using the center point of the target recognition box as the origin, the target recognition box is enlarged according to the target enlargement ratio to obtain the enlarged target recognition box. The target enlargement ratio can be determined based on the distance between the edge of the target recognition box and the edge of the image frame closest to that edge.

[0131] After obtaining the enlarged target recognition bounding box, for each image frame of the face, the region corresponding to the enlarged target recognition bounding box is extracted from the image frame to obtain a sub-image frame. Since the target recognition bounding box may not include areas such as the chin, ears, and hair corresponding to the person, in order to include more facial information in the sub-image frame, that is, to include areas such as the chin, ears, and hair, the target recognition bounding box can be expanded outwards to obtain a sub-image frame that includes the complete face, that is, a more accurate sub-image frame can be obtained.

[0132] During the generation of face videos, if the face in the image frames of the video to be processed is partially occluded (e.g., by hand, object, etc.), the occlusion region can be detected by combining a semantic segmentation model. Using the occlusion mask output by the semantic segmentation model, the position and area percentage of the occluded region are determined: when the occlusion area percentage is less than a preset threshold (e.g., 15%), the original target recognition box expansion logic is maintained; when the occlusion area percentage is greater than or equal to the preset threshold, the expansion ratio is reduced by 30%–50% from the original value, and priority is given to retaining the recognition box range corresponding to the unoccluded core facial areas (e.g., eyes, nose), avoiding excessive expansion that results in the cropped sub-image frames containing a large amount of invalid occluded background, further improving the targeting of subsequent face video adjustments.

[0133] In step 1024, a face video corresponding to the face is generated based on multiple sub-image frames of the face.

[0134] In some embodiments, step 1024 can be implemented in the following way: there is an order among the various image frames in the video to be processed, and each image frame can be marked with an identifier. That is, there is an order among the identifiers, and sub-image frames can be extracted from the image frames. Accordingly, the sub-image frames have identifiers.

[0135] After obtaining multiple sub-image frames of a face, the sub-image frames can be sorted based on the identifier to obtain a sub-image frame sequence. Then, the sub-image frames can be video encoded to obtain the face video corresponding to the face.

[0136] In some embodiments, each sub-image frame has corresponding time information, which characterizes the display time of the sub-image frame in the video to be processed. Multiple sub-image frames can be sorted in ascending order of the time indicated by the time information to obtain a sub-image frame sequence. This sequence can then be video encoded to obtain the face video corresponding to the face. In this way, accurate face videos can be obtained, which are then used to subsequently obtain the target video, thereby improving the accuracy of the target video.

[0137] In some embodiments, step 1024 can also be implemented in the following way, see FIG9, FIG9 is a schematic flowchart of the video processing method provided in the embodiments of this application. The steps 1024 shown in FIG5 are described below with reference to the steps shown in FIG9.

[0138] In step 10241, the scene in which the face is located in the video to be processed is determined.

[0139] In some embodiments, step 10241 can be implemented by analyzing the video to be processed based on image frames of the face in the video to be processed, thereby determining the scene in which the face is located in the video to be processed.

[0140] Specifically, starting from the first image frame in the video to be processed where a face appears, a face tracking algorithm is used to track the face, thereby sequentially identifying image frames containing faces in each image frame of the video to be processed, and then analyzing the scene of the face in each image frame to determine the scene in the video to be processed.

[0141] In some embodiments, the identity of a face can be determined, and behavioral analysis can be performed on the person corresponding to the face to obtain the analysis results, thereby determining the scene in which the face is located in the video to be processed based on the identity of the face and the behavioral analysis results of the person.

[0142] In step 10242, based on the scene in which the face is located in the video to be processed, multiple sub-image frames of the face are sorted to obtain a sub-image frame sequence.

[0143] In some embodiments, step 10242 can be implemented in the following way: if the number of multiple faces in the scene in the video to be processed is 1, then multiple sub-image frames of the face can be sorted according to the identification or time information to obtain a sub-image frame sequence.

[0144] If multiple faces appear in multiple scenes within the video to be processed, the multiple sub-image frames of the faces can be classified according to scene, resulting in sub-image frames corresponding to each scene. For each scene-specific sub-image frame, these frames can be sorted according to identifiers or time information, resulting in a scene-specific sub-image frame sequence. This sequence can then be sorted according to the order in which the scenes appear in the video to be processed, thus obtaining a precise sub-image frame sequence.

[0145] In step 10243, the sub-image frame sequence is video encoded to obtain the face video corresponding to the face.

[0146] In some embodiments, step 10243 can be implemented as follows: after obtaining the sub-image frame sequence, corresponding parameters can be set according to the frame rate, encoding format, resolution, bit rate, etc. of the video to be processed, and the sub-image frame sequence can be input to the video encoder. The video encoder can then perform video encoding on the sub-image frame sequence to obtain the face video corresponding to the face. Through steps 10241-10243, an accurate face video can be obtained. This face video is used to subsequently obtain the target video, thereby improving the accuracy of the target video. Furthermore, this application processes the face video as a whole, which can solve the aforementioned technical problem 1.

[0147] Referring to Figure 3, in step 103, the size of the face video is enlarged to obtain the enlarged face video.

[0148] In some embodiments, step 103 can be implemented by obtaining a preset magnification size, magnifying the size of the face video, and obtaining the magnified face video, the size of which is the magnification size.

[0149] The preset magnification size is determined based on the adjustment effect of the video adjustment model. That is, the video adjustment model’s repair effect on face videos with a size greater than the preset size threshold, and the repair effect on face videos with a size less than or equal to the preset size threshold. Therefore, the magnification size can be set to any size greater than the preset size threshold.

[0150] Here, when enlarging the size of a face video, an adaptive algorithm selection mechanism can be introduced: First, the sharpness score of each sub-image frame in the face video is calculated (based on the sum of edge gradient magnitudes). If the sharpness score is higher than a preset score threshold (e.g., 80 points), a bilinear interpolation algorithm is used for enlargement, balancing enlargement speed and basic image quality. If the sharpness score is lower than the preset score threshold, it automatically switches to a super-resolution algorithm based on generative adversarial networks (e.g., ESRGAN). Through a pre-trained face-specific super-resolution model, facial details (e.g., skin texture, eyelash contours) are repaired while enlarging the size, avoiding the blurring and jaggedness caused by traditional interpolation algorithms. In addition, the enlargement batch can be dynamically adjusted according to the frame rate of the face video during the enlargement process: when the frame rate is higher than 30fps, a batch enlargement mode (processing 10 frames per batch) is used; when the frame rate is lower than or equal to 30fps, a single-frame successive enlargement mode is used to ensure the temporal consistency of the enlarged video.

[0151] In some embodiments, for step 102, for each face in the video to be processed, after identifying image frames in the video to be processed that include faces, the image frames including faces can be filtered based on the size of the faces in the image frames including faces.

[0152] Determine whether the size of the face in the image frame containing the face is greater than a preset size threshold. If the size of the face is greater than the preset size threshold, then the image frame with the face size greater than the preset size threshold is taken as the first image frame, and a first face video corresponding to the face is generated based on the first image frame. The first face video does not execute step 103 shown in Figure 3, but executes step 104 shown in Figure 3.

[0153] If the size of the face is less than or equal to a preset size threshold, then the image frame whose size is less than or equal to the preset size threshold is used as the third image frame, and the above step 102 is performed on the third image frame. Specifically, the steps shown in Figure 5 can be performed on the third image frame. The steps shown in Figure 5 can be replaced with equivalent steps, which will not be elaborated here. Thus, a second face video of the face can be generated for the third image frame, and step 103 can be performed on the second face video of the face.

[0154] In other words, the face video is generated from sub-image frames where the size of the face in the image frame is smaller than the size threshold. This application can enlarge the face video and then make adjustments based on the enlarged face video. This can prevent the model from losing small-sized information during image processing and avoid distortion of small-sized faces, thereby solving the above-mentioned technical problem 3.

[0155] In step 104, the faces in the magnified face video are adjusted to obtain the adjusted face video.

[0156] In some embodiments, step 104 can be implemented by inputting the magnified face video into a video adjustment model, which can adjust the face in the magnified face video to obtain an adjusted face video. In other words, the video adjustment model can perform video restoration on the magnified face video. Specifically, the video adjustment model can perform face restoration on the face in the magnified face video.

[0157] The video adjustment model may include a feature extraction layer, a noise addition layer, a noise reduction layer, and a decoding layer. Referring to Figure 10, which is a schematic diagram of the structure of the video adjustment model provided in an embodiment of this application, the video adjustment model 1001 includes a feature extraction layer 1002, a noise addition layer 1003, a noise reduction layer 1004, and a decoding layer 1005.

[0158] In some embodiments, step 104 can be implemented in the following manner, as shown in FIG11. FIG11 is a schematic flowchart of the video processing method provided in the embodiments of this application. The steps 104 shown in FIG3 will be described below with reference to the steps shown in FIG11.

[0159] In step 1041, video features of the magnified face video are extracted.

[0160] In some embodiments, step 1041 can be implemented by extracting video features of the magnified face video through a feature extraction layer. The feature extraction layer may include a VAE encoder, which can encode the magnified face video into latent space features.

[0161] In step 1042, the video features are iteratively denoised to obtain the first denoised features.

[0162] Here, iterative noise addition refers to the process of repeatedly performing the steps of "multiplying the feature to be processed by the attenuation factor + multiplying the noise by the attenuation factor + summing to obtain the noisy feature" after extracting video features from the magnified face video, until the preset number of repetitions is reached. The "feature to be processed" initially consists of video features, and subsequent iterations use features after the previous noise addition. The "noise" is sampled from a standard normal distribution, and the attenuation factors (first attenuation factor, second attenuation factor) dynamically change with the number of repetitions (the first attenuation factor decreases with increasing repetitions, and the second attenuation factor increases with increasing repetitions) to simulate the "noise degradation process" of video features, providing a noisy feature basis for subsequent "iterative denoising" based on the target text.

[0163] In some embodiments, step 1042 can be implemented by iteratively adding noise to the video features through a noise layer to obtain a first noise-added feature. In some embodiments, the number of iterations of noise addition, i.e. the number of repetitions, can be a preset number or the number of repetitions can be determined by an algorithm. The noise intensity and the first step length can be obtained, and the noise intensity can be multiplied by the first step length to obtain the second step length, wherein the second step length is the number of repetitions.

[0164] Referring to Figure 12, which is a schematic flowchart of the video processing method provided in the embodiment of this application, the iterative noise addition process will be described below with reference to the steps shown in Figure 12. Specifically, step 1042 shown in Figure 11 can be achieved by repeatedly executing the steps shown in Figure 12. That is, by repeatedly executing the steps shown in Figure 12 a certain number of times, the first noise addition feature can be obtained.

[0165] In step 10421, the feature to be processed is obtained. In step 10422, the feature to be processed is multiplied by a first attenuation factor to obtain a first multiplication result. In step 10423, the noise corresponding to the current repetition number is obtained, and the noise corresponding to the current repetition number is multiplied by a second attenuation factor to obtain a second multiplication result. In step 10424, the sum of the first multiplication result and the second multiplication result is used as the noisy feature.

[0166] For ease of understanding, the steps 10421-10424 shown in Figure 12 will be explained below. When the current repetition count (i.e. the current noise addition repetition count) is 0, that is, when the noise addition repetition count is 0, the feature to be processed is the video feature. Then, the video feature can be multiplied by the first attenuation factor to obtain the first multiplication result.

[0167] Different attenuation factors can be set for different number of noise repetitions. The attenuation factor decreases as the second step length increases, which means that the attenuation factor decreases as the number of noise repetitions increases. In other words, the attenuation factor is inversely correlated with the number of noise repetitions. The attenuation factor can be set to any value between 0 and 1.

[0168] The first attenuation factor can be determined based on the attenuation factor, that is, the first attenuation factor is the square root attenuation factor. Correspondingly, the first attenuation factor decreases as the second step size increases, that is, the first attenuation factor decreases as the number of repetitions of adding noise increases. In other words, the first attenuation factor is inversely correlated with the number of repetitions of adding noise. The first attenuation factor is used to control the intensity of noise during each repetition of steps 10421-10424.

[0169] Based on the video features, noise corresponding to the current repetition count can also be obtained, and the noise corresponding to the current repetition count is multiplied by the second attenuation factor to obtain the second multiplication result. The second attenuation factor is determined based on the attenuation factor, that is, the second attenuation factor is the square root of 1 minus the attenuation factor. Correspondingly, the second attenuation factor increases with the increase of the second step size, that is, the second attenuation factor increases with the increase of the number of repetitions of adding noise. In other words, the second attenuation factor is positively correlated with the number of repetitions of adding noise. The second attenuation factor is used to control the intensity of the added noise during each repetition of steps 10421-10424.

[0170] The noise corresponding to the current repetition number is sampled from a standard normal distribution. That is, in each repetition of steps 10421-10424, the noise corresponding to the current repetition number is randomly obtained from the standard normal distribution.

[0171] After obtaining the first and second multiplication results, the sum of the first and second multiplication results is used as the noise-added feature. The noise-added feature is the noise-added feature corresponding to the current repetition count of 0.

[0172] Then, steps 10421-10424 can be repeated. When the current repetition count is not 0, the feature to be processed is the feature after the previous denoising. That is, when the current repetition count is 1, the feature to be processed is the feature after denoising when the repetition count is 0.

[0173] Furthermore, the noise-added feature corresponding to the repetition count of 0 can be multiplied by the first attenuation factor to obtain the first multiplication result. Alternatively, the noise corresponding to the current repetition count can be obtained and multiplied by the second attenuation factor to obtain the second multiplication result.

[0174] After obtaining the first and second multiplication results, the sum of the first and second multiplication results is used as the noise-added feature. The noise-added feature is the noise-added feature corresponding to the current repetition count of 1.

[0175] Continue repeating steps 10421-10424. If the current repetition count is not 0, the feature to be processed is the feature after the previous noisy iteration. That is, if the current repetition count is 2, the feature to be processed is the noisy feature corresponding to the repetition count of 1. Specifically, an equivalent substitution can be performed for the case where the current repetition count is 1, which will not be elaborated here.

[0176] After repeating steps 10421-10424 a preset number of times (corresponding to the second step length), the noise-added feature is used as the first noise-added feature. For example, if the preset number of repetitions is 25, the noise-added feature corresponding to the current repetition count of 25 can be used as the first noise-added feature. In this way, iterative noise addition to video features can be achieved to obtain an accurate first noise-added feature.

[0177] In some embodiments, step 1042 can be implemented using formula (1), which is explained below. Where x t Let a be the noisy feature corresponding to the current repetition count t. t This is the decay factor when the current repetition count is t. x is the first decay factor when the current repetition count is t. t-1 This refers to the feature after adding noise in the previous iteration, which is the feature after adding noise when the current repetition count is t-1. The second attenuation factor, ∈ t This represents the noise at the current repetition count of t.

[0178] In some embodiments, step 1042 can be implemented by Denoising Diffusion Implicit Models (DDIM), and the first noise-added feature can be obtained by the above formula (1). In this way, the video features can be iteratively noise-added to obtain the accurate first noise-added feature.

[0179] Here, when iteratively adding noise to video features, an adaptive noise type selection logic can be introduced: by analyzing the noise distribution characteristics of the magnified face video (such as Gaussian noise, salt-and-pepper noise), if Gaussian noise is detected as the main interference, Gaussian noise is used for iterative noise addition; if salt-and-pepper noise is detected, Poisson noise is switched to add noise, making the noise addition process more consistent with the actual noise scene of the video. Simultaneously, the number of iterations can be dynamically adjusted according to the length of the face video: when the video length is less than 10 seconds, the preset number of repetitions is set to 20; when the video length is between 10 and 30 seconds, the preset number of repetitions is set to 25; and when the video length is greater than 30 seconds, the preset number of repetitions is set to 30, achieving a balance between adjustment accuracy and processing efficiency. During the iterative denoising process, the initial values ​​of the third and fourth attenuation factors can be adjusted according to the complexity of the target text: when the target text contains more than 5 facial description dimensions (such as expressions, makeup, and posture), the initial value of the third attenuation factor is increased by 10% and the initial value of the fourth attenuation factor is decreased by 10%, thereby enhancing the denoising process's sensitivity to the target text.

[0180] In step 1043, the target text corresponding to the face is obtained, and based on the target text, the first noisy feature is iteratively denoised to obtain the second noisy feature.

[0181] In some embodiments, there are multiple ways to obtain the target text corresponding to a face. The first method of obtaining the target text corresponding to a face will be described below. Referring to Figure 13, Figure 13 is a schematic flowchart of the video processing method provided in the embodiment of this application. The first method of obtaining the target text will be described below with reference to the steps shown in Figure 13.

[0182] In step 10431, the first image frame is extracted from the magnified face video.

[0183] In some embodiments, step 10431 can be implemented as follows: for each face, the first image frame in the magnified face video corresponding to that face can be used as the first image frame. Alternatively, for each face, a pre-set key image frame in the magnified face video corresponding to that face can be used as the first image frame. Of course, any image frame in the magnified face video corresponding to the face can also be used as the first image frame according to actual usage requirements.

[0184] In step 10432, target text corresponding to the face is generated based on the first image frame.

[0185] In some embodiments, step 10432 can be implemented in the following way: after obtaining the first image frame, the first image frame can be input into a pre-trained image description model, and the image feature of the first image frame can be extracted by the image description model to obtain the image feature of the first image frame. Based on the image feature of the first image frame, the image description prediction of the first image frame can be performed by the image description model to obtain the description text used to describe the first image frame, and then the target text corresponding to the face can be extracted from the description text.

[0186] The image description model includes a feature extraction layer and an image description prediction layer. For example, referring to Figure 14, Figure 14 is a schematic diagram of the structure of the image description model provided in an embodiment of this application. The image description model 1401 includes a feature extraction layer 1402 and an image description layer 1403.

[0187] The feature extraction layer is configured to extract image features, while the image description prediction layer can be configured according to actual usage requirements. For example, the image description prediction layer can be configured for face recognition, and correspondingly, the descriptive text describing the first image frame can include text representing the face's identity. Alternatively, the image description prediction layer can be configured for emotion recognition, and correspondingly, the descriptive text describing the first image frame can include text representing the face's emotion.

[0188] The training method of the image description model is explained below. A sample image can be input into the image description model. The feature extraction layer can extract the image features of the sample image. Then, based on the image features of the sample image, the image description model can predict the image description of the sample image to obtain the predicted description text used to describe the sample image. The parameters of the image description model can be updated based on the difference between the predicted description text and the corresponding sample text of the sample image, so as to obtain the trained image description model.

[0189] By using the first method of obtaining the target text corresponding to the face, it is possible to generate target text describing the face based on the first image frame in the face video, which can improve the accuracy of the generated target text. The target text is used to adjust the face video, thereby improving the accuracy of the face video adjustment.

[0190] The second method for obtaining the target text corresponding to the face is explained below. For the case where the video to be processed is generated based on the prompt text and the video generation model, the prompt text corresponding to the face can be extracted from the prompt text as the target text. In this way, the target text that accurately matches the face can be obtained. The target text is used to adjust the face video, thereby improving the accuracy of the face video adjustment.

[0191] In practical applications, when acquiring target text corresponding to a face, a multimodal information fusion mechanism can be added: If the video to be processed contains an audio track, emotional features (such as joy, calmness, and anger) in the audio are extracted using an audio emotion recognition model to generate emotional description text (such as "joyful expression" or "calm demeanor"). Simultaneously, head posture information (such as looking forward, looking sideways, or looking upward) corresponding to the face is extracted using a human pose estimation model to generate pose description text (such as "looking directly at the camera" or "looking sideways at 30°"). The basic target text generated by the image description model, the emotional description text extracted from the audio, and the pose description text extracted from the pose are fused together. A weighted voting mechanism (basic text weight 0.6, emotional text weight 0.2, pose text weight 0.2) is used to select core descriptive words, ultimately generating multi-dimensional target text, making subsequent face adjustments more consistent with the overall scene and the state of the person in the video.

[0192] Regarding step 1043, after obtaining the target text, the first noisy feature is iteratively denoised based on the target text to obtain the second noisy feature. In some embodiments, the first noisy feature can be obtained by iteratively denoising the first noisy feature through a denoising layer based on the target text.

[0193] The iterative denoising process is described below. Referring to Figure 15, which is a flowchart of the video processing method provided in this application embodiment, step 1043, "based on the target text, iteratively denoise the first denoising feature to obtain the second denoising feature", can be achieved by repeatedly executing the steps shown in Figure 15. That is, by repeatedly executing the steps shown in Figure 15 until the number of repetitions is reached (the number of repetitions for denoising is the same as the number of repetitions for denoising), the second denoising feature can be obtained.

[0194] In step 10431, the feature to be denoised is obtained. In step 10432, the feature to be denoised is multiplied by the third attenuation factor to obtain the third multiplication result. In step 10433, the predicted noise corresponding to the current repetition count, the feature to be denoised, and the target text is obtained, and the predicted noise is multiplied by the fourth attenuation factor to obtain the fourth multiplication result. In step 10434, the difference between the third multiplication result and the fourth multiplication result is used as the denoised feature.

[0195] For ease of understanding, the steps 10431-10434 shown in Figure 15 will be explained below. When the current repetition count (i.e. the current denoising repetition count) is 0, the feature to be denoised is the first denoising feature. Then, the first denoising feature can be multiplied by the third attenuation factor to obtain the third multiplication result.

[0196] The cumulative attenuation factor varies depending on the number of repetitions. The cumulative attenuation factor can be determined based on the attenuation factor and the number of repetitions for denoising. The attenuation factors in the noise addition process are arranged in descending order of the number of repetitions. The corresponding attenuation factors in the sorted list are deleted sequentially according to the number of repetitions for denoising. The remaining attenuation factors are then summed to obtain the cumulative attenuation factor.

[0197] When the current repetition count is 0, meaning the denoising repetition count is 0, there's no need to delete any attenuation factor in the sorting process. Therefore, all attenuation factors from the denoising process can be summed to obtain the cumulative attenuation factor when the current repetition count is 0. The cumulative attenuation factor decreases as the denoising repetition count increases; that is, the cumulative attenuation factor is inversely correlated with the repetition count.

[0198] The third attenuation factor can be determined based on the cumulative attenuation factor, that is, the third attenuation factor is the reciprocal of the square root of the cumulative attenuation factor. Correspondingly, the third attenuation factor increases with the increase of the number of denoising repetitions, that is, the third attenuation factor is positively correlated with the number of denoising repetitions. The third attenuation factor is used to control the intensity of noise removal during each repetition of steps 10431-10434.

[0199] The predicted noise corresponding to the current repetition count, the feature to be denoised, and the target text is obtained. The predicted noise is predicted by the denoising layer based on the current repetition count, the feature to be denoised, and the target text. The predicted noise is multiplied by the fourth attenuation factor to obtain the fourth multiplication result.

[0200] The fourth attenuation factor is determined based on the cumulative attenuation factor, which is the square root of 1 minus the quotient of the cumulative attenuation factor and the third attenuation factor. The fourth attenuation factor is used to control the intensity of the noise during each repetition of steps 10431-10434.

[0201] After obtaining the results of the third and fourth multiplications, the difference between the third and fourth multiplications is used as the denoised feature. The denoised feature is the feature corresponding to the current repetition count of 0.

[0202] Then, steps 10431-10434 can be repeated. When the current repetition count is not 0, the feature to be denoised is the feature after the previous denoising. That is, when the current repetition count is 1, the feature to be denoised is the feature after denoising when the repetition count is 0.

[0203] Furthermore, the denoised features corresponding to the number of repetitions of 0 can be multiplied by the third attenuation factor to obtain the third multiplication result. The predicted noise corresponding to the current number of repetitions, the features to be denoised, and the target text can also be obtained. The predicted noise can be multiplied by the fourth attenuation factor to obtain the fourth multiplication result.

[0204] After obtaining the results of the third and fourth multiplications, the difference between the third and fourth multiplications is used as the denoised feature. The denoised feature is the feature corresponding to the current repetition count of 1.

[0205] Continue repeating steps 10421-10424. If the current repetition count is not 0, the feature to be denoised is the feature after the previous denoising. That is, if the current repetition count is 2, the feature to be processed is the denoised feature corresponding to the repetition count of 1. Specifically, the case where the current repetition count is 1 can be equivalently replaced, which will not be elaborated here.

[0206] After repeating steps 10431-10434 a preset number of times (corresponding to the second step size), the denoised feature is used as the second denoised feature. For example, if the preset number of repetitions is 25, the denoised feature corresponding to the current repetition count of 25 can be used as the second denoised feature. In this way, iterative denoising of the first denoised feature can be achieved to obtain an accurate second denoised feature.

[0207] In some embodiments, step 1043 can be implemented using formula (2), which is explained below. Wherein, xn is the denoised feature corresponding to the current repetition count of n. When n equals 0, the denoised feature is the first noisy feature. bn is the cumulative attenuation factor when the current repetition count is n. is the third decay factor when the current repetition count is n, and xn-1 is the feature after the previous denoising, which is the denoising feature corresponding to the current repetition count of n-1. It is the fourth attenuation factor. is the predicted noise when the current repetition count is n, which is the predicted noise obtained based on the current repetition count, the features to be denoised, and the target text. c is the target text.

[0208] In some embodiments, step 1043 can be implemented by a Denoising Diffusion Implicit Model (DDIM), specifically by obtaining the second noisy feature using the above formula (2). In this way, the first noisy feature can be iteratively denoised to obtain an accurate second noisy feature.

[0209] In step 1044, the second noisy feature is decoded to obtain the adjusted face video.

[0210] In some embodiments, step 1044 can be implemented by decoding the second noise-adding feature through a decoding layer to obtain the adjusted face video. This method can improve the accuracy of face video adjustment, achieve face video restoration, and thus improve the display effect of the target video.

[0211] The training method of the video adjustment model is explained below. After obtaining the sample video, it can be input into the video adjustment model to extract the sample video features. The sample video features are iteratively denoised to obtain the first denoised feature. The sample text is then obtained, and based on the sample text, the first denoised feature is iteratively denoised to obtain the second denoised feature. The second denoised feature can then be decoded to obtain the predicted video. Based on the difference between the predicted video and the corresponding adjusted sample video, that is, the difference between the image frames of the predicted video and the image frames of the adjusted sample video, the parameters of the video adjustment model are adjusted, thereby obtaining the trained video adjustment model.

[0212] In some embodiments, for the second face video generated from the third image frame, step 103 shown in FIG3 may not be performed on the face video, but step 104 shown in FIG3 may be performed instead, that is, the second face video is adjusted by the video adjustment model to obtain the adjusted second face video.

[0213] In some embodiments, in order to improve the display effect of the target video, step 104 shown in FIG3 can be performed on the video to be processed, that is, the video to be processed is adjusted by the video adjustment model to obtain the adjusted video to be processed.

[0214] Referring again to Figure 3, in step 105, the adjusted face video and the video to be processed are fused to obtain the target video. This method avoids the inconsistency problem between adjacent image frames, ensuring consistency between adjacent image frames in the target video. It fundamentally solves the video flickering and jump problems caused by frame-by-frame repair in related technologies, improving the accuracy of face video adjustment and thus enhancing the display effect of the target video. Furthermore, it does not adjust the entire "video to be processed," but only the key area of ​​the cropped "face video." Background and other non-face areas in the video are not involved in this time-consuming adjustment process. This greatly reduces the amount of data to be processed, significantly lowers the consumption of computing resources, and thus greatly improves the overall speed of video processing while ensuring the face display effect.

[0215] In some embodiments, step 105 is implemented in the following manner, as shown in FIG16. FIG16 is a schematic flowchart of the video processing method provided in the embodiments of this application. The steps 105 shown in FIG3 are described below with reference to the steps shown in FIG16.

[0216] The adjusted face video (i.e., the adjusted first face video) includes multiple second image frames. These second image frames correspond to image frames in the video to be processed that include faces. In some embodiments, the identifier of the second image frame is consistent with the identifier of the corresponding image frame containing a face. In some embodiments, the time information of the second image frame is consistent with the time information of the corresponding image frame containing a face.

[0217] In step 1051A, a gradient mask is created for each second image frame in the adjusted face video.

[0218] In some embodiments, step 1051A can be implemented in the following way: an initial mask with all pixel values ​​of 0 and the same size as the second image frame is created; based on the pixels corresponding to the face in the second image frame, the pixel values ​​of the pixels in the initial mask that are related to the pixels corresponding to the face are set to 1. That is, the pixel values ​​of the area corresponding to the face in the initial mask are set to 1, while the pixel values ​​of the area not corresponding to the face are set to 0.

[0219] The Gaussian kernel and standard deviation can be set according to actual usage requirements. The Gaussian function value is calculated using the Gaussian formula to obtain the Gaussian kernel template. The Gaussian kernel template is then convolved on the initial mask to obtain a gradient mask. The pixel values ​​of the gradient mask decrease from the center to the edge. The gradient mask is a grayscale image. The pixel value in the central region of the gradient mask is 1, and it decreases from the center region to the edge region until the pixel value is 0.

[0220] In step 1052A, based on the gradient mask, the second image frame and the corresponding image frame including the face are fused to obtain the first fused image frame.

[0221] In this process, the pixel of the gradient mask is the first pixel, the pixel in the second image frame is the second pixel, the first pixel of the gradient mask corresponds one-to-one with the second pixel of the second image frame, and the pixel of the image frame containing the face in the video to be processed is the third pixel. The third pixel in the image frame containing the face has a corresponding relationship with the first pixel, that is, the third pixel in the image frame containing the face has a corresponding relationship with the second pixel.

[0222] For each first pixel in the gradient mask, multiply the first pixel by the second pixel corresponding to the first pixel to obtain a first product; multiply the value determined based on the first pixel by the third pixel corresponding to the first pixel to obtain a second product; and use the sum of the first product and the second product as the fourth pixel.

[0223] The fourth pixel can be calculated using formula (3), which is explained below. M4 is the fourth pixel, M1 is the first pixel (its value can be any value between 0 and 1), M2 is the second pixel, M3 is the third pixel, and (1-M1) is a value determined based on the first pixel, which is the difference between 1 and the first pixel. M4 = M1*M2 + (1-M1)*M3 (3)

[0224] After determining the fourth pixel corresponding to each first pixel, that is, after determining the fourth pixel corresponding to the third pixel, the third pixel in the image frame including the face corresponding to the second image frame can be replaced with the fourth pixel corresponding to the third pixel to obtain the first fused image frame.

[0225] Accordingly, the first fused image frame can be determined using formula (4), which will be explained below, where F is the first fused image frame, G is the gradient mask, and rI f For the second image frame, I f For each image frame in the video to be processed that includes a face, the gradient mask, the second image frame, and the image frame containing the face all correspond to each other. F = G * rI f +(1-G)*I f (4)

[0226] Step 1052A allows for the fusion of the second image frame and its corresponding image frame containing a face through a gradient blending method. This creates a smooth transition visual effect between the second image frame and its corresponding image frame containing a face, avoiding black borders between the two images and thus improving the display effect of the first fused image frame.

[0227] In step 1053A, the image frames containing faces in the video to be processed are replaced with the first fused image frames to obtain the target video.

[0228] In some embodiments, the number of first fused image frames is consistent with the number of second image frames in the adjusted face video. After obtaining multiple first fused image frames, the image frames including faces in the video to be processed can be replaced with the first fused image frames to obtain the target video.

[0229] For the case where there is only one face, the image frames containing the face in the video to be processed can be replaced with the corresponding first fused image frames to obtain the target video. For the case where there are multiple faces, for each face, the image frames containing the face in the video to be processed can be replaced with the corresponding first fused image frames. After all the image frames corresponding to all faces have been replaced with the corresponding first fused image frames, the target video is obtained. This method allows for the acquisition of a more accurate target video.

[0230] In some embodiments, step 105 is further implemented in the following manner, as shown in FIG17, which is a schematic flowchart of the video processing method provided in the embodiments of this application. The steps shown in FIG17 will be described below with reference to the steps shown in FIG17. The adjusted face video includes multiple second image frames, and the second image frames correspond to the image frames containing faces in the video to be processed.

[0231] In step 1051B, for each second image frame of the adjusted face video, a face image is extracted from the second image frame, the face image is reduced in size, and a reduced face image is obtained.

[0232] In some embodiments, step 1051B can be implemented in the following way: since the second image frame is extracted from the image frame containing the face in the video to be processed based on the target recognition box or the enlarged target recognition box, and the second image frame includes regions that do not belong to the face, the face image is extracted from the second image frame.

[0233] Since the size of the face video was enlarged in step 103 shown in Figure 3, that is, the second image frame is the image frame corresponding to the face after the size was enlarged, in order to merge the adjusted face video with the video to be processed, the face image can be reduced to obtain a reduced face image.

[0234] In some embodiments, the face image can be reduced in size, wherein the reduced size is greater than or equal to the size of the image frame in the face video, and the reduced size corresponds to different second image frames for different faces. A precisely reduced face image can be obtained through step 1051B.

[0235] In step 1052B, the target region in the image frame containing the face corresponding to the second image frame is determined.

[0236] In some embodiments, step 1052B can be implemented by determining, for a second image frame, an image frame including a face in the video to be processed corresponding to the second image frame based on identification or time information.

[0237] In some embodiments, the target region in the image frame containing the face corresponding to the second image frame can be determined based on the target recognition bounding box or a magnified target recognition bounding box. In some embodiments, the target region in the image frame containing the face corresponding to the second image frame can also be determined based on the face recognition bounding box. The specific settings can be configured according to actual usage requirements.

[0238] In step 1053B, the scaled-down face image and the target region are fused to obtain the second fused image frame.

[0239] In some embodiments, step 1053B can be implemented in the following way: after obtaining the scaled-down face image, the edge region of the scaled-down face image can be determined, wherein the edge region is the region that is far away from the preset pixel interval in the scaled-down face image.

[0240] Then, the pixel gradient of the edge region of the scaled-down face image can be calculated, and the Poisson equation can be established based on the pixel gradient. In other words, a system of linear equations can be established based on the pixel gradient. The system of linear equations can describe the pixel values ​​in the target region calculated by the pixel gradient.

[0241] By using a system of linear equations, the pixel values ​​in the target region can be obtained. Then, the calculated pixel values ​​in the target region can be used to replace the source pixel values ​​in the target region, thereby obtaining the second fused image frame.

[0242] Here, Poisson editing refers to a technique for smoothly fusing a fused face image with a target area in a video, based on pixel gradient information. Its core logic is: extracting the edge pixel gradients of the scaled-down face image, establishing a Poisson equation (a system of linear equations) based on the gradients, and calculating the pixel values ​​of the target area in the video by solving the system of equations, ensuring that the pixel values ​​of the target area and the scaled-down face image are continuous in gradient. In this embodiment, Poisson editing is used to solve the "edge connection problem between the scaled-down face image and the target area in the video," ensuring a natural transition in texture and brightness between the fused face and the background, without obvious stitching marks.

[0243] In step 1054B, the image frames containing faces in the video to be processed are replaced with the second fused image frames to obtain the target video.

[0244] Step 1054B shown in Figure 17 is consistent with step 1053A shown in Figure 16. For details, please refer to the description of step 1053A shown in Figure 16, which will not be repeated here. Through steps 1051B-1054B, the second image frame and the corresponding image frame including the face can be merged by Poisson editing. Similarly, a smooth transition visual effect can be created between the second image frame and the corresponding image frame including the face, avoiding black frames between the second image frame and the corresponding image frame including the face, thereby improving the display effect of the first merged image frame.

[0245] In some embodiments, when fusing the adjusted face video with the video to be processed, an illumination consistency calibration step can be added: First, the illumination parameters of the target region (the region corresponding to the face) in the video to be processed are extracted, including the average brightness value, color temperature value, and illumination direction vector; then, the illumination characteristics of the second image frame are adjusted based on these parameters: the average brightness value of the second image frame is corrected to be consistent with the target region using a brightness mapping algorithm, the color temperature of the second image frame is adjusted using a color temperature conversion matrix, and the shadow distribution of the second image frame is adjusted using a directional illumination simulation algorithm, so that the adjusted face region and the background illumination of the video to be processed remain consistent. For example, if the average brightness of the target region in the video to be processed is 150 cd / m²... 2 If the color temperature is 5500K and the lighting direction is 45° from the upper left, the brightness and color temperature of the second image frame will be corrected to the corresponding values, and a faint shadow will be added to the upper left of the face to avoid problems such as "the face brightness is too high / too low" or "the color temperature deviation causes the skin color to be unnatural" after fusion, thereby improving the visual coherence of the target video.

[0246] In some embodiments, the adjusted second face video, the adjusted first face video, and the adjusted video to be processed can be merged to obtain the target video. For details, please refer to the above description, which will not be repeated here.

[0247] The video processing method provided in this application has the following technical effects, which are described in detail below. This application can generate a face video corresponding to a face based on the video to be processed, and then adjust the face video as a whole, rather than adjusting each image frame separately. This allows for the acquisition of the target video. In this way, the problem of inconsistency between adjacent image frames can be avoided, ensuring consistency between adjacent image frames in the target video, improving the accuracy of face video adjustment, and thus improving the display effect of the target video. This solves the aforementioned technical problems 1-2.

[0248] The video processing method provided in this application embodiment can enlarge the size of the face video during the video processing process, thereby adjusting the face in the enlarged face video. This allows for more precise adjustment of the face, improves the accuracy of face video adjustment, and thus improves the display effect of the target video, thereby solving the above-mentioned technical problems 2-3.

[0249] The video processing method provided in this application embodiment, during the generation of a face video, after obtaining the target recognition box, can enlarge the size of the target recognition box by expanding it outwards. This ensures that the enlarged target recognition box includes content such as the chin, ears, and hair, thereby allowing adjustment of the face video including the chin, ears, and hair. It ensures that when the adjusted face video and the video to be processed are fused, there will be no difference between the content corresponding to the face, improving the accuracy of face video adjustment and enhancing the display effect of the target video.

[0250] In the process of adjusting a face video, the embodiments of this application can adjust the face video based on the target text, which can ensure that the face in the face video is consistent with the target text, thereby improving the accuracy of face video adjustment and improving the display effect of the target video.

[0251] The embodiments of this application do not limit the resolution of the image frames in the video to be processed. Even if the resolution of the image frames in the video to be processed is less than the preset resolution threshold, a target video with a better display effect can still be obtained. Compared with adjusting images with a resolution greater than the preset resolution threshold, the speed of adjusting face videos can be improved, meeting the user's needs. That is, the embodiments of this application can solve the above-mentioned technical problem 4.

[0252] Referring to Figures 18-21, Figure 18 is a schematic diagram of the image frames of an adjusted video in the prior art, Figure 19 is a schematic diagram of the image frames of an adjusted video in the prior art, Figure 20 is a schematic diagram of the image frames of a target video provided in an embodiment of this application, and Figure 21 is a schematic diagram of the image frames of a target video provided in an embodiment of this application.

[0253] As can be seen from Figures 18 and 19, in Figure 18, the mouth and eyes of face 1801 are significantly distorted. The eyes and nose of face 1802 are also significantly distorted. In Figure 19, the eyes 1901 of the face are significantly distorted. All faces in Figures 18 and 19 exhibit distortion, resulting in poor display quality.

[0254] Referring to Figures 20 and 21, in Figure 20, for the same image frame as Figure 18, the mouth and eyes of face 2001 are not distorted compared to face 1801 in Figure 18. Similarly, the eyes and nose of face 2002 are not distorted compared to face 1802 in Figure 18. This means that this application can adjust faces more precisely, improving the accuracy of face video adjustment and thus enhancing the display effect of the target video.

[0255] In Figure 21, the eyes 2101 of the face are not distorted, and the nose and mouth of the face shown in Figure 21 are more coordinated than those shown in Figure 19. It can be seen that this application can adjust the face more precisely, improve the accuracy of face video adjustment, and thus improve the display effect of the target video.

[0256] To facilitate understanding of the video processing method provided in the embodiments of this application, the video processing method provided in the embodiments of this application will be illustrated below with reference to Figures 22-24. Referring to Figure 22, Figure 22 is a flowchart illustration eleven of the video processing method provided in the embodiments of this application. Referring to Figure 23, Figure 23 is a flowchart illustration twelfth of the video processing method provided in the embodiments of this application. Referring to Figure 24, Figure 24 is a flowchart illustration thirteenth of the video processing method provided in the embodiments of this application.

[0257] In step 2201 shown in Figure 22, a video to be processed, including at least one human face, is acquired.

[0258] Regarding step 2201, please refer to the description of step 101 shown in Figure 3. In some embodiments, the content creator (i.e., the user) can input prompt text, which can then be used to generate a video to be processed using the prompt text and a video generation model (or a video engine calling an algorithm). In some embodiments, the user uploads a video to be processed, which is a video containing low-quality faces.

[0259] In step 2202, the number of faces in the video to be processed is identified, and for each face, the image frames in the video to be processed that include that face are determined.

[0260] Regarding step 2202, after each face is identified in the video to be processed, a face identifier (ID) can be created for each face, and then a face tracking algorithm can be used to determine the image frame containing that face from the video to be processed.

[0261] In step 2203, a face video corresponding to the face is generated based on the image frames containing faces in the video to be processed.

[0262] For each face, for each image frame containing the face, a bounding box containing the face is determined in the image frame. The union of multiple bounding boxes of the face is obtained to obtain the target bounding box. Based on the size of the image frame, the size of the target bounding box is enlarged to obtain the enlarged target bounding box. For each image frame of the face, the region corresponding to the enlarged target bounding box is extracted from the image frame to obtain a sub-image frame. Based on multiple sub-image frames of the face, a face video corresponding to the face is generated. The face video can be identified using vi.

[0263] Any face can be identified using an Idi identifier, the bounding box corresponding to the face can be identified using a bbox identifier, and the target bounding box can be identified using a BBOXi identifier. One face corresponds to one target bounding box, that is, one Idi corresponds to one BBOXi.

[0264] In some embodiments, the shape of BBOXi can be determined. If the shape of BBOXi is not a square, BBOXi can be expanded to a square. For example, if the shape of BBOXi is a rectangle, the long side of BBOXi can be expanded (i.e., expanded outward) along the short side of BBOXi and away from the center of BBOXi until BBOXi is expanded to a square.

[0265] Furthermore, using the center point of the BBOXi as the origin, and based on the scaling ratio and the image frame size, the BBOXi can be expanded outwards within the image frame to obtain the expanded target recognition bounding box. The region corresponding to the expanded target recognition bounding box overlaps with the region corresponding to the image frame; that is, the region corresponding to the expanded target recognition bounding box does not exceed the region corresponding to the image frame. The expanded target recognition bounding box can be identified using boardBBOXi.

[0266] In step 2204, the size of the face video is enlarged to obtain an enlarged face video.

[0267] Regarding step 2204, assuming the size of the face video vi is (w, h), the face video vi can be enlarged to obtain an enlarged face video with a size of (W, H), where W is greater than w, H is greater than h, W is the enlarged width, H is the enlarged height (also called enlarged length), w is the original width, and h is the original height (also called original length). W and H can be the same; for example, the enlarged face video can be 512x512 or 768x768. The enlarged face video can be identified using the `scale_vi` identifier. This method allows for the enlargement of the face size, preventing the model from losing information about smaller faces, enabling more precise adjustments to the face, improving the accuracy of face video adjustments, and thus enhancing the display effect of the target video.

[0268] For steps 2202 to 2204, in Figure 23, face recognition can be performed on the image frame 2301 containing the face, thereby determining the recognition box of the face included in the image frame 2301 containing the face. In Figure 23, only the recognition box 2302 is marked, and then the union of multiple recognition boxes of the face can be obtained to obtain the target recognition box. Based on the size of the image frame, the size of the target recognition box is enlarged to obtain the enlarged target recognition box.

[0269] For each image frame of the face, the region corresponding to the enlarged target recognition box is extracted from the image frame to obtain a sub-image frame. For the video to be processed, multiple sub-image frames for the face can be obtained. In Figure 23, only sub-image frame 2303 is marked. Based on the multiple sub-image frames of the face, a face video corresponding to the face is generated.

[0270] In step 2205, the target text is obtained.

[0271] Step 2205 can be found in the descriptions of the first and second acquisition methods above. That is, the first image frame of scale_vi can be used as the first image frame, which can then be input into the image description model to obtain the description text. The target text corresponding to the face can then be extracted from the description text. Alternatively, in the case where a video to be processed is generated from prompt text, the prompt text corresponding to the face can be extracted from the prompt text as the target text.

[0272] In step 2206, video features of the magnified face video are extracted, and the video features are iteratively denoised to obtain the first denoised feature. Based on the target text, the first denoised feature is iteratively denoised to obtain the second denoised feature. The second denoised feature is then decoded to obtain the adjusted face video.

[0273] Regarding step 2206, as shown in Figure 24, the magnified face video 2401 can be input into the feature extraction layer 2403 of the video adjustment model to obtain the video features of the magnified face video. Based on the video features of the magnified face video and random noise 2402, the video features are iteratively denoised to obtain the first denoised feature. Then, the first denoised feature and the features of the target text 2406 can be input into the denoising layer 2404. The denoising layer 2404 can iteratively denoise the first denoised feature to obtain the second denoised feature. Then, the second denoised feature can be decoded to obtain the adjusted face video 2405. The iterative denoising and denoising processes can be found in the corresponding descriptions and will not be elaborated here.

[0274] In step 2207, the adjusted face video and the video to be processed are fused together to obtain the target video.

[0275] For step 2207, please refer to the description of step 105 shown in Figure 3, which will not be repeated here. This application can generate a face video corresponding to the face based on the video to be processed, and then adjust the face video as a whole, rather than adjusting each image frame separately, thereby obtaining the target video. This method avoids the problem of inconsistency between adjacent image frames, ensuring consistency between adjacent image frames in the target video, improving the accuracy of face video adjustment, and thus improving the display effect of the target video. Furthermore, during video processing, this application can enlarge the size of the face video, thereby adjusting the face in the enlarged face video, enabling more precise face adjustment, improving the accuracy of face video adjustment, and thus improving the display effect of the target video.

[0276] The following continues to describe an exemplary structure of the video processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2, the software module stored in the video processing apparatus 455 in the memory 450 may include:

[0277] The acquisition module 4551 is configured to acquire a video to be processed that includes at least one face;

[0278] The generation module 4552 is configured to generate a face video corresponding to each face in the video to be processed, based on image frames in the video to be processed that include the face.

[0279] The adjustment module 4553 is configured to adjust the face in the face video to obtain the adjusted face video;

[0280] The fusion module 4554 is configured to fuse the adjusted face video and the video to be processed to obtain the target video.

[0281] In some embodiments, the number of image frames is multiple; the generation module 4552 is further configured to, for each image frame in the video to be processed that includes the face, determine a recognition box including the face in the image frame; obtain the union of the multiple recognition boxes of the face to obtain a target recognition box; for each image frame of the face, extract the region corresponding to the target recognition box from the image frame to obtain a sub-image frame; and generate a face video corresponding to the face based on the multiple sub-image frames of the face.

[0282] In some embodiments, the generation module 4552 is further configured to, after obtaining the union of multiple recognition boxes of the face to obtain a target recognition box, expand the size of the target recognition box based on the size of the image frame to obtain an expanded target recognition box; and for each image frame of the face, extract the region corresponding to the expanded target recognition box from the image frame to obtain a sub-image frame.

[0283] In some embodiments, the generation module 4552 is further configured to obtain the magnification ratio corresponding to the face; based on the magnification ratio and the size of the image frame, with the center point of the target recognition box as the origin, the target recognition box is magnified in the image frame to obtain the magnified target recognition box; wherein the size of the magnified target recognition box is less than or equal to the size of the image frame.

[0284] In some embodiments, the generation module 4552 is further configured to determine the scene in which the face is located in the video to be processed; based on the scene in which the face is located in the video to be processed, sort the multiple sub-image frames of the face to obtain a sub-image frame sequence; and perform video encoding on the sub-image frame sequence to obtain a face video corresponding to the face.

[0285] In some embodiments, the device further includes an enlargement module configured to enlarge the size of the face video to obtain an enlarged face video; and an adjustment module 4553 configured to adjust the face in the enlarged face video to obtain an adjusted face video.

[0286] In some embodiments, the adjustment module 4553 is further configured to: extract video features of the magnified face video; iteratively add noise to the video features to obtain a first noisy feature; obtain the target text corresponding to the face, and iteratively denoise the first noisy feature based on the target text to obtain a second noisy feature; and decode the second noisy feature to obtain the adjusted face video.

[0287] In some embodiments, the adjustment module 4553 is further configured to extract a first image frame from the magnified face video; and generate target text corresponding to the face based on the first image frame.

[0288] In some embodiments, the adjustment module 4553 is further configured to extract image features from the first image frame using an image description model to obtain image features of the first image frame; based on the image features of the first image frame, perform image description prediction on the first image frame using the image description model to obtain descriptive text for describing the first image frame; and extract the target text corresponding to the face from the descriptive text.

[0289] In some embodiments, the acquisition module 4551 is further configured to acquire prompt text for video generation; input the prompt text into a video generation model to obtain a video generated by the video generation model including at least one face, and use the video generated by the video generation model as the video to be processed; the adjustment module 4553 is further configured to extract prompt text corresponding to the face from the prompt text as the target text.

[0290] In some embodiments, the adjustment module 4553 is further configured to repeatedly execute the following steps until the number of repetitions of the steps reaches a preset number of repetitions: using the noise-added feature as the first noise-added feature; obtaining a feature to be processed, wherein, when the current number of repetitions is 0, the feature to be processed is the video feature, and when the current number of repetitions is not 0, the feature to be processed is the feature after the previous noise addition; multiplying the feature to be processed by a first attenuation factor to obtain a first multiplication result; obtaining the noise corresponding to the current number of repetitions, and multiplying the noise corresponding to the current number of repetitions by a second attenuation factor to obtain a second multiplication result; and using the sum of the first multiplication result and the second multiplication result as the noise-added feature.

[0291] In some embodiments, the magnification module is further configured to compare the size of the face in the image frame including the face with a preset size threshold;

[0292] If the size of the face is less than or equal to the preset size threshold, the size of the face video is enlarged to obtain the enlarged face video.

[0293] In some embodiments, the adjusted face video includes a plurality of second image frames, and the second image frames correspond to image frames in the video to be processed that include the face; the fusion module 4555 is further configured to create a gradient mask for each of the second image frames in the adjusted face video, wherein the pixel values ​​of the gradient mask decrease from the center to the edge; based on the gradient mask, the second image frames and the corresponding image frames including the face are fused to obtain a first fused image frame; the image frames including the face in the video to be processed are replaced with the first fused image frames to obtain the target video.

[0294] The fusion module 4555 is also configured to create an initial mask with all pixel values ​​of 0 and a size consistent with the second image frame;

[0295] Set the pixel value of the region corresponding to the face in the initial mask to 1;

[0296] Obtain the Gaussian kernel template and convolve the Gaussian kernel template on the initial mask to obtain the gradient mask.

[0297] In some embodiments, the first pixel of the gradient mask corresponds one-to-one with the second pixel of the second image frame, including the third pixel in the image frame containing the face and the first pixel having a corresponding relationship; the fusion module 4555 is further configured to, for each first pixel in the gradient mask, multiply the first pixel by the second pixel corresponding to the first pixel to obtain a first product; multiply the value determined based on the first pixel by the third pixel corresponding to the first pixel to obtain a second product; take the sum of the first product and the second product as the fourth pixel; and replace the third pixel in the image frame containing the face corresponding to the second image frame with the fourth pixel corresponding to the third pixel to obtain a first fused image frame.

[0298] In some embodiments, the adjusted face video includes a plurality of second image frames, and the second image frames correspond to image frames in the video to be processed that include the face; the fusion module 4555 is further configured to, for each second image frame of the adjusted face video, extract a face image from the second image frame, reduce the size of the face image to obtain a reduced face image; determine a target region in the image frame containing the face corresponding to the second image frame; fuse the reduced face image and the target region to obtain a second fused image frame; and replace the image frames in the video to be processed that include the face with the second fused image frame to obtain the target video.

[0299] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video processing method described in this application.

[0300] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the video processing method provided in this application, such as the video processing method shown in FIG3.

[0301] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0302] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0303] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0304] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0305] The video processing method provided in this application embodiment can generate a face video corresponding to a face based on the video to be processed, and then adjust the face video as a whole, rather than adjusting each image frame in the video separately, so as to obtain the target video. In this way, the problem of inconsistency between adjacent image frames can be avoided, ensuring the consistency between adjacent image frames in the target video, improving the accuracy of adjusting the face video, and thus improving the display effect of the target video.

[0306] The video processing method provided in this application embodiment can enlarge the size of a face video during the video processing process, thereby adjusting the face in the enlarged face video. This allows for more precise adjustment of the face, improving the accuracy of the face video adjustment and thus enhancing the display effect of the target video.

[0307] The video processing method provided in this application embodiment, during the generation of a face video, after obtaining the target recognition box, can enlarge the size of the target recognition box by expanding it outwards. This ensures that the enlarged target recognition box includes content such as the chin, ears, and hair, thereby allowing adjustment of the face video including the chin, ears, and hair. It ensures that when the adjusted face video and the video to be processed are fused, there will be no difference between the content corresponding to the face, improving the accuracy of face video adjustment and enhancing the display effect of the target video.

[0308] In the process of adjusting a face video, the embodiments of this application can adjust the face video based on the target text, which can ensure that the face in the face video is consistent with the target text, thereby improving the accuracy of face video adjustment and improving the display effect of the target video.

[0309] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

A video processing method, applied to an electronic device, the method comprising: Obtain the video to be processed, which includes at least one face; For each face in the video to be processed, a face video corresponding to the face is generated based on the image frames in the video to be processed that include the face; The faces in the facial video are adjusted to obtain the adjusted facial video; The target video is obtained by fusing the adjusted face video and the video to be processed. According to the method of claim 1, wherein, The number of image frames is multiple; The step of generating a face video corresponding to the face based on image frames including the face in the video to be processed includes: For each image frame in the video to be processed that includes the face, a recognition box including the face is determined in the image frame; Obtain the union of multiple recognition boxes of the face to obtain the target recognition box; For each image frame of the face, the region corresponding to the target recognition box is extracted from the image frame to obtain a sub-image frame; A face video corresponding to the face is generated based on multiple sub-image frames of the face. The method according to claim 1 or 2, wherein, After obtaining the union of multiple recognition boxes of the face to obtain the target recognition box, the method further includes: Based on the size of the image frame, the size of the target recognition box is increased to obtain the enlarged target recognition box; For each image frame of the face, extracting the region corresponding to the target recognition box from the image frame to obtain a sub-image frame includes: For each image frame of the face, the region corresponding to the enlarged target recognition box is extracted from the image frame to obtain a sub-image frame. The method according to claim 3, wherein, The step of enlarging the target recognition box based on the size of the image frame to obtain the enlarged target recognition box includes: Obtain the magnification ratio corresponding to the face; Based on the enlargement ratio and the size of the image frame, the target recognition box is expanded outward in the image frame with the center point of the target recognition box as the origin, to obtain the enlarged target recognition box. The size of the enlarged target recognition box is less than or equal to the size of the image frame. The method according to any one of claims 2 to 4, wherein, The step of generating a face video corresponding to the face based on multiple sub-image frames of the face includes: Determine the scene in which the face is located in the video to be processed; Based on the scene in which the face is located in the video to be processed, the multiple sub-image frames of the face are sorted to obtain a sub-image frame sequence; The sub-image frame sequence is video encoded to obtain the face video corresponding to the face. The method according to any one of claims 1 to 5, wherein, After generating the facial video corresponding to the face, the method further includes: The size of the face video is enlarged to obtain an enlarged face video; The step of adjusting the faces in the face video to obtain the adjusted face video includes: The faces in the magnified face video are adjusted to obtain the adjusted face video. The method according to claim 6, wherein, The step of adjusting the face in the magnified face video to obtain the adjusted face video includes: Extract the video features from the magnified face video; The video features are iteratively denoised to obtain the first denoised features; Obtain the target text corresponding to the face, and based on the target text, iteratively denoise the first noisy feature to obtain the second noisy feature; The second noisy feature is decoded to obtain the adjusted face video. The method according to claim 7, wherein, The step of obtaining the target text corresponding to the face includes: Extract the first image frame from the magnified face video; Based on the first image frame, the target text corresponding to the face is generated. The method according to claim 8, wherein, The step of generating target text corresponding to the face based on the first image frame includes: Image features of the first image frame are obtained by extracting image features from the first image frame using an image description model; Based on the image features of the first image frame, the image description model is used to predict the image description of the first image frame to obtain the descriptive text used to describe the first image frame. Extract the target text corresponding to the face from the description text. The method according to any one of claims 7 to 9, wherein, The acquisition of the video to be processed, which includes at least one face, includes: Get the prompt text used for video generation; The prompt text is input into the video generation model to obtain a video generated by the video generation model that includes at least one face. The video generated by the video generation model is used as the video to be processed. The step of obtaining the target text corresponding to the face includes: Extract the prompt text corresponding to the face from the prompt text and use it as the target text. The method according to any one of claims 7 to 9, wherein, The iterative noise addition to the video features to obtain the first noisy feature includes: Repeat the following steps until the number of repetitions reaches a preset number of repetitions, and use the noise-added feature as the first noise-added feature; Obtain the feature to be processed, wherein, when the current repetition count is 0, the feature to be processed is the video feature, and when the current repetition count is not 0, the feature to be processed is the feature after the previous noise addition; The feature to be processed is multiplied by the first attenuation factor to obtain the first multiplication result; Obtain the noise corresponding to the current repetition count, and multiply the noise corresponding to the current repetition count by the second attenuation factor to obtain the second multiplication result; The sum of the first multiplication result and the second multiplication result is used as the feature after adding noise. The method according to any one of claims 6 to 11, wherein, The process of enlarging the size of the face video to obtain an enlarged face video includes: The size of the face in the image frame including the face is compared with a preset size threshold; If the size of the face is less than or equal to the preset size threshold, the size of the face video is enlarged to obtain the enlarged face video. The method according to any one of claims 1 to 12, wherein, The adjusted face video includes multiple second image frames, and the second image frames correspond to the image frames in the video to be processed that include the face. The process of fusing the adjusted face video and the video to be processed to obtain the target video includes: For each of the second image frames in the adjusted face video, a gradient mask is created, wherein the pixel values ​​of the gradient mask decrease from the center to the edge; Based on the gradient mask, the second image frame and the corresponding image frame including the face are fused to obtain the first fused image frame; The image frames containing the human face in the video to be processed are replaced with the first fused image frames to obtain the target video. The method according to claim 13, wherein, The first pixel of the gradient mask corresponds one-to-one with the second pixel of the second image frame, including the third pixel in the image frame of the face which corresponds to the first pixel; The step of fusing the second image frame and the corresponding image frame including the face based on the gradient mask to obtain the first fused image frame includes: For each first pixel in the gradient mask, the first pixel is multiplied by the second pixel corresponding to the first pixel to obtain a first product; the value determined based on the first pixel is multiplied by the third pixel corresponding to the first pixel to obtain a second product; the sum of the first product and the second product is used as the fourth pixel. The third pixel in the image frame containing the face corresponding to the second image frame is replaced with the fourth pixel corresponding to the third pixel to obtain the first fused image frame. The method according to claim 13, wherein, The creation of the gradient mask includes: Create an initial mask with all pixel values ​​of 0 and the same size as the second image frame; Set the pixel value of the region corresponding to the face in the initial mask to 1; Obtain the Gaussian kernel template and convolve the Gaussian kernel template on the initial mask to obtain the gradient mask. The method according to any one of claims 1 to 15, wherein, The adjusted face video includes multiple second image frames, and the second image frames correspond to the image frames in the video to be processed that include the face. The process of fusing the adjusted face video and the video to be processed to obtain the target video includes: For each second image frame of the adjusted face video, a face image is extracted from the second image frame, and the face image is reduced in size to obtain a reduced face image; Determine the target region in the image frame containing the face corresponding to the second image frame; The reduced face image and the target region are fused to obtain a second fused image frame; The image frames containing the human face in the video to be processed are replaced with the second fused image frames to obtain the target video. A video processing apparatus, the apparatus comprising: The acquisition module is configured to acquire a video to be processed that includes at least one face; The generation module is configured to generate a face video corresponding to each face in the video to be processed, based on image frames in the video to be processed that include the face. The adjustment module is configured to adjust the faces in the face video to obtain the adjusted face video; The fusion module is configured to fuse the adjusted face video and the video to be processed to obtain the target video. An electronic device, the electronic device comprising: Memory, configured to store computer-executable instructions or computer programs; When a processor is configured to execute computer-executable instructions or computer programs stored in the memory, it implements the video processing method according to any one of claims 1 to 16. A computer-readable storage medium storing computer-executable instructions or a computer program, which, when executed by a processor, implement the video processing method according to any one of claims 1 to 16. A computer program product comprising computer-executable instructions or a computer program that, when executed by a processor, implements the video processing method according to any one of claims 1 to 16.

Images