Model training method, apparatus, device, medium, and product

By stitching together feature representations of high- and low-resolution multimedia data along the channel dimension and training a diffusion model, the problem of low accuracy in generating high-resolution videos in existing technologies is solved, training efficiency and robustness are improved, and the application scenarios of the model are expanded.

WO2026233344A1PCT designated stage Publication Date: 2026-11-12TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/095108
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-05-07
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing super-resolution techniques are prone to generating false details when generating high-resolution videos, resulting in low generation accuracy and insufficient training efficiency and robustness.

Method used

By stitching together the noisy feature representations of high-resolution and low-resolution multimedia data along the channel dimension, feature fusion is performed using a diffusion model, and the model is trained using a loss function to improve training efficiency and robustness.

Benefits of technology

It enables a comprehensive consideration of information from different dimensions in multimedia data, improves the training efficiency of diffusion models and the accuracy of generating high-resolution data, and expands the application scenarios of the models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026095108_12112026_PF_FP_ABST
    Figure CN2026095108_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The present application discloses a model training method, an apparatus, a device, a medium, and a product. The method comprises: acquiring first multimedia data and second multimedia data; adding a first added noise to the first multimedia data to obtain first noise-added data, and adding the first added noise to the second multimedia data to obtain second noise-added data; concatenating a first noise-added feature representation of the first noise-added data and a second noise-added feature representation of the second noise-added data in a channel dimension by means of a diffusion model to be trained, so as to obtain a first concatenated feature representation; and predicting a first denoised noise on the basis of the first concatenated feature representation, determining a first loss function on the basis of the first denoised noise, and using the first loss function to train said diffusion model, so as to obtain a first diffusion model.
Need to check novelty before this filing date? Find Prior Art

Description

Model training methods, devices, equipment, media and products

[0001] This application claims priority to Chinese Patent Application No. 202510602204.0, filed on May 9, 2025, entitled “Multimedia Data Processing Method, Apparatus, Device, Medium and Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of machine learning, and in particular to a model training method, apparatus, device, medium, and product. Background Technology

[0003] With the development of computer technology, more and more fields need to upgrade the resolution of existing low-resolution images or videos to obtain high-resolution images or videos in order to improve the accuracy of data analysis.

[0004] In some methods, the techniques for generating high-resolution images / videos from low-resolution images / videos mainly rely on super-resolution (SR) techniques. For example, the upscaling a video (Upscale-A-Video) technique is used to decompose the input low-resolution video into single-frame images, and then a super-resolution network (such as SR convolutional neural network, enhanced SR generative adversarial network, etc.) is applied to each frame image to generate a higher resolution image.

[0005] Technical content

[0006] This application provides a model training method, apparatus, device, medium, and product that can achieve multi-scale information fusion through channel-dimensional feature splicing processing. This enables comprehensive consideration of multimedia data with the same content but different resolutions across different dimensions, improving the training efficiency and robustness of the resulting diffusion model. The technical solution is as follows.

[0007] This application provides a model training method executed by an electronic device, the method comprising:

[0008] Acquire first multimedia data and second multimedia data, the first multimedia data and the second multimedia data have the same content but different resolutions, and the first resolution of the first multimedia data is higher than the second resolution of the second multimedia data;

[0009] Adding a first noise to the first multimedia data to obtain first noisy data, and adding the first noise to the second multimedia data to obtain second noisy data;

[0010] The first concatenated feature representation is obtained by concatenating the first noisy feature representation of the first noisy data and the second noisy feature representation of the second noisy data in the channel dimension using the diffusion model to be trained. The number of channels in the first concatenated feature representation is the sum of the number of channels in the first noisy feature representation and the second noisy feature representation.

[0011] Using the diffusion model to be trained, a first denoised noise is predicted based on the first spliced ​​feature representation, a first loss function is determined based on the first denoised noise, and the diffusion model to be trained is trained using the first loss function to obtain a first diffusion model.

[0012] This application also provides another model training method, executed by an electronic device, the method comprising:

[0013] Acquire third multimedia data and fourth multimedia data, wherein the third multimedia data is image data including biological objects, and the fourth multimedia data is video data including the biological images;

[0014] A second noise is added to the fourth multimedia data to obtain the fourth noisy data;

[0015] By concatenating the third data feature representation of the third multimedia data and the fourth noisy feature representation corresponding to the fourth noisy data in the channel dimension using the diffusion model to be trained, a second concatenated feature representation is obtained, wherein the number of channels in the second concatenated feature representation is the sum of the number of channels in the third data feature representation and the fourth noisy feature representation;

[0016] Using the diffusion model to be trained, the second denoised noise is predicted based on the second spliced ​​feature representation, the second loss function is determined based on the second denoised noise, and the diffusion model to be trained is trained using the second loss function to obtain the second diffusion model.

[0017] This application embodiment also provides a model training apparatus, the apparatus comprising:

[0018] The first acquisition module is used to acquire first multimedia data and second multimedia data, wherein the first multimedia data and the second multimedia data have the same content but different resolutions, and the first resolution of the first multimedia data is higher than the second resolution of the second multimedia data.

[0019] The first noise-adding module is used to add first noise to the first multimedia data to obtain first noise-adding data, and to add the first noise to the second multimedia data to obtain second noise-adding data;

[0020] The first splicing module is used to splice the first noisy feature representation of the first noisy data and the second noisy feature representation of the second noisy data in the channel dimension through the diffusion model to be trained, so as to obtain the first spliced ​​feature representation, wherein the number of channels of the first spliced ​​feature representation is the sum of the number of channels of the first noisy feature representation and the second noisy feature representation;

[0021] The first training module is used to predict the first denoised noise based on the first concatenated feature representation using the diffusion model to be trained, determine the first loss function based on the first denoised noise, and train the diffusion model to be trained using the first loss function to obtain the first diffusion model.

[0022] This application embodiment also provides another model training apparatus, wherein the apparatus includes:

[0023] The second acquisition module is used to acquire third multimedia data and fourth multimedia data, wherein the third multimedia data is image data including biological objects, and the fourth multimedia data is video data including the biological images;

[0024] The second noise-adding module is used to add second noise to the fourth multimedia data to obtain fourth noise-adding data;

[0025] The second splicing module is used to splice the third data feature representation of the third multimedia data and the fourth noise-added feature representation corresponding to the fourth noisy data in the channel dimension through the diffusion model to be trained, so as to obtain the second spliced ​​feature representation, wherein the number of channels of the second spliced ​​feature representation is the sum of the number of channels of the third data feature representation and the fourth noise-added feature representation;

[0026] The second training module is used to predict the second denoised noise based on the second concatenated feature representation using the diffusion model to be trained, determine the second loss function based on the second denoised noise, and train the diffusion model to be trained using the second loss function to obtain the second diffusion model.

[0027] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement any of the model training methods described in the above embodiments of this application.

[0028] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement any of the model training methods described in the above embodiments of this application.

[0029] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the model training methods described in the above embodiments.

[0030] Brief description of the attached figures

[0031] Figure 1 is a structural block diagram of a training system provided in some embodiments of this application;

[0032] Figure 2 is a flowchart of model training provided in some embodiments of this application;

[0033] Figure 3 is a flowchart of model training provided in some other embodiments of this application;

[0034] Figure 4 is a training architecture diagram of a diffusion model to be trained provided in some embodiments of this application;

[0035] Figure 5 is a flowchart of model training provided in some embodiments of this application;

[0036] Figure 6 is a flowchart of model training provided in some embodiments of this application;

[0037] Figure 7 is a model architecture diagram of video generation provided in some embodiments of this application;

[0038] Figure 8 is a schematic diagram of block processing provided in some embodiments of this application;

[0039] Figure 9 is a schematic diagram of feature splicing provided in some embodiments of this application;

[0040] Figure 10 is a schematic diagram of the interface for resolution enhancement provided in some embodiments of this application;

[0041] Figure 11 is a schematic diagram of the interface for resolution enhancement provided in some other embodiments of this application;

[0042] Figure 12 is a schematic diagram of the attitude transformation interface provided in some embodiments of this application;

[0043] Figure 13 is a schematic diagram of the attitude transformation interface provided in some other embodiments of this application;

[0044] Figure 14 is a schematic diagram of the dynamic portrait interface provided in some embodiments of this application;

[0045] Figure 15 is a schematic diagram of the dynamic portrait interface provided in some other embodiments of this application;

[0046] Figure 16 is a structural block diagram of a model training device provided in some embodiments of this application;

[0047] Figure 17 is a structural block diagram of a model training device provided in some other embodiments of this application;

[0048] Figure 18 is a structural block diagram of a server provided in some other embodiments of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0050] In some methods, the technique of generating high-resolution images from low-resolution images mainly relies on super-resolution techniques. Although video super-resolution techniques can effectively convert low-resolution videos into high-resolution videos, the generation of high-resolution videos is achieved by guessing high-frequency details through algorithms, which can easily generate some false details, resulting in lower accuracy in the generation of high-resolution videos.

[0051] This application presents a model training method that achieves multi-scale information fusion through channel-dimensional feature concatenation. This allows for comprehensive consideration of multimedia data with the same content but different resolutions across different dimensions, improving the training efficiency and robustness of the diffusion model. The method proposed in this application can be applied to various multimedia data processing scenarios, such as game video resolution enhancement, film and television restoration, video enhancement, medical image super-resolution generation, and satellite remote sensing image enhancement, without limitation.

[0052] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0053] The training system involved in the embodiments of this application will be described. The model training method provided in the embodiments of this application can be implemented by the terminal alone, by the server, or by the terminal and the server through data interaction. The embodiments of this application do not limit this. In some embodiments, the model training method is described using the interaction between the terminal and the server.

[0054] For illustration, please refer to Figure 1. This training system involves a terminal 110 and a server 120, which are connected via a communication network 130.

[0055] In some embodiments, terminal 110 has a data acquisition function, which is a function for multimedia data, such as video clips, images, etc.; for example, terminal 110 can acquire multimedia data through a camera; terminal 110 can also generate multimedia data through image / video generation technology, etc.

[0056] In some embodiments, terminal 110 acquires first multimedia data and second multimedia data.

[0057] The first multimedia data and the second multimedia data have the same content but different resolutions, and the first resolution of the first multimedia data is higher than the second resolution of the second multimedia data.

[0058] As an illustration, after the terminal 110 obtains the first multimedia data based on the camera, it performs data compression processing on the first multimedia data to obtain second multimedia data with lower resolution, etc.

[0059] In some embodiments, the terminal 110 analyzes the first multimedia data and the second multimedia data itself; or, the terminal 110 sends the first multimedia data to the server 120 through the communication network 130, or the terminal 110 sends the first multimedia data and the second multimedia data to the server 120 through the communication network 130, and the server 120 performs the analysis process. The following description takes the server 120 performing the analysis process as an example.

[0060] In some embodiments, server 120 acquires first multimedia data and second multimedia data.

[0061] Indicatively, server 120 acquires the first multimedia data and the second multimedia data sent by terminal 110; or, server 120 acquires the first multimedia data sent by terminal and then compresses it to obtain the second multimedia data with a lower resolution, etc.

[0062] In some embodiments, the server 120 performs noise-adding processing on the first multimedia data to obtain first noisy data, and performs noise-adding processing on the second multimedia data to obtain second noisy data.

[0063] To illustrate, noise addition processing is a method of adding noise to multimedia data. For example, after adding the same noise to the first multimedia data and the second multimedia data, the first noisy data corresponding to the first multimedia data is obtained, and the second noisy data corresponding to the second multimedia data is obtained.

[0064] In some embodiments, the server 120 concatenates the first noisy feature representation of the first noisy data and the second noisy feature representation of the second noisy data in the channel dimension using the diffusion model 121 to be trained, to obtain a first concatenated feature representation, wherein the number of channels in the first concatenated feature representation is the sum of the number of channels in the first noisy feature representation and the second noisy feature representation.

[0065] Schematic, the diffusion model 121 to be trained is pre-deployed in server 120. The diffusion model 121 extracts a first noisy feature representation of the first noisy data and a second noisy feature representation of the second noisy data. Then, the diffusion model 121 concatenates the first and second noisy feature representations along the channel dimension to obtain a first concatenated feature representation. This first concatenated feature representation includes not only the high-resolution first multimedia data OP60-250116...

[0066] The deep information also includes deep information from low-resolution second multimedia data.

[0067] Among them, channel-dimensional stitching is a weighted fusion method with learnable weighting coefficients. During the training process of the diffusion model to be trained, different channels can flexibly learn corresponding channel weights. The channel weights determine the importance of each channel in the final output, and the channel weights are the corresponding weighting coefficients. The channel weights corresponding to each channel are adjusted accordingly as the diffusion model to be trained is trained, rather than being manually set. Therefore, by adaptively and dynamically adjusting the channel weights, the contradiction between preserving object attributes and improving sharpness caused by manually adjusting the rate of decrease in channel weights is effectively avoided.

[0068] Furthermore, the first multimedia data and the second multimedia data have the same content but different resolutions, so they can be regarded as a pair of training data used in the training process of the diffusion model to be trained; correspondingly, the first noisy feature representation and the second noisy feature representation can also be regarded as a pair of training feature representations used in the training process of the diffusion model to be trained.

[0069] Therefore, during the training process of the diffusion model to be trained, the performance of multimedia data with the same content at different resolutions will be learned based on the first splicing feature representation, so as to improve the reverse noise removal process of the first diffusion model obtained after training.

[0070] In some embodiments, server 120 predicts first denoised noise based on first splicing feature representation, determines first loss function based on first denoised noise, and trains diffusion model to be trained using first loss function to obtain first diffusion model.

[0071] Indicatively, the first concatenated feature representation contains noisy information, therefore denoising processing is required. The denoising noise used in this process is the predicted data, such as through multiple denoising time steps, predicting the denoised noise corresponding to each time step. The first denoised noise predicted based on the first concatenated feature representation can be used to determine the first loss function. For example, the difference between the first denoised noise and the first added noise can be used as the first loss function. The first loss function is then used to train the diffusion model to obtain the first diffusion model. The first added noise is the true value of the noise added to the first multimedia data and the second multimedia data, and the first denoised noise is the predicted value of the noise added to the first multimedia data and the second multimedia data.

[0072] The first diffusion model, trained in this way, is used to adjust the resolution of the multimedia data.

[0073] In illustrative terms, after training the first diffusion model, for any low-resolution multimedia data (such as low-resolution images, videos, or even noisy images) that needs resolution upscaling, inputting it into the first diffusion model allows it to gradually denoise the low-resolution multimedia data to generate high-resolution multimedia data based on the relationships and differences learned during training. This achieves the goal of adjusting the resolution of the multimedia data. Typically, adjusting the resolution of multimedia data involves increasing the resolution; however, in some cases, the model can also be trained to learn how to decrease the resolution, which is not limited here.

[0074] It is worth noting that the aforementioned terminals include, but are not limited to, mobile terminals such as mobile phones, tablets, portable laptops, smart voice interaction devices, smart home appliances, and in-vehicle terminals, and can also be desktop computers, etc.; the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers.

[0075] Based on the above-mentioned terminology and application scenarios, the model training method provided in this application will be explained. Taking the application of this method to a server as an example, as shown in Figure 2, the method includes the following steps S210 to S240.

[0076] S210, acquire the first multimedia data and the second multimedia data.

[0077] The first multimedia data and the second multimedia data have the same content but different resolutions, with the first resolution of the first multimedia data being higher than the second resolution of the second multimedia data.

[0078] In some embodiments, multimedia data includes video clips, images, and other data. Since the first multimedia data and the second multimedia data are multimedia data with different resolutions but the same content, after obtaining the first multimedia data with a higher resolution, a resolution reduction method (e.g., downsampling) can be used to obtain the second multimedia data. This method can ensure the consistency of the content of the first multimedia data and the second multimedia data, while meeting the requirements of multi-resolution processing.

[0079] This is an illustrative example of using a resolution downscaling method to process high-resolution multimedia data and obtain low-resolution multimedia data.

[0080] For example, downsampling methods reduce the resolution by decreasing the number of pixels in multimedia data. For instance, converting a high-resolution image (e.g., 4000*3000 pixels) to a low-resolution image (e.g., 800*600 pixels) can be achieved by discarding pixels or averaging multiple pixels, such as through mean filtering, max pooling, Gaussian blur, and downsampling.

[0081] As an illustration, for multimedia data in the form of video clips, high-resolution multimedia data can also be processed by video down-resolution to obtain low-resolution multimedia data.

[0082] In some embodiments, multimedia data with the same content but different resolutions can be downloaded from the network, so that the multimedia data with higher resolution is used as the first multimedia data and the multimedia data with lower resolution is used as the second multimedia data, thereby performing the subsequent binding training process.

[0083] That is, since the first multimedia data and the second multimedia data are multimedia data with the same content but different resolutions, they can be regarded as a pair of training data used in the training process of the diffusion model to be trained; correspondingly, the first noisy feature representation and the second noisy feature representation can also be regarded as a pair of training feature representations used in the training process of the diffusion model to be trained.

[0084] S220, add first noise to the first multimedia data to obtain first noisy data, and add first noise to the second multimedia data to obtain second noisy data.

[0085] Indicatively, by adding the same first noise to the first multimedia data and the second multimedia data, we obtain the first noisy data corresponding to the first multimedia data, and the second noisy data corresponding to the second multimedia data.

[0086] In some embodiments, the first noise is such as Gaussian noise, random noise, pseudo noise, periodic noise, etc., and is not limited herein.

[0087] S230, by concatenating the first noisy feature representation of the first noisy data and the second noisy feature representation of the second noisy data in the channel dimension using the diffusion model to be trained, a first concatenated feature representation is obtained, wherein the number of channels in the first concatenated feature representation is the sum of the number of channels in the first noisy feature representation and the second noisy feature representation.

[0088] Schematic, a first noisy feature representation is extracted from the first noisy data and a second noisy feature representation is extracted from the second noisy data using a feature extraction network. Then, the first and second noisy feature representations are concatenated along the channel dimension to obtain a first concatenated feature representation. This first concatenated feature representation includes not only the deep information of the high-resolution first multimedia data but also the deep information of the low-resolution second multimedia data. The feature extraction network can be, for example, a convolutional neural network or a variational autoencoder, etc., and is not limited here.

[0089] Channel-dimensional stitching is a weighted fusion method with learnable weighting coefficients. Different channels can flexibly learn their corresponding channel weights based on the loss function used during the training of the diffusion model. These channel weights determine the importance of each channel in the final output, and are thus the corresponding weighting coefficients. During channel stitching, the channel weights are adjusted accordingly as the diffusion model trains, rather than being manually set. Therefore, by adaptively and dynamically adjusting the channel weights, the trade-off between preserving object attributes and improving sharpness caused by manually adjusting the rate of decrease in channel weights is effectively avoided.

[0090] S240: Using the diffusion model to be trained, predict the first denoised noise based on the first concatenated feature representation, determine the first loss function based on the first denoised noise, and train the diffusion model to be trained using the first loss function to obtain the first diffusion model.

[0091] Indicatively, the first concatenated feature representation is a feature representation containing noisy information, therefore it needs to be denoised. The denoised noise used in the denoising process is the predicted data, such as by gradually implementing the denoising process through multiple denoising time steps, predicting the denoised noise corresponding to each time step; the first denoised noise predicted based on the first concatenated feature representation is used to determine the first loss function. For example, the first loss function is constructed based on the difference between the first denoised noise and the first added noise, and the diffusion model to be trained is trained using the first loss function to obtain the first diffusion model.

[0092] The first diffusion model, trained in this way, is used to adjust the resolution of the multimedia data.

[0093] In illustrative terms, after training the first diffusion model, for any low-resolution multimedia data (such as low-resolution images, videos, or even noisy images) that needs resolution upscaling, inputting it into the first diffusion model allows it to gradually denoise the low-resolution multimedia data to generate high-resolution multimedia data based on the relationships and differences learned during training. This achieves the goal of adjusting the resolution of the multimedia data. Typically, adjusting the resolution of multimedia data involves increasing the resolution; however, in some cases, the model can also be trained to learn how to decrease the resolution, which is not limited here.

[0094] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0095] In summary, by performing concatenation processing on the first and second noisy feature representations at the channel dimension, multi-scale information fusion can be achieved. While preserving the global structure and semantic information of the low-resolution second multimedia data, richer details and local texture information can be captured through high-resolution images. This enables a comprehensive consideration of information from different dimensions of the same multimedia data. It also helps to guide the generation process of high-resolution multimedia data with prior knowledge of the resolution of low-resolution multimedia data, reducing the complexity of the generation task and improving the training efficiency and robustness of the diffusion model to be trained. Furthermore, it helps the trained first diffusion model to achieve the purpose of generating high-resolution data even when given low-resolution or noisy data, thus expanding the application scenarios of the first diffusion model.

[0096] In some embodiments, the diffusion model includes a diffusion transformer (DiT) network layer. A velocity prediction method is used to allow the DiT network layer to predict the velocity, i.e., the denoising velocity during the denoising process, thereby predicting the denoised noise and implementing the model training process. Illustratively, as shown in Figure 3, S240 shown in Figure 2 can also be implemented as S310 to S330 as follows.

[0097] S310, performs self-attention analysis on the first spliced ​​feature representation through the diffusion model to be trained, and predicts the first denoised noise.

[0098] Indicatively, the first concatenated feature representation is the feature representation obtained by concatenating the first noisy feature representation and the second noisy feature representation. Therefore, the first concatenated feature representation includes noise information and needs to be denoised.

[0099] In the denoising process of the first spliced ​​feature representation, it is necessary to consider the denoising noise required when predicting the denoising process based on the first spliced ​​feature representation; for example, self-attention analysis is performed on the first spliced ​​feature representation through the diffusion model to be trained to predict the first denoising noise.

[0100] Illustratively, self-attention analysis helps the diffusion model under training better understand the interdependencies between different sub-feature representations by calculating the relationships between multiple sub-feature representations in the first concatenated feature representation. During the backdiffusion process of the diffusion model under training, the goal is to predict the noise components from the first concatenated feature representation and then remove this noise, i.e., predict the first denoised noise.

[0101] In some embodiments, the self-attention mechanism is implemented through a diffusion converter in the diffusion model to be trained, thereby enabling the diffusion model to predict, based on the first concatenated feature representation of the input, the sub-feature representations that belong to noise and the sub-feature representations that do not belong to noise; and to predict the first denoising noise case in reverse through the sub-feature representations that belong to noise.

[0102] In some embodiments, the diffusion model to be trained is a pre-trained model, so the diffusion model to be trained can predict the denoised noise to be removed given a first concatenated feature representation. Illustratively, the predicted first denoised noise is represented as a noise feature representation, which represents the noise portion to be removed in the current step, such as the denoised noise feature representation being the content of a sub-feature representation belonging to noise in the first concatenated feature representation.

[0103] In some embodiments, denoising is performed at at least one time step based on the first spliced ​​feature representation to achieve a progressive denoising process based on the first spliced ​​feature representation.

[0104] A time step, also known as time embedding (represented as a feature representation), refers to the step-by-step processing of the model in forward diffusion (adding noise) and backward diffusion (removing noise) as a diffusion model to be trained. Each time step represents a processing stage.

[0105] In some embodiments, the time step in the noise-adding process is referred to as the noise-adding time step. The noise-adding process is a process of gradually converting the original multimedia data (such as images, video clips, etc.) into noisy data (such as pure noise). The noise-adding time step is at least one discrete time stage in the noise-adding process. At each noise-adding time step, noise (such as Gaussian noise) is added to the current multimedia data (which may be the original multimedia data or the multimedia data after noise addition based on the previous noise-adding time step) according to a predefined noise addition rule.

[0106] To illustrate, as the noise-adding time step increases, the noise in the multimedia data increases, while the original information in the multimedia data gradually decreases until it eventually becomes completely random noise at a certain time step.

[0107] In some embodiments, a time step in the denoising process is referred to as a denoising time step. The denoising process is the process of gradually restoring multimedia data (such as a clear image or a clear video clip) from noisy data (such as pure noise or multimedia data including noise). A denoising time step is at least one discrete time stage in the denoising process. At each denoising time step, the noise to be removed is predicted, and noise is reduced based on the current multimedia data (which may be pure noise data or multimedia data denoised based on the previous denoising time step). Typically, multiple denoising time steps are set, with larger denoising (removing more noise) performed at earlier denoising time steps and smaller denoising (removing less noise) at later time steps.

[0108] In other words, a time step in a diffusion model refers to a stage in data processing. The noise-adding time step involves adding noise, while the noise-removing time step involves removing noise. The diffusion model being trained optimizes the denoising process at each denoising time step (e.g., more accurately predicting the noise to be removed at each denoising time step) to better denoise and generate clear data (images or video clips). Typically, there is a one-to-one correspondence between denoising and noise-adding time steps.

[0109] To illustrate, in the process of reverse prediction, the denoising process performed at each time step is achieved by using self-attention analysis to predict the denoising noise at the predicted time step, thus realizing a gradual denoising process.

[0110] In some embodiments, the denoising speed is obtained by analyzing the denoising feature representation and denoising time step through the velocity prediction network layer in the diffusion model to be trained. The denoising speed is the processing rate of the denoising process predicted by the diffusion model to be trained.

[0111] The first splicing feature indicates that denoising is performed through at least one denoising time step.

[0112] In a schematic representation, denoising speed refers to the processing rate of data from a noisy state to a restored original signal state in a diffusion model. The diffusion model to be trained reconstructs the original data by progressively introducing noise and then denoising it through multiple reverse processes. In this process, each time step (denoising time step) represents the transition from the current noisy state to the target clear state. By analyzing the velocity prediction network layer of the diffusion model, the denoising speed can be derived by calculating the changes in the denoising feature representation between different time steps. In a schematic representation, the denoising speed is predicted using the Velocity Prediction (V-Prediction) method.

[0113] In some embodiments, the first splicing feature is predicted based on the denoising speed to represent the corresponding first denoised noise.

[0114] In a schematic way, predicting the denoised noise based on the first concatenated feature representation after obtaining the denoising speed actually involves inferring the nature and intensity of the noise by calculating the denoising rate at different time steps in the diffusion model. During the diffusion process, the model continuously adjusts the denoising operation at each time step by learning the denoising feature representation, thereby gradually eliminating noise and restoring the original signal at each time step.

[0115] In some embodiments, based on the denoising speed and the first concatenation feature representation, the denoised noise corresponding to at least one denoising time step is predicted. Each denoising time step corresponds to one denoised noise.

[0116] To illustrate, the denoising process is performed based on the denoising speed, which represents the noise removal rate per unit denoising time step, based on the first spliced ​​feature representation. Therefore, by starting the denoising process with the first spliced ​​feature representation, the denoised noise at each denoising time step can be obtained.

[0117] By calculating the denoising speed, the model can predict the noise components contained in the current denoised feature representation at a specific denoising stage. For example, a slower denoising speed may mean that the noise at that time step is relatively strong, and the feature representation contains more irrelevant information; while a faster denoising speed indicates that there are fewer noise components, and the model can recover the target signal more quickly.

[0118] S320, acquire the first noise added to the first multimedia data and the second multimedia data.

[0119] In some embodiments, the noise used to add noise to the first multimedia data and the noise used to add noise to the second multimedia data are generally the same; the first noise is noise data added on the basis of the first multimedia data, and also noise data added on the basis of the second multimedia data.

[0120] In some embodiments, the first noise is Gaussian noise, random noise, etc., which is not limited here.

[0121] S330, determine the first loss function based on the first denoised noise, and use the first loss function to train the diffusion model to be trained to obtain the first diffusion model.

[0122] Indicatively, the difference between the predicted first denoised noise and the first added noise used in the multimedia data denoising process is compared to obtain noise difference data (noise loss value); the diffusion model to be trained is trained using the noise difference data to obtain the first diffusion model.

[0123] In some embodiments, sample guidance information is acquired during the training process of the diffusion model to be trained. The sample guidance information is used to guide the model to learn and adjust the resolution of the second multimedia data based on the first multimedia data.

[0124] For example, sample guidance information, such as sample text, is text content that guides the model to learn and adjust the resolution of the second multimedia data based on the first multimedia data. If the sample text represents the content of the first multimedia data in more detail, and the second multimedia data is low-resolution data with less detail, the sample guidance information can be used as a control condition to better control the resolution improvement purpose.

[0125] In some embodiments, a first loss function is constructed based on the difference between the first denoised noise and the first added noise. The diffusion model to be trained is then used to train the model. The goal is to improve the model's generation performance by optimizing the different characteristics of the denoised and added noise, thereby obtaining the first diffusion model. During the training of the diffusion model, the model is typically trained to gradually recover the original signal by progressively introducing noise and training the network to denoise it. During training, the model learns how to handle different noise intensities and distribution characteristics.

[0126] In this method, by analyzing the differences between the first added noise and the first denoised noise, the characteristic differences of the noise can be captured, and these differences can be used to improve the model's learning ability. Through this training strategy, the diffusion model to be trained can effectively identify the pattern of the first added noise and adjust the denoising operation according to the reverse process of the first denoised noise.

[0127] Once the diffusion model to be trained has successfully learned this process, this difference can be further passed to the first diffusion model trained, enabling it to more accurately remove noise and reconstruct signals during the inference phase, thereby improving the quality of generated images, reducing noise residue, and exhibiting better operational stability and result accuracy in generation tasks.

[0128] In some embodiments, for a first denoising time step, a first denoising time step corresponding to the first denoising time step is obtained; denoised noise corresponding to the first denoising time step is obtained; and a diffusion model to be trained is trained based on the difference between the denoised noise corresponding to the first denoising time step and the denoised noise corresponding to the first denoising time step to obtain a first diffusion model.

[0129] Indicatively, the first denoising time step corresponds to the first denoising time step. Although the denoising process can be achieved through one or multiple denoising steps with multiple preset denoising time steps, the denoising and denoising processes usually correspond to the same time step. Therefore, the corresponding first denoising time step can be determined based on the first denoising time step. For example, the time step corresponding to the process of obtaining the first denoised image from the original image through denoising is the first denoising time step. Then, in the denoising process, the time step corresponding to the process of obtaining the last denoised image from the penultimate denoised image through denoising is the first denoising time step, and the two correspond one-to-one.

[0130] Each noise-adding time step corresponds to a noise-adding noise, representing the noise added at each noise-adding time step. The diffusion model to be trained is trained using the difference between the denoised noise corresponding to each denoising time step and the noise-adding noise corresponding to the corresponding noise-adding time step, thus obtaining the first diffusion model.

[0131] Indicatively, the mean squared error (MSE) between the denoised noise at each denoising time step and the added noise at the corresponding denoising time step is obtained as a noise loss value representing the difference. The diffusion model to be trained is then trained using the noise loss value to obtain the first diffusion model.

[0132] In some embodiments, denoising processing is performed on the first spliced ​​feature representation based on sample guidance information to obtain a denoised feature representation; a first loss function is constructed based on the difference between the denoised feature representation and the first data feature representation corresponding to the first multimedia data; and the first loss function is used to train the diffusion model to be trained to obtain the first diffusion model.

[0133] In a schematic manner, sample guidance information is used as a control condition to achieve denoising of the first spliced ​​feature representation, thereby obtaining a denoised feature representation after denoising is completed. The denoised feature representation is the denoising result predicted by the diffusion model to be trained during the training process. The first data feature representation corresponding to the high-resolution first multimedia data is extracted. The denoised feature representation is compared with the first data feature representation corresponding to the high-resolution first multimedia data, thereby training the diffusion model to be trained with the feature representation difference (feature representation loss value) and obtaining the first diffusion model.

[0134] In some embodiments, decoding processing is performed on the denoised feature representation to obtain decoded data, a first loss function is constructed based on the difference between the decoded data and the first multimedia data, and a first diffusion model is obtained by using the first loss function to train the diffusion model.

[0135] In a schematic manner, the decoding process is implemented through a decoder. The denoising feature representation is input into the decoder to obtain the decoded data. The data difference (data loss value) between the decoded data and the first multimedia data is obtained. The diffusion model to be trained is trained with the data difference to obtain the first diffusion model.

[0136] Figure 4 shows an architecture diagram for training a diffusion model provided in some embodiments. The following explanation uses video clip 410 as the first multimedia data and text 420 as the sample guidance information as an example.

[0137] The video segment 410 is input into the encoder 430 to extract the first data feature representation; in addition, the video segment 410 is subjected to resolution compression processing to obtain a low-resolution video segment (not shown in the figure); the low-resolution video segment is input into the encoder 430 to extract the second data feature representation; noise is added to the first data feature representation to obtain a first noisy feature representation, and the same noise is added to the second data feature representation to obtain a second noisy feature representation.

[0138] Alternatively, noise is added to video segment 410 to obtain first noisy data, and video segment 410 is processed by resolution compression to obtain low-resolution video segment (not shown in the figure); the same noise is added to the low-resolution video segment to obtain second noisy data; a first noisy feature representation of the first noisy data is extracted, and a second noisy feature representation of the second noisy data is extracted.

[0139] In addition, a large language model was used to extract text feature representations for text 420.

[0140] The first and second noisy feature representations are concatenated to obtain a concatenated feature representation. Based on this concatenated feature representation, denoised noise is predicted under the guidance of text 420. The diffusion model to be trained is then trained based on the difference between the denoised noise and the noisy noise in the noisy processing. For example, the concatenated feature representation and the text feature representation are fused to obtain a fused feature representation. The fused feature representation is then used to predict denoised noise and train the diffusion model to be trained.

[0141] In some embodiments, the compression encoding process of video segment 410 is completed by a causal 3D autovariable encoder; the input text 420 is semantically encoded using a large language model, and the generated text feature representation is used as a conditional control signal; the model is input with Gaussian noise and conditional vectors, and learns to generate the corresponding latent spatial representation through a diffusion process, and finally the latent spatial representation is reconstructed into an image or video sequence by a 3D autovariable decoder.

[0142] Indicatively, the above architecture achieves a recursive representation of dynamic relationships between video frames by modeling the temporal dimension under causal convolution constraints (a characteristic possessed by the autovariable encoder and autovariable decoder), while simultaneously coupling a multimodal alignment mechanism that integrates textual semantics and visual features.

[0143] It is worth noting that the above process of training the diffusion model to be trained through noise difference, feature representation difference, and data difference is only an illustrative example. The above loss values ​​can be used selectively or at least two can be used in combination to train the diffusion model to be trained. This application does not limit this.

[0144] In the embodiments of this application, the diffusion model learns how to denoise by predicting the denoising speed through a speed prediction method during training. This helps the network layer to more accurately control the denoising speed during the denoising process and improve the denoising effect. In addition, the denoising speed can also help the network layer better capture the impact of different noise levels on the generated image, avoiding the problems of over-denoising or under-denoising. This helps to restore high-resolution multimedia data during training, improve the quality and diversity of generated images, and thus help to improve the performance of generated multimedia data in inference applications.

[0145] In some embodiments, during the application of the first diffusion model, noisy multimedia data and data guidance information are acquired, and a denoising process is performed on the noisy multimedia data using the first diffusion model to obtain denoised multimedia data. Indicatively, as shown in Figure 5, S240 shown in Figure 2 can also be implemented as S510 to S540 as follows.

[0146] S510, acquire the first noisy multimedia data and data guidance information.

[0147] The first noisy multimedia data is multimedia data containing noise, such as random noise data or low-resolution multimedia data; the first noisy multimedia data includes noisy video clips, noisy images, etc.

[0148] The data guidance information is used to guide the denoising process on the first noisy multimedia data. This data guidance information includes, for example, text related to the first noisy multimedia data. The data guidance information guides the denoising process on the first noisy multimedia data to obtain higher-resolution multimedia data.

[0149] When the first noisy multimedia data is a noisy video segment, the data guidance information is used to guide the removal of noise from each noisy video frame in the noisy video segment; when the first noisy multimedia data is a noisy image, the data guidance information is used to guide the removal of noise from the noisy image.

[0150] S520, the first noisy data feature representation corresponding to the first noisy multimedia data is extracted through the first diffusion model.

[0151] Schematic illustration: The first diffusion model is the diffusion model obtained after training the aforementioned diffusion model to be trained. The first diffusion model includes a feature extraction network; the feature extraction network is used to perform feature extraction on the first noisy multimedia data to obtain the first noisy data feature representation corresponding to the first noisy multimedia data. The feature extraction network may be such as a convolutional neural network, a variational autoencoder, etc., and is not limited here.

[0152] S530, under the guidance of the data guidance information, denoising processing is performed on the first noisy data feature representation to obtain the first denoised data feature representation.

[0153] As an illustration, since the data guidance information is used as a control signal (control condition) in the first diffusion model, denoising processing is performed on the first noisy data feature representation based on the data guidance information to obtain the first denoised data feature representation.

[0154] In some embodiments, the first diffusion model trained has accurately learned the noise distribution and noise prediction rules in the denoising process during the training phase. Therefore, after inputting the first noisy data feature representation into the model, the sub-feature representation corresponding to the noise can be identified and deleted, thereby obtaining the first denoised data feature representation.

[0155] The denoising process is constrained by data-guided information, so that the first denoised data feature representation conforms to the information represented by the data-guided information. The denoising process is performed with the goal of making the first denoised data feature representation similar to the information feature representation corresponding to the data-guided information in the vector space (e.g., if the data-guided information is text, it represents semantic similarity, etc.), thereby obtaining the first denoised data feature representation.

[0156] S540, decode the feature representation of the first denoised data to obtain the first denoised multimedia data. The resolution of the first denoised multimedia data is higher than the resolution of the first noisy multimedia data.

[0157] In illustrative terms, after obtaining the first denoised data feature representation, the first denoised data feature representation is decoded by a decoding network layer (decoder) to restore the denoised multimedia data; the decoding network layer may be a variational autodecoder, a deconvolutional layer, etc., and is not limited here.

[0158] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0159] In summary, a comprehensive consideration of multi-dimensional information on multimedia data with the same content at different resolutions helps guide the generation process of high-resolution multimedia data with prior knowledge of the resolution of low-resolution multimedia data, reducing the complexity of the generation task, improving the training efficiency and robustness of the diffusion model to be trained, and also helps the first diffusion model to achieve the purpose of generating high-resolution data even when low-resolution data or even noisy data is obtained, thus expanding the application scenarios of the first diffusion model.

[0160] In this application embodiment, the application method of the trained first diffusion model is introduced, which uses data-guided information to guide the denoising process, which helps to improve denoising accuracy, avoid over-denoising, and enhance the realism and detail of multimedia data generation.

[0161] In some embodiments, the diffusion model to be trained can generate video clips based on given multimedia data such as images or animations; the trained first diffusion model can improve the resolution of images or video clips. Both the diffusion model to be trained and the trained first diffusion model can use the aforementioned data guidance information. The data guidance information can be text, or, if the input multimedia data (images or video clips) includes biological objects (such as human characters), the identified skeletal diagram can be used as the data guidance information. Illustratively, as shown in Figure 6, the embodiment illustrated in Figure 2 may further include the following steps S610 to S630.

[0162] S610: Acquire multimedia data to be analyzed.

[0163] Indicatively, the input multimedia data is a video clip or image that needs to be analyzed by at least one of the diffusion models to be trained and the first diffusion model obtained from the training.

[0164] In some embodiments, the multimedia data to be analyzed is used to generate video clips from images through a diffusion model to be trained; the multimedia data to be analyzed is used to generate video clips from animated images through a diffusion model to be trained; or, the multimedia data to be analyzed is used to generate videos from short videos through a diffusion model to be trained, wherein the number of video frames in the short video is less than the number of video frames in the video clip.

[0165] In some embodiments, the multimedia data to be analyzed is subjected to a first diffusion model for resolution enhancement of an image or video segment.

[0166] In a schematic manner, multimedia data to be analyzed is obtained through methods such as camera capture or network download, and the multimedia data to be analyzed is input into at least one of the diffusion models to be trained and the first diffusion model.

[0167] S620, when the multimedia data to be analyzed includes biological objects, extract the skeletal diagrams corresponding to the biological objects.

[0168] Indicative, the multimedia data to be analyzed includes visual elements, with biological objects being a form of visual element, typically dynamic visual elements such as people and animals.

[0169] In some embodiments, the multimedia data to be analyzed is a video segment, which includes multiple video frames. If any video frame includes a biological object such as a person, the video segment is considered as multimedia data including a biological object. Alternatively, the multimedia data to be analyzed is an image. If the image includes a biological object such as a person, the image is considered as multimedia data including a biological object.

[0170] For example, if the multimedia data to be analyzed includes biological objects, then the corresponding skeletal diagram of the organism is extracted.

[0171] In illustrative terms, the skeletal diagram corresponding to a biological object is used to characterize the skeletal structure of the biological object in the image. The skeletal diagram can be identified based on an image that includes the biological object.

[0172] In some embodiments, the method for extracting skeletal maps from human images mainly relies on human pose estimation techniques, with the goal of extracting the joint positions and skeletal structure of a person from the image.

[0173] This example illustrates the use of human detection algorithms to identify human regions in an image (e.g., outputting a bounding box to pinpoint the basic position of the person). Human detection algorithms include at least one of the following: You Only Look Once (YOLO) and Faster Region-based Convolutional Neural Network (Faster R-CNN). Then, a human pose estimation algorithm is used to detect keypoints within the human region. This algorithm can locate the joints of the person (e.g., shoulders, elbows, knees) and output the coordinates of each joint. Human pose estimation algorithms include OpenPose (Open Source Library for Real-Time Multi-Person Keypoint Detection) and High-Resolution Networks. At least one of the following algorithms is used: Network (HRNet), etc. In addition, a skeleton map is constructed based on the detected key points according to the connection relationship of the human skeleton (such as from the neck to the shoulder, from the shoulder to the elbow, etc.). These key points form a skeletal structure by connecting lines, representing the body posture of the person. Furthermore, unreliable key points can be removed and the skeleton structure can be optimized through post-processing to ensure the accuracy and coherence of the skeleton map. The key points in the skeleton map can also be smoothed and denoised to improve the visualization effect.

[0174] S630, in the process of analyzing the multimedia data to be analyzed using at least one of the diffusion models to be trained and the first diffusion model, multimedia generated data corresponding to the multimedia data to be analyzed is generated using the skeleton diagram as data guiding information.

[0175] Indicatively, the diffusion model to be trained and the first diffusion model serve as diffusion models with denoising capabilities. If the diffusion model to be trained or the first diffusion model is used to analyze the multimedia data to be analyzed, control conditions can usually be set during the denoising process to better guide the denoising process.

[0176] In some embodiments, if the multimedia data to be analyzed includes biological objects, in order to ensure the consistency of biological objects in the generated multimedia data, the skeletal diagrams corresponding to the biological objects are used as control conditions in the denoising process, thereby guiding the generation of multimedia data.

[0177] Based on the skeleton diagram as a control condition, denoising processing is performed on the noise-adding feature representation corresponding to the multimedia data to be analyzed, so as to obtain the multimedia generated data; at this time, the skeleton diagram is regarded as a form of data guidance information.

[0178] In some embodiments, the data guidance information also includes text, such as using the text and skeleton diagram together as control conditions to perform denoising processing on the noisy feature representation to be analyzed, so as to obtain multimedia generated data.

[0179] In some embodiments, the text is used as a separate control condition to perform denoising processing on the noisy feature representation to be analyzed, in order to obtain multimedia generated data.

[0180] Indicatively, the data guidance information includes text, acquiring text and multimedia data to be analyzed; during the analysis of the multimedia data to be analyzed using at least one of the diffusion models to be trained and the first diffusion model, multimedia generated data corresponding to the multimedia data to be analyzed is generated using text as data guidance information.

[0181] In other words, text and skeletal diagrams are two forms of data guidance information. Text can be used alone as data guidance information, skeletal diagrams can be used alone as data guidance information, or both text and skeletal diagrams can be used together as data guidance information. There is no limitation here.

[0182] In some embodiments, if the multimedia data to be analyzed is analyzed using at least one of the diffusion model to be trained and the first diffusion model, an additional visual coding network layer is added to the diffusion model that needs to perform the analysis process. The visual coding network layer is used to encode the input multimedia data to be analyzed, thereby controlling the generation process through the original image.

[0183] In some embodiments, the image feature representation corresponding to the multimedia data to be analyzed is extracted; the noise-added feature representation to be analyzed is obtained after adding noise to the multimedia data to be analyzed; the image feature representation and the noise-added feature representation to be analyzed are spliced ​​together to obtain the image splicing feature representation; the image splicing feature representation is denoised using the skeleton map as data guiding information to generate multimedia generated data corresponding to the multimedia data to be analyzed.

[0184] In a schematic manner, image feature representation, noise, and multimedia data to be analyzed are input together into a noisy network layer. The noisy network layer adds noise to the multimedia data to be analyzed and extracts the noisy feature representation to be analyzed from the noisy multimedia data to be analyzed. The image feature representation is added as initialization information to at least one of the beginning and end of the noisy feature representation to be analyzed to obtain the image stitching feature representation.

[0185] In some embodiments, text is used as a control condition to perform denoising processing on the image stitching feature representation to obtain multimedia generation data; or, a skeleton map is used as a control condition to perform denoising processing on the image stitching feature representation to obtain multimedia generation data; or, both text and skeleton map are used as control conditions to perform denoising processing on the image stitching feature representation to obtain multimedia generation data, which is not limited here.

[0186] In some embodiments, taking text as an example, the text guides the generation of a character rotation video with a preset posture based on the multimedia data to be analyzed (such as a person image); then the above method can be used to perform denoising processing on the text or text and skeletal diagram as control conditions for the noisy feature representation or image stitching feature representation to be analyzed, so as to obtain multimedia generated data.

[0187] Figure 7 shows a schematic diagram of the architecture for performing processing on the multimedia data to be analyzed.

[0188] Schematic, the multimedia data to be analyzed is an image 710, which is used to generate a video of a character rotating in a standard pose. In some embodiments, the above-described diffusion model to be trained is used as the backbone network. The diffusion model to be trained includes pre-trained weights learned through pre-training and also obtains prior knowledge of a large amount of high-quality multimedia data (such as video data) based on the pre-training process.

[0189] An additional visual coding network layer 720 is added on top of the skeleton network to encode the input image 710 and inject the encoded information into the network. Based on the characteristics of the rotated video, the encoded information is used as initialization information to apply to the beginning and end parts of the noisy feature representation to be analyzed after adding noise. The skeleton map 730 obtained based on the image 710 is introduced as a control condition to control the proportion of character generation in the character's standard pose rotation video.

[0190] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0191] In this embodiment, a skeletal diagram is further described as being used as data guidance information to obtain the content of the multimedia generated data. When the multimedia data to be analyzed includes biological objects, by considering the skeletal diagram to constrain the denoising process of the diffusion model to be trained or the first diffusion model, the problem of poor accuracy in generating biological objects can be greatly avoided, ensuring the expressiveness of biological objects in the multimedia data. In addition, an additional visual coding network layer is introduced to obtain image stitching feature representation to perform the denoising process, which can constrain the accuracy of the multimedia generated data by the multimedia data to be analyzed itself, thereby improving the generation accuracy.

[0192] In some embodiments, taking the above model training method applied to a game scene as an example, game multimedia data includes game video data (such as game video clips) and game image data (such as game screenshots and game character images); or, the above model training method can also be applied to a special effects production scene, and special effects multimedia data includes special effects video data (such as special effects video clips) and special effects image data (such as special effects screenshots and special effects character images).

[0193] In some embodiments, the above-described model training method can be used to process multimedia data in at least one of the following ways: adjustment and variation.

[0194] (a) Perform super-resolution processing on multimedia data.

[0195] Super Resolution (SR) is an image processing technique designed to recover higher resolution images from low-resolution images using algorithms. Therefore, super resolution can transform blurry or low-resolution images into clearer and more detailed images. It has wide applications in image processing, computer vision, and machine learning, especially in areas such as game image processing, video enhancement, medical imaging, and improving the clarity of satellite image processing.

[0196] In some embodiments, super-resolution technology is implemented through deep learning technology, which trains a model using the correspondence between a large number of low-resolution images and high-resolution images, and uses the model to recover high-resolution images from low-resolution images.

[0197] In some embodiments, taking the processing of game video data as an example, the super-resolution technology based on game video data adopts a generative diffusion model as the backbone network. Generative diffusion models include: generative diffusion model based on 1.3 billion parameters of mixed video (13BHunyuanVideo), denoising diffusion probabilistic models (DDPMs), score-based generative models (SBGM), etc., which are not limited here.

[0198] As an illustration, when using at least one of the above diffusion models for multimedia data analysis, the pre-training weights learned by the generative diffusion model during pre-training are inherited. During pre-training, the generative diffusion model can learn a large amount of prior knowledge from high-quality video data. For example, using a mixed-parameter video generative diffusion model based on 1.3 billion parameters as the base model (skeleton network) for multimedia data can better avoid problems such as video flickering and abrupt changes in detail. It can also better complete the filling of details and improve the clarity of low-resolution videos while maintaining the properties of the video elements, resulting in high-resolution videos.

[0199] In some embodiments, a latent-channel-concat scheme is used to perform analysis on multimedia data.

[0200] To illustrate, taking the diffusion model to be trained as a pre-trained base model as an example, there are noise addition and prediction rate processes during the training process of the diffusion model to be trained.

[0201] (1) Noise addition process.

[0202] The noise addition process includes at least one time step, which can be called a noise addition time step. At each noise addition time step, the noise addition process continues based on the previous multimedia data / noisy multimedia data. For example, if there is one noise addition time step, noise is added to the original multimedia data to obtain noisy multimedia data; or, if there are two noise addition time steps, noise is added to the original multimedia data to obtain noisy multimedia data 1, and noise is added to noisy multimedia data 1 to obtain noisy multimedia data 2. The noise added at different noise addition time steps can be the same or different, which is not limited here.

[0203] Taking multimedia data as an image as an example, at least one noise-adding time step is set, and noise is added to the image once at each noise-adding time step; taking multimedia data as a video segment as an example, the video segment includes multiple video frames, each video frame is regarded as an image, at least one noise-adding time step is set, and noise is added to multiple video frames once at each noise-adding time step.

[0204] In some embodiments, for different noise-adding time steps, the latent features of high-resolution multimedia data are added with the noise of the corresponding noise-adding time step, and then concatenated with the latent features of the noise-adding low-resolution multimedia data in the channel dimension.

[0205] During the training process of the diffusion model to be trained, high-resolution multimedia data refers to the original multimedia data with higher resolution (i.e., the first multimedia data mentioned above, corresponding to the first resolution), and low-resolution multimedia data refers to multimedia data with lower resolution (i.e., the second multimedia data mentioned above, corresponding to the second resolution, which is lower than the first resolution).

[0206] In some embodiments, a resolution reduction operation is performed on high-resolution multimedia data to obtain low-resolution multimedia data.

[0207] This is an illustrative example of using a resolution downscaling method to process high-resolution multimedia data and obtain low-resolution multimedia data.

[0208] For example, downsampling methods reduce the resolution by decreasing the number of pixels in multimedia data. For instance, converting a high-resolution image (e.g., 4000*3000 pixels) to a low-resolution image (e.g., 800*600 pixels) can be achieved by discarding pixels or averaging multiple pixels, such as through mean filtering, max pooling, Gaussian blur, and downsampling.

[0209] As an illustration, for multimedia data in the form of video clips, high-resolution multimedia data can also be processed by video down-resolution to obtain low-resolution multimedia data.

[0210] For example, for each video frame in a video clip, the resolution is reduced frame by frame to decrease the frame size and obtain lower-resolution multimedia data. Furthermore, since a video clip consists of multiple video frames, the resolution of the multimedia data can also be reduced by decreasing the frame rate, such as from 30 frames per second to 15 frames per second. Additionally, video compression techniques can be used to compress high-resolution multimedia data to obtain lower-resolution multimedia data.

[0211] Since low-resolution multimedia data is obtained by reducing the resolution of high-resolution multimedia data, the content of high-resolution multimedia data and low-resolution multimedia data is the same, but the resolution is different.

[0212] When the multimedia data is video clips, there is a one-to-one correspondence between multiple high-resolution video frames in high-resolution multimedia data and multiple low-resolution video frames in low-resolution multimedia data; when the multimedia data is images, there is a correspondence between high-resolution images and low-resolution images.

[0213] In some embodiments, at least one noise-adding time step is set during the noise-adding process. For any noise-adding time step, it corresponds to the first noise-adding feature representation extracted for high-resolution multimedia data after this noise-adding, and also corresponds to the second noise-adding feature representation extracted for low-resolution multimedia data after this noise-adding. The first noise-adding feature representation and the second noise-adding feature representation under the same noise-adding time step are concatenated in the channel dimension to obtain a concatenated feature representation. Since there is at least one noise-adding time step, each noise-adding time step corresponds to a concatenated feature representation.

[0214] In some embodiments, although the noise added at different noise-adding time steps may be different, the noise added to high-resolution multimedia data and low-resolution multimedia data at the same noise-adding time step is generally set to be the same.

[0215] To illustrate, taking multimedia data as an example, for noise addition time step 1, after performing noise addition processing on the high-resolution image at noise addition time step 1, the first noise addition feature representation corresponding to the noise-added high-resolution image is extracted; similarly, after performing noise addition processing on the low-resolution image at noise addition time step 1, the second noise addition feature representation corresponding to the noise-added low-resolution image is extracted; at noise addition time step 1, the first noise addition feature representation and the second noise addition feature representation are concatenated in the channel dimension to obtain the concatenated feature representation corresponding to noise addition time step 1.

[0216] To illustrate, taking multimedia data as a video segment as an example, for the noise-adding time step 1, after performing noise-adding processing on the high-resolution video segment in noise-adding time step 1, the first noise-adding feature representation corresponding to the noise-adding high-resolution video segment is extracted (e.g., the first noise-adding feature representation is the feature representation extracted for the entire noise-adding high-resolution video segment; or, the first noise-adding feature representation is obtained by fusing the noise-adding high-resolution video frame feature representations corresponding to multiple noise-adding high-resolution video frames after performing feature extraction on each noise-adding high-resolution video frame in the noise-adding high-resolution video segment). Similarly, for the low-resolution video segment in noise-adding time step 1... After the low-resolution video segment is subjected to noise processing, the second noise-added feature representation corresponding to the noise-added low-resolution video segment is extracted (e.g., the second noise-added feature representation is the feature representation extracted for the entire noise-added low-resolution video segment; or, the second noise-added feature representation is obtained by fusing the noise-added low-resolution video frame feature representations corresponding to multiple noise-added low-resolution video frames after performing feature extraction on each noise-added low-resolution video frame in the noise-added low-resolution video segment); at noise-adding time step 1, the first noise-added feature representation and the second noise-added feature representation are concatenated in the channel dimension to obtain the concatenated feature representation corresponding to noise-adding time step 1.

[0217] (2) Predicted rate.

[0218] To illustrate, a diffusion model based on a converter architecture (Diffusion Transformer, DiT) is introduced, and a velocity field prediction method (V-Prediction) is adopted. By modeling the linear combination of noise and data (velocity field v) during the noise addition process, the reverse generation of the denoising process is guided. The velocity field v is the predicted dynamic characteristic of the noise addition process, or simply velocity v.

[0219] In some embodiments, the core idea of ​​the diffusion model is to enable the model to gradually recover data from noise through a series of diffusion and denoising steps. Each diffusion time step adds noise to the multimedia data, and during denoising, the model progressively reverses the denoising steps to recover the original data. The denoising time steps typically correspond one-to-one with the diffusion time steps. Therefore, the task of V-Prediction is to predict the noise at each diffusion time step to guide the denoising process.

[0220] Indicatively, since velocity v is a linear combination of noise and multimedia data, reflecting the evolution direction of the noise-adding path, velocity can be predicted during the training phase of the diffusion model to be trained using the noise-adding multimedia data and the noise-adding time step; the denoising process can be indirectly guided by velocity.

[0221] In some embodiments, for the aforementioned high-resolution multimedia data and low-resolution multimedia data, based on the process of adding noise and concatenating channel dimensions, at least one noise-adding time step corresponds to a concatenated feature representation.

[0222] Schematic, at noisy time step 1, the first noisy feature of high-resolution multimedia data after noisy addition is represented as a[H,W,C1], and the second noisy feature of low-resolution multimedia data after noisy addition is represented as b[H,W,C2]. The concatenated feature representation 1 at noisy time step 1, obtained by concatenating along the channel dimension, is [H,W,C1+C2], where H and W are the height and width, respectively, and C1 and C2 represent the number of channels. In other words, concatenating along the channel dimension essentially uses the union of the channel data (e.g., r1, g1, b1) in the first noisy feature representation and the channel data (e.g., r2, g2, b2) in the second noisy feature representation (i.e., r1, g1, b1, r2, g2, b2) as the channel data in the concatenated feature representation 1, so that the number of channels in the concatenated feature representation 1 is the sum of the number of channels in the first two representations. Since the content of high-resolution multimedia data and low-resolution multimedia data is the same, C1 and C2 are the same. Therefore, the splicing feature representation 1 at the noisy time step 1 can also be represented as [H,W,2C].

[0223] Schematic, the stitched feature representation is input into a diffusion transformer (e.g., DiT) network layer to predict speed, such as the rate of change between different stitched feature representations at different noise addition time steps. This speed can then be used to predict the added noise and the original input multimedia data (e.g., predicting at least one of the stitched feature representation, first multimedia data, and second multimedia data), guiding the gradual removal of noise during denoising until a clear image is recovered. The predicted speed *v* in this process acts as a denoising progress bar (i.e., denoising speed), helping the network determine how much noise should be removed at each denoising time step.

[0224] In some embodiments, since the number of channels in the concatenated feature representation obtained after concatenating the channel dimensions is large (e.g., from C to 2C), the convolutional layer structure used to control the number of channels in the diffusion model to be trained will be modified accordingly during the model training process.

[0225] In some embodiments, the number of output channels of a convolutional layer (i.e., the number of convolutional kernels) typically determines the expressive power of the network; controlling the number of channels means adjusting or modifying the number of output channels during the design of the convolutional layer.

[0226] In some embodiments, the convolutional layer structure is adjusted by using half of the weights inherited from the pre-trained diffusion model and the other half of the weights initialized with zero.

[0227] In this approach, half of the weights are inherited from the pre-trained diffusion model. This means that the weights of some convolutional kernels (e.g., half of the convolutional kernels) retain the weight values ​​obtained during pre-training, thus preserving existing knowledge in the network, accelerating convergence, and improving model performance. The other half is initialized to zero, meaning that the weights of the other half of the convolutional kernels (e.g., the other half of the convolutional kernels) are initialized to zero. This allows the network to learn new features during training without being affected by the original weights. This method enables the model to utilize previously learned knowledge during training, thereby accelerating the training process and giving the model more freedom to explore new features without being limited by the original weights.

[0228] In some embodiments, the channel portion corresponding to high-resolution multimedia data inherits the weights of the pre-trained diffusion model, while the channel portion corresponding to low-resolution multimedia data is initialized with zero weights; or, the channel portion corresponding to low-resolution multimedia data inherits the weights of the pre-trained diffusion model, while the channel portion corresponding to high-resolution multimedia data is initialized with zero weights, without limitation here.

[0229] In some embodiments, the loss function configuration is consistent with that of the pre-trained diffusion model during the model training process. Efficient learning of large-scale video data is achieved through distributed training strategies such as sequence parallelism and parameter parallelism, combined with 16-bit floating point (FP16) mixed precision optimization.

[0230] In some embodiments, an additional block-based training inference method is proposed to further reduce inference time by reducing the number of inference steps and the window size.

[0231] In some embodiments, the input noisy multimedia data is adjusted using a block-based inference strategy through the first diffusion model to obtain denoised multimedia data, wherein the resolution of the denoised multimedia data is higher than that of the noisy multimedia data. Specifically, adjusting the input noisy multimedia data using a block-based inference strategy through the first diffusion model to obtain denoised multimedia data includes: dividing the noisy multimedia data into multiple noisy blocks with overlapping regions according to a preset overlap ratio; performing independent resolution adjustment on each noisy block using the first diffusion model to obtain a corresponding denoised block; and performing an arithmetic average operation on the pixel values ​​in the overlapping regions between adjacent denoised blocks to fuse the denoising results of adjacent denoised blocks.

[0232] In some embodiments, a certain overlap ratio is preset, and the overlapping regions are averaged through a block operation with a certain overlap ratio.

[0233] To illustrate, when the multimedia data is a video clip, the block processing targets the video frames within that clip. For each video frame, multiple pixels within the frame are divided into blocks using a preset overlap ratio. Pixel overlap may exist between different blocks, meaning there may be overlapping regions. The pixel values ​​of multiple pixels within the overlapping regions are then averaged, i.e., the pixel mean is taken. Similarly, when the multimedia data is an image, the block processing targets the image itself, and the processing method is the same as for video frames, which will not be elaborated upon here.

[0234] Figure 8 shows a schematic diagram of block inference operation, where each point can be regarded as a pixel. For example, 1, 2, 5, and 6 form a block A; 2, 3, 6, and 7 form a block B; and 3, 4, 7, and 8 form a block C. There is an overlapping region 1 between blocks A and B, and an overlapping region 2 between blocks B and C. The pixel values ​​of pixels 2 and 6 in overlapping region 1 are the average pixel values ​​of all pixels in overlapping region 1, and the pixel values ​​of pixels 3 and 7 in overlapping region 2 are the average pixel values ​​of all pixels in overlapping region 1.

[0235] By using the block division method described above, which is based on a preset overlap ratio, the sensitivity of the model analysis process to the window size can be reduced, and the time consumption and computational efficiency can be further reduced by decreasing the window size.

[0236] In some embodiments, a block training strategy is adopted during the training of the diffusion model to be trained. A three-dimensional training sample unit is constructed by using a 768×768 pixel space and a 129-frame temporal window. The temporal length is aligned with the processing specifications of the pre-training data used by the diffusion model to be trained during the pre-training process. Two training units are randomly selected for each sample, which can optimize memory usage and improve computational efficiency during the spatiotemporal joint modeling process.

[0237] In some embodiments, when applying the trained first diffusion model, an innovative block-based inference strategy is employed to segment the input multimedia data (such as video clips) into multiple overlapping regions (each video frame includes multiple overlapping regions). During the inference diffusion process, each block is operated on independently (e.g., each block is adjusted independently for resolution). Then, an averaging operation is performed on the overlapping regions of adjacent blocks (i.e., an arithmetic average of the pixel values ​​in the overlapping regions between adjacent blocks) to eliminate boundary artifacts. Furthermore, the number of inference steps can be reduced through acceleration methods such as distillation, which, combined with the reduced time and resource consumption brought by block-based inference, improves computational efficiency.

[0238] In some embodiments, as shown in Figure 9, which is a schematic diagram of channel-dimensional splicing feature representation, multiple blocks 910 obtained by the above-mentioned block processing can be used as analysis objects to perform efficient analysis and feature extraction processes for image or video frames. This can also improve the efficiency of splicing analysis, reduce time consumption, and improve computational efficiency, which is not limited here.

[0239] In some embodiments, as shown in Figures 10 and 11, a schematic diagram of an interface for generating a high-resolution game video based on a low-resolution game video using the above method is provided.

[0240] In Figure 10, although the low-resolution game screen 1010 can display the overall picture effect, it has less detail; in contrast, the high-resolution game screen generated by the first diffusion model based on the low-resolution game screen 1020 has more detail and is clearer.

[0241] Similarly, in Figure 11, although the low-resolution game screen 1110 can display the overall picture effect, it has less detail; in contrast, the high-resolution game screen generated by the first diffusion model based on the low-resolution game screen 1120 has more detail and is clearer.

[0242] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0243] (ii) When the multimedia data includes biological objects, perform standard pose rotation video generation processing on the biological objects.

[0244] Indicatively, multimedia data includes at least one visual element, and biological objects are one such visual element that may exist. Biological objects are usually dynamic visual elements such as people and animals.

[0245] In some embodiments, the multimedia data is a video segment, which includes multiple video frames. If any video frame includes a biological object such as a person, the video segment is considered to be multimedia data including a biological object. Alternatively, the multimedia data is an image. If the image includes a biological object such as a person, the image is considered to be multimedia data including a biological object.

[0246] In some embodiments, taking multimedia data as an image and the image including biological objects as people as an example, standard pose rotation video generation processing can be performed on the people.

[0247] The standard posture rotation video generation process is used to predict the video formed by a person rotating a certain angle (such as 360 degrees, 180 degrees, etc.) in a standard posture (preset posture) based on a person image. For example, if the standard posture is a standing posture with arms outstretched and the preset angle is 360 degrees, then it is necessary to predict the video formed by the task rotating 360 degrees in a standing posture with arms outstretched.

[0248] In some embodiments, a training process is performed using a pre-trained diffusion model (e.g., 13B HunyuanVideo, etc.) as the backbone network (base model). The diffusion model to be trained includes pre-trained weights learned during pre-training and also obtains prior knowledge of a large amount of high-quality multimedia data (e.g., video data) based on the pre-training process.

[0249] In some embodiments, an additional visual encoding network layer is added to the diffusion model to be trained. The visual encoding network layer is used to encode the input image (such as a person image) and inputs the image feature representation corresponding to the image, noise and the person image itself into the noise-adding network layer. The noise-adding network layer adds noise to the person image and extracts the noise-adding feature representation from the noise-adding person image. The image feature representation is added as initialization information to at least one of the beginning and end of the noise-adding feature representation to obtain the image stitching feature representation.

[0250] For example: the image stitching feature representation is obtained by stitching together the image feature representation in the manner of "image feature representation - noise-added feature representation - image feature representation"; or, the image stitching feature representation is obtained by stitching together the image feature representation in the manner of "image feature representation - noise-added feature representation"; or, the image stitching feature representation is obtained by stitching together the image feature representation in the manner of "noise-added feature representation - image feature representation".

[0251] In some embodiments, based on the noise addition and denoising features of the diffusion model, while using the text corresponding to the image as a control condition, an additional skeleton map corresponding to the image is introduced as a control condition, thereby comprehensively considering the text and skeleton map in the denoising process to achieve the denoising process.

[0252] In illustrative terms, the text used as a control condition is text content constructed based on the input image, used to constrain the content and presentation of the video generated from the image. For example, if the image includes at least one scene element, the text is used to describe the performance of at least one scene element (such as action, state, or interactive behavior).

[0253] For example: if the image includes a standing person and a building, and the input text is "person walks into building", then the desired video is a video of "person walks into building"; or, if the multimedia data is a video clip, and the input text is a text description that you want to extend the length of the video clip, then a longer video clip will be generated.

[0254] In illustrative terms, the skeletal diagram corresponding to the image is used to characterize the skeletal structure of biological objects in the image. The skeletal diagram can be identified based on an image that includes biological objects.

[0255] In some embodiments, the method for extracting skeletal maps from human images mainly relies on human pose estimation techniques, with the goal of extracting the joint positions and skeletal structure of a person from the image.

[0256] To illustrate, a human detection algorithm is used to identify human figures in an image. Then, a human pose estimation algorithm is used to detect key points in the human figure region. The human pose estimation algorithm can locate the joints of the human figure (such as shoulders, elbows, knees, etc.) and output the coordinates of each joint. In addition, a skeleton map is constructed based on the detected key points according to the connection relationship of the human skeleton (such as from the neck to the shoulder, from the shoulder to the elbow, etc.).

[0257] Using skeletal maps as control conditions allows for better control over the proportion of people generated from images when generating videos based on images.

[0258] In summary, as shown in Figure 7, taking multimedia data as a person image as an example, the input text is the text content constructed based on the input person image, used to describe the content and presentation of the video generated based on the person image; therefore, Figure 7 is the architecture diagram of video generation based on person images.

[0259] In some embodiments, character videos with a resolution of 1024 (width or height of 1024 pixels; such as 1024*1024, or 1024*768, etc.) are used as the training set during training. Images of different poses and actions are rendered using the same character model (person) (to simulate different input reference images), and then dynamic images of the character model (person) rotating 360° in a standard pose (sample rotation videos) are rendered as paired training data. Each pose image and 360° rotation dynamic image serve as a pair of training data, and multiple pose images each correspond to a pair of training data, thus enabling more comprehensive pose training of the model based on a single character model.

[0260] In some embodiments, as shown in Figures 12 and 13, the interface diagram is a video output of a character's standard pose rotation based on a character image obtained using the above method.

[0261] In Figure 12, input any character image 1210. If the standard posture is a standing posture with arms outstretched, the generated video frame 1220 of the character's standard posture rotation output video is generated.

[0262] Similarly, in Figure 13, input any character image 1310. If the standard posture is a standing posture with arms outstretched, the generated video frame 1320 of the character's standard posture rotation output video is generated.

[0263] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0264] (iii) When the multimedia data is dynamic portrait data.

[0265] In some embodiments, the training process is performed using the aforementioned diffusion model to be trained as the backbone network. The diffusion model to be trained includes pre-trained weights learned through pre-training. Therefore, when using the diffusion model to be trained, prior knowledge of a large amount of high-quality multimedia data (such as video data) is also obtained based on its pre-training process. The pre-training process of the diffusion model to be trained fundamentally avoids the problems of image flickering and abrupt changes in detail in the generated video, thereby better maintaining the consistency of the subject.

[0266] In some embodiments, dynamic character portrait data with a resolution of 512 (width or height of 512 pixels; such as 512*512, or 512*N, or N*512, where N is a positive integer) is used as the training set for training. The first frame image of the dynamic character portrait video and the dynamic character portrait video are used as paired training data.

[0267] Dynamic character portraits refer to the use of animation or other dynamic effects to depict characters' poses (usually static 2D images) in games or animations, showcasing changes in movement, facial expressions, or emotional fluctuations, making the characters more vivid and expressive. Dynamic portraits build upon traditional static portraits by adding subtle dynamic effects or animations to enhance the character's expressiveness. Common effects in dynamic portraits include: Facial expressions: The character's expression changes with the plot or dialogue, such as smiling, surprise, or frowning; Detailed movements: For example, subtle movements of the character's eyes and mouth, or slight swaying of hair, making the character appear more realistic; Dynamic backgrounds: While the character portrait itself remains static, the background can have dynamic effects, such as changes in light and shadow, or rustling grass, enhancing the overall sense of movement; Poise adjustments: The character may make slight pose adjustments in different situations, such as moving slightly during dialogue or making larger movements in intense scenes; Overlaying special effects: Sometimes, lighting effects, flashing eyes, or other visual effects are added to emphasize emotions or atmosphere.

[0268] Acquiring dynamic character portrait data as training data for a diffusion model can improve temporal stability and detail generation capabilities, making characters in games or animations more vivid and three-dimensional, and providing viewers or players with a stronger sense of immersion, especially in scenarios such as visual novels, romance adventures, and combat games.

[0269] In some embodiments, after training the diffusion model to be trained based on dynamic portrait data, the inference process is consistent with the prediction process of the diffusion model to be trained.

[0270] In some embodiments, as shown in Figures 14 and 15, the content of the video is generated by dynamically drawing the character using the above method.

[0271] In Figure 14, any input image 1410 can be converted into a video with dynamic character portrait effects (any video frame 1420 is shown in the figure), such as cherry blossoms fluttering; similarly, in Figure 15, any input image 1510 can be converted into a video with dynamic character portrait effects (any video frame 1520 is shown in the figure), such as skirts fluttering, blinking, etc.

[0272] It is worth noting that the above are merely illustrative examples, and the embodiments of this application are not limited thereto.

[0273] This application describes training applications for multiple functions based on the diffusion model to be trained. In super-resolution video generation, the weighting coefficients can be adaptively and dynamically adjusted through channel fusion, effectively avoiding the contradiction between maintaining object attributes and improving sharpness caused by manually adjusting the rate of decrease in weighting coefficients. At the same time, it effectively reduces the dependence on window size and overlapping area size, improving the robustness of the first diffusion model trained. Furthermore, a visual encoding network layer can be introduced based on the diffusion model to be trained, thereby generating multi-angle rotating videos for characters and improving the controllability of characters. Applying it to dynamic character portrait scenes can also improve temporal stability and detail generation capabilities.

[0274] This application also provides another model training method, which is executed by an electronic device (client or server), and mainly includes the following steps:

[0275] S11. Acquire third multimedia data and fourth multimedia data, wherein the third multimedia data is image data including biological objects and the fourth multimedia data is video data including said biological objects;

[0276] S12. Add a second noise to the fourth multimedia data to obtain the fourth noisy data;

[0277] S13. By using the diffusion model to be trained, the third data feature representation of the third multimedia data and the fourth noise-added feature representation corresponding to the fourth noise-added data are concatenated in the channel dimension to obtain the second concatenated feature representation, wherein the number of channels in the second concatenated feature representation is the sum of the number of channels in the third data feature representation and the fourth noise-added feature representation.

[0278] In some embodiments, the augmented computation model to be trained is a generative diffusion model trained with the model. Generative diffusion models include, for example, a generative diffusion model based on 13B HunyuanVideo with 1.3 billion parameters, denoising diffusion probabilistic models (DDPMs), score-based generative models (SBGM), etc., which are not limited here.

[0279] In some embodiments, the fourth noise-added feature representation corresponding to the fourth noise-added data is obtained by the following method: extracting features from the fourth noise-added data to obtain a third noise-added feature representation; extracting the skeletal map of the biological object in each frame of the fourth multimedia data to obtain a skeletal map video composed of the extracted skeletal maps; extracting features from the skeletal map video to obtain a skeletal feature representation; and fusing the third noise-added feature representation and the skeletal feature representation to obtain a fourth noise-added feature representation.

[0280] S14. Using the diffusion model to be trained, predict the second denoised noise based on the second splicing feature representation, determine the second loss function based on the second denoised noise, and train the diffusion model to be trained using the second loss function to obtain the second diffusion model.

[0281] In some embodiments, the second loss function can be constructed based on the difference between the second denoised noise and the second added noise. Here, the second added noise is the true value of the noise added to the fourth multimedia data, and the second denoised noise is the predicted value of the noise added to the fourth multimedia data.

[0282] In some embodiments, the third multimedia data is an image including a biological object (the biological object in the image can be in any pose), and the fourth multimedia data is a video of the biological object performing a rotational motion according to a preset pose (e.g., a character's standard pose). Using the image of the biological object in any pose and the video of it performing a rotational motion according to the preset pose as a pair of training data, a second diffusion model is trained. The second diffusion model can output a video of a character rotating in a standard pose based on the input image of the character.

[0283] In other embodiments, the third multimedia data is the first frame of the animated video, and the fourth multimedia data is the animated video itself. Using the first frame of the animated video and the animated video as a pair of training data, a diffusion model is trained to obtain a second diffusion model. This second diffusion model can generate a video with dynamic portrait effects based on the input portrait image.

[0284] In some embodiments, after training the second diffusion model, multimedia generation data can be obtained using the second diffusion model.

[0285] In a schematic manner, fifth multimedia data is acquired, which is image data; if the fifth multimedia data includes biological objects, the skeletal diagram corresponding to the biological objects is extracted; and multimedia generated data corresponding to the fifth multimedia data is generated using the skeletal diagram corresponding to the biological objects in the fifth multimedia data as data guiding information through a second diffusion model.

[0286] In some embodiments, generating multimedia data corresponding to the fifth multimedia data using the skeletal diagram corresponding to the biological object in the fifth multimedia data as data guidance information includes: obtaining a fifth noisy feature representation extracted after adding noise to the fifth multimedia data; and performing denoising processing on the fifth noisy feature representation under the guidance of the data guidance information through a second diffusion model to obtain the multimedia data corresponding to the fifth multimedia data.

[0287] In some embodiments, generating multimedia data corresponding to the fifth multimedia data using the skeletal diagram corresponding to the biological object in the fifth multimedia data as data guidance information includes: extracting image feature representations corresponding to the fifth multimedia data; obtaining a fifth noisy feature representation extracted after adding noise to the fifth multimedia data; splicing the image feature representation and the fifth noisy feature representation to obtain an image splicing feature representation; and performing denoising processing on the image splicing feature representation under the guidance of the data guidance information through a second diffusion model to generate multimedia data corresponding to the fifth multimedia data.

[0288] In this application embodiment, a second diffusion model is obtained by training a diffusion model to be trained, and the application method of the second diffusion model is described. A visual encoding network layer can be introduced based on the diffusion model to be trained, thereby generating multi-angle rotating videos for characters and improving the controllability of characters; applying it to dynamic portrait scenes can also improve temporal stability and detail generation capabilities.

[0289] Figure 16 is a structural block diagram of a model training apparatus provided in some embodiments of this application. As shown in Figure 16, the apparatus includes the following parts:

[0290] The first acquisition module 1610 is used to acquire first multimedia data and second multimedia data, wherein the first multimedia data and the second multimedia data have the same content but different resolutions, and the first resolution of the first multimedia data is higher than the second resolution of the second multimedia data.

[0291] The first noise-adding module 1620 is used to add first noise to the first multimedia data to obtain first noise-adding data, and to add the first noise to the second multimedia data to obtain second noise-adding data;

[0292] The first splicing module 1630 is used to splice the first noisy feature representation of the first noisy data and the second noisy feature representation of the second noisy data in the channel dimension through the diffusion model to be trained, so as to obtain the first spliced ​​feature representation, wherein the number of channels of the first spliced ​​feature representation is the sum of the number of channels of the first noisy feature representation and the second noisy feature representation;

[0293] The first training module 1640 is used to determine a first denoising noise based on the first spliced ​​feature representation using the diffusion model to be trained, determine a first loss function based on the first denoising noise, and train the diffusion model to be trained using the first loss function to obtain a first diffusion model.

[0294] Figure 17 is a structural block diagram of a model training apparatus provided in some other embodiments of this application. As shown in Figure 17, the apparatus includes the following parts:

[0295] The second acquisition module 1710 is used to acquire third multimedia data and fourth multimedia data, wherein the third multimedia data is image data including biological objects and the fourth multimedia data is video data including the biological objects.

[0296] The second noise-adding module 1720 is used to add second noise to the fourth multimedia data to obtain fourth noise-adding data;

[0297] The second splicing module 1730 is used to splice the third data feature representation of the third multimedia data and the fourth noise-added feature representation corresponding to the fourth noise-added data in the channel dimension through the second diffusion model to be trained, so as to obtain the second spliced ​​feature representation, wherein the number of channels of the second spliced ​​feature representation is the sum of the number of channels of the third data feature representation and the fourth noise-added feature representation;

[0298] The second training module 1740 is used to predict the second denoised noise based on the second spliced ​​feature representation using the second diffusion model to be trained, determine the second loss function according to the second denoised noise, and train the second diffusion model to be trained using the second loss function to obtain the second diffusion model.

[0299] It should be noted that the multimedia data processing apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multimedia data processing apparatus and the multimedia data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0300] Figure 18 shows a schematic diagram of the server structure provided in some embodiments of this application. Specifically, it includes the following structure.

[0301] Server 1800 includes a Central Processing Unit (CPU) 1801, a system memory 1804 including Random Access Memory (RAM) 1802 and Read Only Memory (ROM) 1803, and a system bus 1805 connecting the system memory 1804 and the CPU 1801. Server 1800 also includes a mass storage device 1806 for storing the operating system 1813, application programs 1814, and other program modules 1815.

[0302] Mass storage device 1806 is connected to central processing unit 1801 via a mass storage controller (not shown) connected to system bus 1805. Mass storage device 1806 and its associated computer-readable media provide non-volatile storage for server 1800. That is, mass storage device 1806 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drive.

[0303] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1804 and mass storage device 1806 described above can be collectively referred to as memory.

[0304] According to various embodiments of this application, server 1800 can also be connected to a remote computer on a network, such as the Internet. That is, server 1800 can be connected to network 1812 via network interface unit 1811 connected to system bus 1805, or it can also use network interface unit 1811 to connect to other types of networks or remote computer systems (not shown).

[0305] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0306] Embodiments of this application also provide a computer device including a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the multimedia data processing method provided in the above-described method embodiments. In some embodiments, the computer device may be a terminal or a server.

[0307] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the multimedia data processing method provided in the above-described method embodiments.

[0308] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the multimedia data processing methods described in the above embodiments.

[0309] In some embodiments, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments described above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0310] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0311] The above description is merely an embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model training method, performed by an electronic device, the method comprising: Acquire first multimedia data and second multimedia data, wherein the first multimedia data and the second multimedia data have the same content but different resolutions, and the first resolution of the first multimedia data is higher than the second resolution of the second multimedia data; Adding a first noise to the first multimedia data to obtain first noisy data, and adding the first noise to the second multimedia data to obtain second noisy data; The first concatenated feature representation is obtained by concatenating the first noisy feature representation of the first noisy data and the second noisy feature representation of the second noisy data in the channel dimension using the diffusion model to be trained. The number of channels in the first concatenated feature representation is the sum of the number of channels in the first noisy feature representation and the second noisy feature representation. Using the diffusion model to be trained, a first denoised noise is predicted based on the first spliced ​​feature representation, a first loss function is determined based on the first denoised noise, and the diffusion model to be trained is trained using the first loss function to obtain the first diffusion model.

2. The method according to claim 1, wherein, The method further includes: Obtain sample guidance information, which is used to guide the diffusion model to be trained to learn to perform resolution adjustment on the second multimedia data based on the first multimedia data; Predicting the first denoised noise based on the first splicing feature representation includes: predicting the first denoised noise based on the first splicing feature representation and guided by the sample guidance information.

3. The method according to any one of claims 1 to 2, wherein, Determining the first loss function based on the first denoised noise includes: The first loss function is constructed based on the difference between the first denoised noise and the first added noise.

4. The method according to any one of claims 1 to 3, wherein, After predicting the first denoised noise based on the first spliced ​​feature representation, the method further includes: performing denoising processing on the first spliced ​​feature representation based on the first denoised noise to obtain a denoised feature representation; The step of determining the first loss function based on the first denoised noise includes: constructing the first loss function based on the difference between the denoised feature representation and the first data feature representation corresponding to the first multimedia data.

5. The method according to any one of claims 1 to 4, wherein, After predicting the first denoised noise based on the first concatenated feature representation, the process further includes: performing denoising processing on the first concatenated feature representation based on the first denoised noise to obtain a denoised feature representation; and performing decoding processing on the denoised feature representation to obtain decoded data. Determining the first loss function based on the first denoised noise includes: constructing the first loss function based on the difference between the decoded data and the first multimedia data.

6. The method according to any one of claims 1 to 5, wherein, After obtaining the first diffusion model, the process also includes: Acquire first noisy multimedia data and data guidance information, wherein the data guidance information is used to guide the denoising of the first noisy multimedia data; The first noisy data feature representation corresponding to the first noisy multimedia data is extracted using the first diffusion model; Guided by the data guidance information, the first noisy data feature representation is subjected to denoising processing to obtain the first denoised data feature representation; Decoding the first denoised data feature representation yields the first denoised multimedia data, the resolution of the first denoised multimedia data being higher than the resolution of the first noisy multimedia data.

7. The method according to any one of claims 1 to 5, wherein, After obtaining the first diffusion model, the process also includes: Using the first diffusion model, a block inference strategy is employed to adjust the resolution of the input second noisy multimedia data, resulting in second denoised multimedia data. The resolution of the second denoised multimedia data is higher than that of the second noisy multimedia data.

8. The method according to claim 7, wherein, The step of using the first diffusion model and employing a block-based inference strategy to distinguish and adjust the input second noisy multimedia data to obtain second denoised multimedia data includes: According to a preset overlap ratio, the second noisy multimedia data is divided into multiple noisy blocks with overlapping regions; The first diffusion model is used to perform independent resolution adjustment on each noisy block to obtain the denoised block corresponding to the noisy block. An arithmetic mean operation is performed on the pixel values ​​in the overlapping region between adjacent denoising blocks to fuse the denoising results of adjacent denoising blocks.

9. A model training method, performed by an electronic device, the method comprising: Acquire third multimedia data and fourth multimedia data, wherein the third multimedia data is image data including biological objects, and the fourth multimedia data is video data including the biological objects; A second noise is added to the fourth multimedia data to obtain the fourth noisy data; By concatenating the third data feature representation of the third multimedia data and the fourth noisy feature representation corresponding to the fourth noisy data in the channel dimension using the diffusion model to be trained, a second concatenated feature representation is obtained, wherein the number of channels in the second concatenated feature representation is the sum of the number of channels in the third data feature representation and the fourth noisy feature representation; Using the diffusion model to be trained, the second denoised noise is predicted based on the second spliced ​​feature representation, the second loss function is determined based on the second denoised noise, and the diffusion model to be trained is trained using the second loss function to obtain the second diffusion model.

10. The method according to claim 9, wherein, The step of determining the second loss function based on the second denoised noise includes: The second loss function is constructed based on the difference between the second denoised noise and the second added noise.

11. The method according to claims 9 to 10, wherein, The fourth noise-added feature representation corresponding to the fourth noise-added data is obtained using the following method: The features of the fourth noisy data are extracted to obtain the third noisy feature representation; Extract the skeletal map of the biological object from each frame of the fourth multimedia data to obtain a skeletal map video composed of the extracted skeletal maps; extract the features of the skeletal map video to obtain a skeletal feature representation; The third noisy feature representation and the skeletal feature representation are fused to obtain the fourth noisy feature representation.

12. The method according to any one of claims 9 to 11, wherein, The fourth multimedia data is a video of the biological object performing a rotational motion according to a preset posture; Alternatively, the third multimedia data may be the first frame of the animated video, and the fourth multimedia data may be the animated video itself.

13. The method according to any one of claims 9 to 12, wherein, The method further includes: Acquire fifth multimedia data, wherein the fifth multimedia data is image data; If the fifth multimedia data includes biological objects, extract the skeletal diagram corresponding to the biological objects; Multimedia generation data corresponding to the fifth multimedia data is generated using the second diffusion model and the skeletal diagram corresponding to the biological object as data guidance information.

14. The method according to claim 13, wherein, The process of generating multimedia data corresponding to the fifth multimedia data using the skeletal diagram corresponding to the biological object as data guidance information includes: Obtain the fifth noise-added feature representation extracted from the fifth multimedia data after adding noise; Guided by the data guidance information, the fifth noisy feature representation is denoised using the second diffusion model to obtain the multimedia generated data corresponding to the fifth multimedia data.

15. The method according to claim 13, wherein, The process of generating multimedia data corresponding to the fifth multimedia data using the skeletal diagram corresponding to the biological object as data guidance information includes: Extract the image feature representation corresponding to the fifth multimedia data; Obtain the fifth noise-added feature representation extracted from the fifth multimedia data after adding noise; By concatenating the image feature representation and the fifth noisy feature representation, an image concatenation feature representation is obtained; Guided by the data guidance information, the second diffusion model performs denoising processing on the image stitching feature representation to generate the multimedia generation data corresponding to the fifth multimedia data.

16. A model training device, wherein, The device includes: The first acquisition module is used to acquire first multimedia data and second multimedia data, wherein the first multimedia data and the second multimedia data have the same content but different resolutions, and the first resolution of the first multimedia data is higher than the second resolution of the second multimedia data. The first noise-adding module is used to add first noise to the first multimedia data to obtain first noise-adding data, and to add the first noise to the second multimedia data to obtain second noise-adding data; The first splicing module is used to splice the first noisy feature representation of the first noisy data and the second noisy feature representation of the second noisy data in the channel dimension through the diffusion model to be trained, so as to obtain the first spliced ​​feature representation, wherein the number of channels of the first spliced ​​feature representation is the sum of the number of channels of the first noisy feature representation and the second noisy feature representation; The first training module is used to predict first denoised noise based on the spliced ​​feature representation using the diffusion model to be trained, determine a first loss function based on the first denoised noise, and train the diffusion model to be trained using the first loss function to obtain a first diffusion model.

17. A model training device, wherein, The device includes: The second acquisition module is used to acquire third multimedia data and fourth multimedia data, wherein the third multimedia data is image data including biological objects and the fourth multimedia data is video data including the biological objects; The second noise-adding module is used to add second noise to the fourth multimedia data to obtain fourth noise-adding data; The second splicing module is used to splice the third data feature representation of the third multimedia data and the fourth noise-added feature representation corresponding to the fourth noise-added data in the channel dimension through the second diffusion model to be trained, so as to obtain the second spliced ​​feature representation, wherein the number of channels of the second spliced ​​feature representation is the sum of the number of channels of the third data feature representation and the fourth noise-added feature representation; The second training module is used to predict the second denoised noise based on the second spliced ​​feature representation using the second diffusion model to be trained, determine the second loss function according to the second denoised noise, and train the second diffusion model to be trained using the second loss function to obtain the second diffusion model.

18. A computer device, wherein, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the multimedia data processing method as described in any one of claims 1 to 15.

19. A computer-readable storage medium, wherein, The storage medium stores at least one program, which is loaded and executed by a processor to implement the multimedia data processing method as described in any one of claims 1 to 15.

20. A computer program product, wherein, It includes computer instructions that, when executed by a processor, implement the multimedia data processing method as described in any one of claims 1 to 15.